Society for Improving and Measuring Procedural Learning in the Operating Room (SIMPL OR) app as a Practical Measure

Characteristics of Practical Measures	Descriptions of the SIMPL OR App
Is closely tied to a theory of improvement	The SIMPL OR app is used to support multiple improvement efforts with a common aim to increase practice-readiness among surgery trainees. When it is used as a measure for the aim of “every trainee achieves practice-readiness on one specialty-defined index procedure within each postgraduate year,” observations within individuals are transformed into readiness scores using a prediction model. The readiness score is also used as an outcome measure for testing teaching and learning strategies. Another metric of interest is the number of times trainees request feedback. This metric is used as a PDSA measure to test change ideas designed to improve trainee engagement in feedback-seeking behaviors. (See network improvement project and article on readiness)
Provides actionable information to drive positive changes in practice	The SIMPL Collaborative learned from the users that the SIMPL OR app has face validity. In other words, it is not difficult for users to understand what gets measured. On the one hand, trainees can use the SIMPL OR app to reflect on their performance and to receive formative feedback from faculty, which can guide their efforts in improving their practices. On the other hand, faculty can use aggregated SIMPL OR app data to personalize their teaching and supervision in the operating room. Moreover, program directors can use the data to identify procedures that are particularly challenging and design change ideas to support faculty and trainees. Recently, to maximize the actionability of the data, the Collaborative has invested in optimizing and differentiating data visualizations.
Captures variability in performance	The SIMPL OR app captures variability in trainee performance across operating procedures over time. It sounds straightforward, but when implemented in practice, various implementation challenges may emerge and can add noise to the data — this can potentially make the results less interpretable and hence less actionable. For example, in programs with a smaller faculty size, trainees may not be able to gather feedback from multiple faculty members.
Demonstrates predictive validity	A statistical prediction model has been developed by the SIMPL Collaborative to make sense of observations. Once a data point is entered into the system, a readiness score is generated to estimate the likelihood of success in the next operation. The Collaborative is in the process of understanding how trainees and faculty make sense of readiness scores. In addition, the SIMPL OR app results are expected to predict end-of-rotation performance evaluation results. However, because the latter lacks variability, the Collaborative is planning on shifting focus to studying the predictive relationships between SIMPL OR app results and post-graduation/career outcomes.
Is minimally burdensome to users	Being a phone application makes the SIMPL OR app highly accessible and minimally burdensome. On top of that, the SIMPL OR app embraces user autonomy. Trainees can choose whether to request feedback after a case closes, while faculty can choose whether to accept a feedback request and if they are to dictate a qualitative response. This design helps both trainees and faculty perceive the use of the SIMPL OR app as something they want to do instead of something they have to do.
Functions within social processes that support improvement culture	The SIMPL OR app is used as an assessment for improvement, not an assessment for accountability. Trainees are allowed to request feedback as little as one time or as many times as needed to inform their learning. Given the voluntary nature of asking for feedback, the SIMPL Collaborative designed change ideas to encourage trainees to request more feedback.
Is reported on in a timely manner	Trainees are only required to be formally evaluated through direct observation twice a year. The SIMPL OR app makes the evaluation process simpler and thus encourages more frequent feedback. Once a case closes, the trainee can request feedback from the supervising surgeon. After the surgeon provides feedback by answering three questions on the app, the system will instantly notify the trainee about the survey responses.

Question on Practical Measures Inspired by the SIMPL OR App

Are practical measures synonymous with quick-and-dirty measures?
Practical measures like the SIMPL OR app can often be implemented swiftly and easily. Yet, the process of developing and refining practical measures is usually far from easy. “In diving into the technical assessment properties of [SIMPL OR], we have opened up interesting improvement opportunities,” said Dr. Andrew Krumm, a member of the SIMPL Collaborative. What contributes to a surgeon’s rating of trainee performance? The Collaborative found that an attending surgeon’s rating reflects not only trainee competence, but also a number of other factors, such as the rarity of the procedure and, most of all, the attending surgeon’s own perception of good performance — a factor that contributes a majority of the variance in attending surgeons’ ratings. To make sure that the SIMPL OR app provides accurate assessment of trainee performance to inform improvement efforts, the Collaborative tested change ideas (e.g., specifying key procedures to be assessed, retraining raters) to minimize unwanted sources of variation. Examining the psychometric properties of a measure, identifying problems, and testing potential solutions take expertise, time, and resources; however, these efforts allow improvers to be confident about the efficacy of change being measured using the app’s data.

Student Engagement Exit Ticket as a Practical Measure

Characteristics of Practical Measures	Descriptions of the Student Engagement Exit Ticket
Is closely tied to a theory of improvement	Although the Student Engagement Exit Ticket measure was not tied to any formal theory of improvement, it was in alignment with teachers’ hypotheses of how changes they made would lead to improvements. The measure is associated with a problem of practice (i.e., difficulty motivating students) shared by a group of teachers, and it assesses ideal outcomes identified by the group of teachers (e.g., students’ intrinsic interest in lessons). Teachers who created the measure together used it to test their individual change ideas and learned from one another.
Provides actionable information to drive positive changes in practice	Because the measure was primarily designed by teachers, the data collected using the survey could provide a window for teachers into aspects of their students’ engagement that they cared about most and worked hardest to improve. This should increase the likelihood of them acting on the data meaningfully and constructively.
Captures variability in performance	For the group of teacher-designers involved, the measure served as a common measurement for improvement. By examining variation in the common measure across classrooms, teachers could answer questions about what works, for whom, and under what conditions.
Demonstrates predictive validity	The measure was constructed in the midst of the COVID-19 pandemic to solve urgent problems of practice that arose as schools transitioned from in-person to remote learning. Hence, the predictive validity of the measure was not studied.  In the process of developing this measure, teachers provided look-fors related to their problem of practice and identified success indicators. Though not formally studied, this approach to the design enabled teachers to make informal predictions about variation in student engagement.
Is minimally burdensome to users	During the outbreak of the COVID-19 pandemic, students were asked to fill out many surveys. A lot of them experienced survey fatigue. To make it easier on students, the professional learning design team led by Annie Wilhelm, Associate Professor at the Southern Methodist University, kept the measure short – only two close-ended items were included. Also, the survey was built using Google Forms (a tool that the teachers and students had previous experience using) and was administered during class time. As a result, students should be less likely to perceive completing the measure as “one more thing” to do.
Functions within social processes that support improvement culture	The measure is a product of group ideation. As teachers brainstormed outcomes worth tracking, they engaged in collaborative visioning of what success could look like. Once a shared vision was established, teachers could then 1) backward-map the changes they would make individually to actualize the vision, 2) implement the changes, 3) collect data using the measure to learn about the implementation, and 4) participate in collective sense-making by going through a notice-and-wonder data protocol. All of these processes together transformed the measure from a conventional exit ticket to a learning tool for improvement.
Is reported on in a timely manner	Teachers who tested their change ideas using the measure received reports at a professional development session following testing. Such a delay might make it harder for teachers to make connections between student experience and practices they tried as part of their change ideas, especially if they did not also document those practices.

Question on Practical Measures Inspired by the Student Engagement Exit Ticket Measure

What can improvers do to make the design process of practical measures more practical?
It is not uncommon for improvers to find the process of developing practical measures laborious. Sometimes, even after a lot of hard work has been put into designing a practical measure and making it “rigorous,” those who are closest to the work, such as teachers, still find it difficult to see how using the measure can help them improve their practice, and the measure may end up sitting on the shelf.

One way to address this challenge is to involve practitioners early on in the design process of practical measures. Annie Wilhelm, who built the design process for the Student Engagement Exit Ticket, said, “I was trying to make the [design process] as close to [teachers’] practice and their needs as possible.” For Annie, it’s important for teachers to have a sense of agency over their own learning. Although the teacher-driven process that she crafted was “possibly wrong and definitely incomplete,” it “got something out there that would give [teachers] something to work with.”

Annie’s example demonstrates that when it comes to bringing in teachers to create practical measures, it is okay to start small. In addition, modeling and scaffolding might be needed to help teachers better understand not only the steps but also the value of the process. As their capacity builds, teachers will be more prepared to iterate on practical measures to optimize their rigor, impact, and practicality.

Student Independent Writing Time as a Practical Measure

Characteristics of Practical Measures	Descriptions of the Student Independent Writing Time Measure
Is closely tied to a theory of improvement	The Student Independent Writing Time measure — and other measures such as the Writing Conference Tracker — is a part of a system of measures developed to test a theory of improvement with an aim to increase student proficiency in writing and with drivers focusing on student learning opportunities and instructional practices. It is a driver measure that aligns specifically with the “write independently for 20-30 minutes every day” driver. To address this driver, teachers have created different change ideas, including planning for no more than two “teachable moments” during the direct instruction part of the lesson to save time for independent writing.
Provides actionable information to drive positive changes in practice	The measure provides information for teachers to answer the question, “How much independent writing time is happening during writing lessons?” Without the measure, teachers might gauge the impact of their change ideas by intuition. Through use of the measure, teachers can discern patterns of their instructional practice and make adaptations to ensure independent writing time happens consistently. In the Literacy Improvement Partnership, some teachers leveraged the measure results to fine-tune their work in their classrooms and observed increases in student independent writing.
Captures variability in performance	As teachers log students’ independent writing time, the data entered are used to populate run charts automatically. Each run chart captures variation in independent writing time within a classroom over time. A benefit of using run charts to display data is that it allows practitioners to distinguish between common cause variation (e.g., variation to be expected from a process) and special cause variation (e.g., variation induced by change ideas). If a teacher can pinpoint when a change is induced on a run chart, comparison of the data pattern pre- and post-change can be done to assess empirically whether the change is an improvement.
Demonstrates predictive validity	Although the Literacy Improvement Partnership team did not conduct an analysis of predictive validity of the measure, research from the field suggests that student writing time matters for students’ improved writing.1
Is minimally burdensome to users	Although the logging of independent writing time is not time-consuming (the process takes less than a few minutes), it is one more thing that teachers have to do. Teachers in the Literacy Improvement Partnership found the daily documentation burdensome when sustained for about two months. Using the measure in learning sprints that last for 4-5 weeks and are spaced apart was preferred.
Functions within social processes that support improvement culture	The teacher-report data is not used in teacher evaluation. Yet, because it is not anonymous, teachers doing the reporting may be concerned about who has access to the data. This can have implications on teacher willingness to be transparent about their practice. Efforts directed toward building a culture of social learning and a sense of psychological safety around innovation and experimentation should be prioritized to invigorate teacher reflection and collaborative inquiry. In the case of the Literacy Improvement Partnership, the hub supports schools in establishing a learning huddle structure to do just that.
Is reported on in a timely manner	Teachers are expected to log the independent writing time daily. Some teachers enter data into the designated Google Sheet after a writing lesson or at the end of a school day. Other teachers choose to document daily using paper and pencil and engage in online data entry once at the end of each week. In either case, they can use the automatically populated run charts to reflect on their practice any time they deem it valuable.

¹Graham, S., Bollinger, A., Booth Olson, C., D’Aoust, C., MacArthur, C., McCutchen, D., & Olinghouse, N.(2012).Teaching elementary school students to be effective writers: A practice guide (NCEE 2012- 4058). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/ wwc/publications_reviews.aspx#pubsearch.

Questions on Practical Measures Inspired by the Student Independent Writing Time Measure

What are the functions of practical measures in improvement beyond working as a tool to test a theory of improvement?
In the process of identifying a practical measure to test a change, practitioners discover along the way what the “must-have” and “nice-to-have” elements of the change are. Measurement in many ways shapes intervention. In some cases, measurement is even a major part of the intervention. Sola Takahashi, a researcher who led the development of the Student independent Writing measure, said in an interview, “The recording of the measure was a change idea in itself…Just by virtue of tracking this information, teachers were pressed to attend to an important aspect of their instruction that they may not have been thinking about.”

What contributes to the effective use of practical measures?
Although practitioners are increasingly trained to use data to inform practice (as shown by the increased adoption of data-driven instruction), routine reflection and learning from practice-based data may be hampered by suboptimal school and district context (e.g., a top-down decision-making culture). To maximize what practitioners get out of using practical measures to improve the student learning experience, system leaders should not only focus on building practitioner analytical capacity, but also invest in building coherent structures, processes, and norms that constitute a scientific professional learning system. The learning huddle used in the Literacy Improvement Partnership is an example of such an investment.

Financial Aid Application Completion as a Practical Measure

Characteristics of Practical Measures	Descriptions of the Inclusive Classrooms Survey Measure
Is closely tied to a theory of improvement	The Financial Aid Application Completion measure is used as a driver measure to monitor shifts in financial access, which is one of the three drivers included in the CARPE College Access Network’s theory of improvement. The theory is that improvement in Free Application for Federal Student Aid (FAFSA) application completion rate will contribute to improvement in college enrollment rate (i.e., percentage of students enrolled in tertiary institutions where they are most likely to graduate).
Provides actionable information to drive positive changes in practice	By disaggregating the data by network, by school, by student — in some cases, by counselor and by targeted student group — the hub has transformed publicly available but hard-to-use data into accessible and actionable information for diverse stakeholders. It turns out that the aggregate dashboards are the most helpful for school leaders, such as assistant principals, to make data-driven decisions to create conditions that maximize the impact of change ideas targeting FAFSA application completion. Meanwhile, the by-student data is the most pivotal to the work of counseling team members, as they can use the information to prioritize and keep their efforts consistent with the Multi-Tiered System of Supports (MTSS) framework.
Captures variability in performance	By building dashboards, the CARPE hub provides means for schools to understand variation among student groups and among individuals, as well as variation over time (e.g., 2019-20 data plotted against 2020-21 data). In addition, the CARPE hub visualizes the data aggregated by school using 1) small multiples ordered by the number of students in each school and 2) frequency-table-sorted based on percentage of application completers. These visuals capture variation across schools and allow the hub to be agile in strategizing coaching plans to meet the schools where they are at.
Demonstrates predictive validity	The CARPE hub has built dashboards for both Financial Aid Application Completion and Verified College Enrollment (data pulled from the National Student Clearinghouse) and has examined how changes in one are related to changes in the other. The hub found that among schools that had the largest increases in FAFSA application completion in 2018-19, there were increases in college enrollment — more at two-year colleges than four-year institutions. This finding was promising, but given that not as much variance in college enrollment was accounted for by FAFSA application completion as expected, the hub concluded that focusing solely on FAFSA application completion was not enough to achieve the network’s aim. Therefore, the hub expanded the improvement work to other drivers like college application completion.
Is minimally burdensome to users	One of many advantages of using extant measures like the Financial Aid Application Completion measure is that users do not have to spend extra time on collecting and documenting data. The time saved can be used in implementing social learning routines (e.g., data huddles) that facilitate knowledge consolidation and cross-pollination.
Functions within social processes that support improvement culture	The CARPE hub encourages the schools to interact with the Financial Aid Application Completion data system at least bi-weekly so the measure is more of a tool for learning and less of a tool for evaluation. This aligns with the hub’s emphasis on building a network culture that prioritizes measurement for improvement over measurement for accountability. All data displays and routines are set up in ways that facilitate collaboration and not competition.
Is reported on in a timely manner	The by-network and by-school data are updated weekly by the CARPE hub while the by-student data is updated bi-weekly by a data team lead from each school. The processes are highly standardized, and the frequency of data pulls may increase during critical periods, particularly in October and from January to March, as it is important to both start strong and end strong. Timely information can be provided to improvers because the hub invests time and resources not only in designing and testing data infrastructure and routines but also in building the capacity of network members to organize, visualize, and understand data so they can up their data game when the time calls for it.

Question on Practical Measures Inspired by the Financial Aid Application Completion Measure

How might we deal with a lack of consensus over how something should be measured?
The improvement principle “make the work user-centered” is applicable here. For the CARPE College Access Network, the operational definition of the primary driver, financial access, is FAFSA completion, CalGrant awardance, and paying for college.

With such a definition, one would expect the measures of the driver to be straightforward. However, as the CARPE hub members dove into calculating the FAFSA completion rate, they realized that there was little agreement over which denominator should be used. Should it be the total number of senior students enrolled in a school? Or, should it only count the number of senior students who have expressed interest in going to college?

To address this challenge, the CARPE hub encouraged schools to submit their own calculations. The hub then used that data to calibrate the data pulled from the California Student Aid Commission Race to Submit Dashboard and the Department of Education websites. The advantage of using an approach like this is that it could potentially help elevate user voice and build trust in the process and the data, which is crucial to continuous improvement.

As the school-reported FAFSA completion rates started to go up, the CARPE hub needed to address another key improvement question: Were those changes really improvements? After auditing the changes using denominators consistent with publicly available enrollment data, the CARPE hub could confidently conclude that the changes they were observing were real improvements and not merely changes in the school-reported measure. 

Participation Measure as a Practical Measure

Characteristics of Practical Measures	Descriptions of the Participation Measure
Is closely tied to a theory of improvement	Following the increase in educational exclusion during the pandemic, the Un Buen Comienzo (UBC) Improvement Network shifted its focus from attendance to participation and developed an aim for 80% of preschool and kindergarten children participating in at least three learning activities each week. The Participation Spreadsheet/Tracker is a tool to track a leading outcome measure. It monitors student engagement in learning experiences and helps teachers, school administrators, and network leaders identify individual students who are falling behind or at risk of not finishing successfully so they can develop personalized interventions for them. This measure is tied to the network’s theory of improvement, and tracking student engagement in learning activities supports their efforts to prevent students from experiencing further educational exclusion.
Provides actionable information to drive positive changes in practice	This measure relies on teachers to collect weekly data on each student’s participation. Using this information, the hub team generates weekly classroom- and school-level reports to share with teachers and school leaders. School teams can then review the disaggregated data to better understand patterns in participation and discern which classroom practices are helping enable student engagement and which are hindering engagement. The participation data are also shared in monthly district meetings, led by trained participation/attendance managers, to engage different school teams in collective sense-making, surface bright or blind spots, and identify timely strategies for improving participation.
Captures variability in performance	Once teachers record the student participation data, the Participation Spreadsheet/Tracker automatically feeds and produces weekly Data Studio reports that record student participation rates across classrooms, schools, districts, and the entire network over time. The data is used to populate a run chart, included in the weekly reports shared with teachers, that monitors the percentage of students who have achieved the participation goal each week and those with a 0% participation rate. This visualization tool captures the different participation rates by school and classroom so teachers and school leaders can then use the available data to attend to the variability in student performance.
Demonstrates predictive validity	According to the UBC Network’s theory of improvement, tracking student participation through a more holistic approach focused on engagement in different types of learning activities (synchronous, asynchronous, or onsite) can help school teams prevent educational exclusion (children with 0% weekly participation) and attain their goal of 80% of children participating in three learning activities a week. Additionally, the student participation data could function like an on-track indicator. While related outcome data is not available, we would expect the participation data to predict other variables like course completion and academic achievement.
Is minimally burdensome to users	For most teachers, recording participation on the prepared Google Sheet was easy and not very time consuming. However, some teachers were less familiar with using Google Apps and needed initial support as they transitioned from recording participation/attendance in the SIGE database to utilizing this new tool. After completing the Tracker/Spreadsheet, weekly data reports are automatically generated and sent to teachers for review. This process of collecting and documenting data was therefore systematized for teachers and integrated into their daily routines.
Functions within social processes that support improvement culture	The UBC Network had strong structures and processes in place to support peer-to-peer learning around their improvement effort. Monthly meetings were held in each district to share the best pedagogical and leadership practices. These meetings were facilitated by the district’s assigned attendance manager who was trained by the Network in strategies for improving attendance and participation. During these meetings, the attendance managers shared the district participation results, analyzed the data, and provided the team an opportunity to engage in collective sense-making. During these monthly meetings, teams were able to come together in a safe, low-stakes environment to study and learn from the participation data so they could improve their outcomes. In addition, representatives from each district met with the hub team to hear from each other and identify lessons learned.
Is reported on in a timely manner	Teachers were expected to log student participation data into the Spreadsheet/Tracker each week. On the following Monday, weekly data reports were made available for school teams, which then used that data to evaluate how they were doing and identified students at risk of educational exclusion. The monthly district meetings, in particular, provided a structure for the team to make timely decisions based on their data that directly responded to the needs of their local contexts.

Question on Practical Measures Inspired by the Participation Measure

How do you maintain standardization across the network, while ensuring local needs are sufficiently addressed?
A core improvement principle insists that “variation in performance is the core problem to address.” This process requires us to center data, elevate different voices, and be disciplined about standard work processes in order to ensure that the change idea can be carried out reliably by various people, even working under different conditions. To do so, we must use practice-based evidence to inform standard work processes.

Across the UBC network, 14 districts shared this participation measure and data collection tool. What was unique about this process is that the network did not assign the learning activities for participating districts. Instead, they empowered the schools to select the activities that were most suitable for their students given the available resources and unique circumstances of the communities they served, and provided guidance to ensure quality learning experiences were implemented as needed. To support the data collection and use of the participation measure, the network did define key terms related to participation (synchronous, asynchronous, etc.), so that data was recorded consistently across sites. This approach also allowed network leaders to analyze participation data across districts and build processes to support the overall improvement effort.

Inclusive Classrooms Survey as a Practical Measure

Characteristics of Practical Measures	Descriptions of the Inclusive Classrooms Survey Measure
Is closely tied to a theory of improvement	In partnership with Panorama Education and the Rennie Center for Education Research & Policy, Teaching Matters has developed the Inclusive Classrooms Survey to monitor shifts in a primary driver. The measure has six factors and taps into domains of inclusive classroom practices that the NSI theorizes that, if fostered, can help students stay on track in 8th grade. The six domains of focus are classroom belonging, classroom engagement, cultural awareness and action, rigorous expectations, teacher-student relationship, and valuing of subject.
Provides actionable information to drive positive changes in practice	The information provided by the measure is actionable for improvers at different levels. Using the dashboards created by Panorama and data analysis protocol designed by the Teaching Matters hub, teachers can individually and collectively surface glows (i.e., bright spots) and grows (i.e., areas of concern) and work with school leaders and coaches to identify strategies — change ideas coming from the Teaching Matters NSI Change Menu and the Panorama Playbook — to better student experience. Meanwhile, the Teaching Matters hub uses the survey data to populate p-charts as the team seeks to unveil and understand variation among classrooms and schools. Through attending to variation systematically, the hub makes progress in answering the questions of what works, for whom, and under what conditions, as well as coming up with plans to support schools in addressing variation in student experience.
Captures variability in performance	Depending on the individual teacher’s or school’s choice of change idea, different domains of inclusive classroom practices may be targeted. If a teacher chooses a change idea that is designed to help students build a sense of classroom belonging, the teacher may not expect positive changes in student valuing of subject. Because the data are broken down by domains, improvers can study variability in student experience across domains and check if the observed variability aligns with their predictions.
Demonstrates predictive validity	According to the Teaching Matters NSI’s theory of improvement, inclusive classroom practices should predict 8th grade on track. To test this hypothesized causal link, the Teaching Matters hub examined the relationship between the two constructs using the Inclusive Classrooms Survey and ELA screener assessments (i.e., iReady/MAP Growth). The results suggested that improvement in inclusive classroom practices was positively related to change in the percentage  of students meeting ELA growth goals. Although further investigation is warranted, these results are evidence that the survey has predictive validity. This is important because it means that if growth in survey results is detected, it can be assumed that the network is making progress toward its aim.
Is minimally burdensome to users	It takes intentional planning to ensure smooth administration of surveys. It is no exception in the case of the Inclusive Classrooms Survey. Teaching Matters hub members described rostering as being a “medium-level lift” for schools, although they provide schools with rosters pulled from the New Visions for Public Schools data platform. Reviewing and revising rosters and coordinating as needed can be time-consuming. Also, given that the Inclusive Classrooms Survey is a 33-item measure that includes three open-ended items, doing the survey more than once a year can feel burdensome to students, especially if teachers do not share with students what they are learning from their feedback.
Functions within social processes that support improvement culture	The Teaching Matters hub acknowledges the importance of building the capacity of network members to engage in collective sensemaking of the survey data. First, the hub rolled out a data analysis protocol at a network convening. The protocol centers around three questions: 1) What are your students telling you? 2) What does that mean they are asking you to do? 3) How might you address their ask? At the convening, network members had an opportunity to practice using the protocol to examine network-level data and experience the power of reflecting as a group on the three questions. They were then asked to apply the protocol to analyze school-level data at school improvement team meetings. Through this social learning process, network members can have a better understanding of both the survey data and data analysis protocol and how they can embed both in their improvement routines.
Is reported on in a timely manner	Given that the Inclusive Classrooms Survey is designed to assess students’ general perceptions of their classroom experience (that are relatively stable and take time to change) and not their specific perceptions of lesson(s) at a particular moment in time, the survey is administered three times a year in Teaching Matters NSI schools. Following survey administration, schools will receive results consolidated in dashboards. This allows schools to consistently monitor changes in student perceptions of inclusive classroom practices. Moreover, survey administration is often situated before and after action periods when schools engage in active testing of change ideas. This can make connecting improvement in change ideas to improvement in drivers easier.

Question on Practical Measures Inspired by the Inclusive Classrooms Survey

What are the ways to build buy-in around a shared practical measure?
Within an improvement network, there are often schools with different instructional models and priorities. Sometimes it may be hard for schools or classrooms to commit to administering common measures in the testing of a theory of improvement.

The Teaching Matters hub addresses this challenge by making the measurement work student-centered. Because the Inclusive Classrooms Survey is student-reported and contains open-ended questions, educators can hear and learn from students in their own perspective and voice. Elevating student voice is crucial in doing this work. Such a focus brings together educators who are dedicated to dismantling inequities.

Also, the Teaching Matters hub invests time and resources in explicating a shared why. Building inclusive classroom practices may sound like the right approach to achieving broader accessibility, but each educator and student may envision such practices differently. To help the network establish a shared vision and commitment, the hub created a slide deck that describes six key domains of inclusive classroom practices and how they are measured by the Inclusive Classrooms Survey. This resource was shared with network schools, and schools that used this resource reported receiving more actionable data that they could learn from to inform their practice. The more schools can utilize data for improvement, the better the buy-in to collect and analyze the data.

Furthermore, the Teaching Matters hub develops structures and routines to support administration and analysis of the survey. The hub works with schools to set up a survey coordinator role and provides training to the coordinators. Additional support is also offered by the Teaching Matters improvement coaches. Besides helping teachers and school leaders learn from the survey data, coaches share what they learn from working with school teams with hub members to facilitate continuous process improvement.

The case of the Teaching Matters NSI illustrates that it takes strategic hub leadership moves to lead a network to arrive at a shared practical measure that resonates with improvers. It may be a challenging journey, but it will be worth it in the long run, as having common measures helps improvers learn from and reduce variation in performance, which is at the core of the problem of educational disparity.