“During the pandemic it was very difficult for students to log onto classes that were synchronous because they were in parts of Chile that are very rural and they didn’t have access to the internet, so the foundation looked for different ways to measure how students were participating in learning.”

— Pablo Muñoz, Head of Continuous Quality Improvement and Evaluation for the UBC Improvement Network

Introduction

In order to be “practical,” a practical measure for improvement must take into account the specific context in which it is being used – the time, the place, and the situation. It must be relevant and provide data that help practitioners address the challenges they are currently facing. Effective measures are often born out of practitioners reflecting on questions like What do we need to know right now to do our job well? and What information will help us better support our students or families in the current context? These kinds of measures reflect an unspoken urgency around improving the target outcomes. The Participation Spreadsheet/Tracker was designed to address specific challenges related to student participation during the COVID-19 pandemic. It highlights the importance of creating measures that are responsive to the needs of front-line improvers while taking into account their context so that practitioners have access to the information they need to improve their practice and processes.

The Need for Measurement

The Un Buen Comienzo (UBC) Improvement Network, supported by Fundación Educacional Oportunidad, focuses on improving early childhood education in Chile. Since its founding in 2018, the UBC Network has sought to create a space for collaboration, professional development, and pedagogical innovation among its network members representing over 150 schools across two regions in Chile.

Prior to the COVID-19 pandemic, the network focused on improving language outcomes for first- and second-year students in Early Childhood and Care programs (NT1 and NT2), which includes pre-kindergarten- and kindergarten-aged children from 4 to 6 years old. One of the primary drivers identified to achieve this goal was reducing the percentage of students who were chronically absent during the school year – missing more than 10% of school days (see Figure 1). To support this effort, the network hub leveraged data that was submitted by schools to the Chilean Ministry of Education through the General Student Information System to develop a platform that allowed them to track absenteeism across schools and specific classrooms in their network.

Figure 1. UBC Network’s Driver Diagram (2018-19) with primary driver focusing on attendance

[Image source: 2022 Summit Poster: A new perspective of school attendance in the Un Buen Comienzo Network during the pandemic]

However, in 2020, as the COVID-19 pandemic dramatically altered the way schools were operating across Chile, the Ministry of Education no longer required schools to submit classroom attendance data. With the General Student Information System no longer in use, schools lacked a way to monitor student attendance and needed the ability to understand whether students were engaging in the new structures for remote learning. Schools needed better visibility regarding which students were falling through the cracks. 

This dramatic change in context prompted the UBC Network to shift its efforts and theory of improvement to focus on student participation (see Figure 2), and they developed a measure and new data-collection tools to monitor their progress. A key challenge the network encountered as they shifted their overall aim was to clearly define what was meant by “student participation” as opposed to “student attendance.” Most people share a common understanding of the term “attendance” as it relates to schooling – it is a measure that indicates how often a student comes to school or attends a specific class. However, the UBC Network recognized that this measure was no longer relevant as schools were implementing various forms of remote learning experiences. With this new perspective, the network turned its attention to “student participation,” which they defined as the interaction of students with the school through different learning activities such as worksheet packets, synchronous online class sessions, and face-to-face instruction. (These activities varied widely by school depending on the student population, school location (rural vs. city), and other factors). 

Figure 2. UBC Network’s Driver Diagram (2020-21) with aim focusing on student participation

[Image source: 2022 Summit Poster: A new perspective of school attendance in the Un Buen Comienzo Network during the pandemic]

The Measure: Participation Spreadsheet/Tracker

The network hub created a Participation Spreadsheet/Tracker (sample spreadsheet translated to English) that allowed teachers to enter the activities that each student engaged in each week. The Participation Spreadsheet/Tracker includes a tab that provides clear instructions regarding how teachers should categorize each learning activity (which was important given the broad range of activities at each site) and tabs for each month where teachers track each students’ participation. Under the data table for each month, the spreadsheet also includes a summary table that automatically displays which students met the participation target (three or more activities) each week.

The core purpose of a practical measure for improvement is that it provides usable, relevant information to front-line practitioners to inform their practice and their improvement efforts. The Participation Spreadsheet/Tracker that the UBC Network developed, although relatively simple, responded to a key need that schools and educators faced, particularly as their work shifted during the pandemic. The student participation data collected with this tool allowed teachers, school administrators, and network leaders to track progress toward meeting their revised aim of 80% of children participating in at least three learning activities each week. Additionally, most teachers found the data collection form easy to use (when internet connectivity was not an issue), and the summary table provided timely data that indicated which students school staff might need to follow up with regarding low participation. The hub team also embedded social processes and routines in network meetings to support collective learning around the data to improve outcomes across sites, which are discussed further below.

Insights from Using the Measure: Developing routines for data collection and network learning

In order for a measure to provide actionable information and guide learning across a network, it is important that clear routines regarding data collection are established. To support use of the Participation Spreadsheet/Tracker, the hub created a spreadsheet for each classroom in the network. Each week, teachers were asked to complete the spreadsheet by indicating each student’s participation. These spreadsheets were linked to DataStudio, which generated reports for individual classrooms, an entire district, and for all students in the UBC Network. Every Monday, the hub team shared weekly classroom- and school-level reports with teachers and school leaders (see Figure 3). School staff used this information to evaluate how they were doing and consider strategies to improve participation.

Figure 3. Participation report for one school in the network

This report displays the weekly participation data for a classroom in the network. In the top-left corner, the two circles on the first row report the number of students who have not participated in any activity from May to the date of the report. The two circles in the second row indicate the number of weeks the classroom has met their local participation goal. The bar chart displays the percentage of students who have engaged in each type of learning activity (asynchronous, on-site, or synchronous) for each week that month. The run chart at the bottom of the report shows the percentage of students who achieved the participation goal each week over the course of the school year. Image source: Screenshot from recorded interview on 5/23/22.

To support learning and improvement across the network, one person from each district was assigned as the point person focusing on the participation improvement effort. This participation/attendance manager received additional training from the hub regarding strategies to improve attendance and participation. They also led monthly meetings in their district to encourage school leaders to share instructional and leadership practices. During these meetings, the participation/attendance manager shared their district’s participation data and led school teams to analyze the data and share best practices. Based on these discussions, the teams then identified strategies to test using Plan-Do-Study-Act cycles in order to improve students’ participation in the different learning experiences. At the end of the year, representatives from all the districts in the network met with the hub team to discuss the participation data and to identify lessons learned.

Impact of Measurement for Improvement

As schools shifted to new models during the pandemic, many students in Chile and around the world experienced educational exclusion because of difficulty accessing remote learning activities. Utilizing the Participation Spreadsheet/Tracker allowed schools and classroom teachers in the UBC Network to identify children with low participation and implement targeted strategies to prevent them from experiencing further educational exclusion. Additionally, analyzing participation data at monthly district meetings promoted knowledge sharing between school teams and supported them in developing timely strategies that responded to local realities.

Based on the data recorded in the Participation Spreadsheet/Tracker, the UBC Network made significant progress toward achieving their aim from May to December 2021 (26 weeks). 75% of students participated in at least three learning activities per week in 22 out of the 26 weeks. They also reduced the percentage of students not participating in any learning activities each week from 16% in Week 1 to 0% in Week 26 (see Figure 4).

Figure 4. UBC Network Participation Data from May to December 2021

“Just by virtue of tracking this information, teachers were pressed to attend to an important aspect of their instruction that they may not have been thinking about.”

— Sola Takahashi, Senior Research Associate, Improvement Science, WestEd

Introduction

The Student Independent Writing Time measure is relatively simple and straightforward. It does what the name suggests — tracks the number of minutes students spend in independent writing. However, the power of this measure comes not just from the data collected but how it is used. It provides a strong example of how the social processes built around a measure can encourage users to attend to new information, challenge assumptions, and work collaboratively toward instructional improvement.

The Need for Measurement

The Regional Education Lab West (REL West) at WestEd worked with two schools in Reno, Nevada, as part of the K-6 Literacy Improvement Project to increase student achievement in writing. As they developed their theory of improvement, they identified four student learning experiences to support this aim: (1) engagement in writing, (2) collaboration with peers, (3) independent writing time (20-30 minutes daily), and (4) opportunities to reflect on and improve writing. Although students’ independent writing time is a critical classroom practice that paves the way for improvement in writing, the hub leadership team found that it was not consistently implemented at the two schools and identified it as an important measure to track.

The Measure: Student independent Writing Time

The team prepared a Google spreadsheet template that teachers could use to record the number of minutes students spend in independent writing time as well as the total number of minutes for the writing lesson.

As times are added to the spreadsheet, two charts are populated automatically, showing the total lesson time compared to the amount of time spent in independent writing and the time students spend writing each day (see Figures 1 and 2).

Figure 1. Total lesson time compared to independent writing time

Note. “Getting Ready” is the short form of “Getting Ready to Write,” and in this case can be understood as whole group instruction.

Figure 2. Time spent writing per day

Measures for improvement are intended to support learning within a system. The Google spreadsheet templates provide a simple way for teachers to record their data regarding student independent writing time each day. The charts generated in the spreadsheets also make it possible for teachers to analyze the data quickly, identify and implement changes in instruction, and determine whether the changes led to the improvement they were hoping to see. Additionally, because this measure is tied to one of the key levers in the theory of improvement, it can be used to refine and test the theory while allowing teachers to track improvement. The student independent writing time measure was part of a larger system of measures that evolved over time and helped the team learn about their theory of improvement.

Insights from Using the Measure: Social processes that support using data for improvement

In order to spur instructional improvements based on this data, the partnership team needed to build effective data analysis routines among teachers. Taking advantage of a professional learning structure that already existed in the school sites, the team regularly met with teachers during their grade-level professional learning community (PLC) meetings. Every grade-level team also had a “team lead” who received additional coaching about team facilitation and the inquiry process and met with other leads to share change ideas across grade-level teams.

To kick off the improvement project with grade-level teams, the research team focused on building trust with and among teachers. They utilized a discussion protocol in grade-level team meetings that prompted teachers to share their observations about what was happening in their teaching of writing. Teachers reflected on the successes and challenges they were currently experiencing and unpacked common challenges together.

After a few weeks of establishing these learning routines, the partnership team then introduced the idea of collecting independent writing time data, and teachers began recording their data on the provided template. It’s important to note that initially the data was not shared with other teachers in the group. This encouraged teachers to honestly report on their classroom practices, and many were surprised to discover that they actually gave students less time to write than they intended.

After trust was established within the group, teachers began sharing their data with one another. In grade-level meetings, the team utilized a “Learning Huddle Protocol” to reflect on their recent teaching experiences, examine the data, discuss challenges, and determine next steps. These huddles were not only critical for moving the improvement project forward but also encouraged a shift in mindsets in many teachers as they became more comfortable inquiring about their practice. Teachers started having open conversations about their writing instruction and many teachers left the meetings with change ideas to try.

By centering the conversations around the Student Independent Writing Time measure, one member of the partnership team shared that it helped teachers “get away from just their gut feeling about how things were going — pushing them to attend to certain features of the classroom that they might not have been attending to otherwise.” Partnership lead Kim Austin explains, “This process of collecting practice data not only made instruction visible, it made instructional practice visible over time. Seeing the graphics and visual displays of how long the lessons were taking — and how much writing time students were getting, or in many cases, not getting — raised the teachers’ awareness of the instructional decisions they were making every day and how these decisions were impacting their students’ learning opportunities.”

Impact of Measurement for Improvement

The Student Independent Writing Time measure enabled teachers to really take note of their instructional practices around writing. However, this recognition alone might not yield instructional improvement. The social processes and collaborative inquiry routines embedded within the grade-level team meetings built trust and promoted honest reflection as teachers made sense of the data together. These conversations led to collaborative problem solving and equipped teachers with ideas and strategies to improve writing instruction in their classrooms. While testing new ideas, teachers captured data along the way and had time to step back to observe their own practice. In addition to the overall improvement in student writing, perhaps most importantly, these collaborative inquiry routines, utilized alongside the measure, supported a more intentional and disciplined implementation of the curriculum and shifts in teachers’ mindsets around how to improve their practice.

“We’ve had a lot of success helping people transform [data from the state agency] to something that is more actionable and useful to the work they are trying to do.”

— Ben Sanoff, Director of Data Analytics for CARPE College Access Network

Introduction

Practical measures aren’t necessarily about collecting more data or designing something totally new. Most school systems have more data than they know what to do with. Instead, what if we took data that was already available and made it useful? The Financial Aid Application Completion measure does just that. Using extant data from the state education agency, this measure provides information to school staff that is timely and actionable, fueling their improvement efforts.

The Need for Measurement

The CARPE College Access Network,which includes over 30 high schools in Southern California, is focused on increasing the percentage of students who are Black, Latinx, indigenous, or from low-income backgrounds who apply, enroll, and ultimately go to colleges they are likely to graduate from. To meet this aim, the network identified three primary areas to focus on: 1) financial access, including the completion of financial aid applications; 2) college application and enrollment; and 3) decreasing “summer melt” (ensuring that students enroll in college once admitted). 
As the improvement effort was getting started, the network analytics team discovered that most schools lacked tools to track college applications and enrollment as well as summer melt. Collecting this information proved to be challenging for schools, because it required following up directly with students. To circumvent these logistical challenges, the network analytics team began to wonder, What data related to our theory of improvement is already available? This led them to data regarding the submission of college financial aid applications that was reported by the California Student Aid Commission (CSAC). The team decided to find a way to share this information back to the schools in their network so that it could inform their processes for supporting students.

The Measure: Financial Aid Application Completion

Without financial support, many students are less likely to attend college. This measure tracks a key lever in the network’s theory of improvement, and ensures that this potential barrier to college attendance is addressed. Based on data reported by the CSAC, the network analytics team developed an online dashboard for each school that displays the number and percentage of students who have completed financial aid applications. The dashboard also includes a run chart that shows the number of financial aid applications completed each week and as compared to the prior year (see Figure 1). Each school is asked to set a goal for the percentage of students they want to complete financial aid applications. Using this target, the dashboard displays the number of applications that are still needed and how many applications will need to be submitted each week (prior to the deadline) in order for the goal to be met. 

Figure 1. Sample run chart from the school dashboard displaying number of applications completed each week compared to the previous year

One of the key features of a measure for improvement is that it serves as a signal for actions users can take. To this end, the data collected using the Financial Aid Application Completion measure is organized and displayed in multiple formats to support different users engaged in the improvement effort. For example, school administrators might refer to the dashboard to identify the percentage of students who have already completed applications and the number of students completing applications each week to determine if the school is on track to meet its aim (see Figure 2). Based on this information, the administrator can assess their current processes and determine if new processes or strategies might be needed to promote further progress. Additionally, a school counselor can review the by-student data to determine which students have not completed their application and use that information to guide their  outreach to students and families. Having this information allows school staff to more effectively design processes to support students as they apply for financial aid, illustrating another feature of measurement for improvement: it is seen as valuable to the target users. 

Figure 2. Sample table from the school dashboard displaying the percentage of completed applications and progress toward aim

Insights from Using the Measure: Transforming extant data into actionable information

To develop this measure, the network analytics team started with determining what data to pull and where. Their first guiding question was: What do schools need? The answer was surprisingly simple: Schools needed to know which students had not yet completed financial aid applications.

Next, they asked: What is missing or needed from public data reported by the state? The state report only provided aggregate data regarding the number of financial aid applicants from a school and listed the names of students who had completed the application. However, what schools really needed to know was the names of students who had not completed the application, so that school staff could clearly identify the students to follow up with.

The third critical question was: How do we make the state data useful so that it can support our improvement efforts? The network analytics team developed a weekly routine for downloading the data from the state website and uploading it to the hub’s data system to populate the dashboard, which allows network leaders and school staff to easily monitor progress that teams are making toward their goal. Additionally, a data lead from each school uploads the by-student data file to the network’s database platform every other week (or provides the file to the hub to do so). This information is used to create a table for each school that includes the names of students who need to complete a financial aid form or who have an error that needs to be addressed. The hub also supports schools in determining which students to prioritize for follow-up and assigning tasks to appropriate staff.

Impact of Measurement for Improvement

Using the Financial Aid Application Completion measure has encouraged school teams to shift their mindset about how to best support students in their college journey. Instead of waiting to provide assistance to students who ask for it, college counseling teams review the data and take a more proactive approach by providing support for students who need it at that moment. Additionally, this measure effectively transforms state-level aggregate data into practical information that school teams can use to design interventions that increase college access for traditionally underrepresented students. According to CARPE College Access Network’s website, schools in their network have measurably increased college access and improved financial aid completion rates by 11 percentage points for schools that joined the network in 2019 and 18 percentage points for schools that joined in 2021.

Figure 3. Chart showing percentage of students who applied for financial aid utilizing the FAFSA or California Dream Act Application (CADAA) at network schools from 2018-2023

“This is a familiar problem… How do we formally assess and monitor the development of different teaching skills? We need to collect data from the front lines of practice in an efficient way.”

— Andrew Krumm, Assistant Professor, University of Michigan, and Chief Data Scientist for the SIMPL Collaborative

Introduction

Key characteristics of practical measures are that they should be minimally burdensome and relatively easy and quick to use. However, whether a measure is seen as “practical” depends somewhat on who you are asking. In particular, front-line practitioners are often very constrained for time and reluctant to take on “one more thing.” To collect essential data from practitioners to inform improvement efforts, it can help to get a little creative. The Society for Improving and Measuring Procedural Learning’s (SIMPL) performance assessment app (SIMPL OR) is an example of how technology can be used to extend the possibility of what is considered feasible when collecting data from busy front-line practitioners like attending, or supervising, surgeons.

The Need for Measurement

As a quality improvement collaborative focused on investigating and developing tools, curriculum, and policy to improve the training of physicians, SIMPL first addressed gathering data on surgical trainee performance over time. Most assessment in surgical training is too infrequent and does not capture the full range of procedures and learning experiences a trainee engages in. This gap highlighted a key area to address in order to improve overall trainee performance: increasing the data and feedback trainees received on their developing skills. Given the time constraints that both training and attending surgeons faced, the SIMPL Collaborative faced a critical and common challenge for those involved in change efforts focused on improving practice: How can we assess and provide feedback to trainees in a rapid, easy way?

The Measure: SIMPL OR (Smartphone-Based Performance Assessment App)

The team developed a smartphone-based formative assessment tool that consists of three questions. Using three separate rating scales, attendings rate the overall complexity of the case, amount of autonomy given to the trainee, and how well they performed the focal part of a procedure. If the supervisor would like to provide information beyond the three questions, he or she can also dictate feedback that the trainee can later listen to. Based on the data collected from evaluations, the measure provides simple data displays so that trainees can quickly see how many procedures they have completed and their performance, and it also allows trainees to review feedback from supervising physicians.

Figure 1. Screenshots of SIMPL OR app

Whether utilized in a hospital with training surgeons or in a school with beginning teachers, measures for improvement must be minimally burdensome and timely. SIMPL OR utilizes what most people have readily available — their phone — and provides a quick and simple way for trainees to request and review feedback. Supervisors are only required to complete three questions, which they can do on a short elevator ride or over lunch. This system allows trainees to receive feedback quickly and signals changes that they might make to improve their performance in future tasks. Another key feature of measures for improvement is that they should be perceived as valuable to the target users. Both training physicians and their supervisors are invested in seeing trainees develop key skills and improve performance over time. The SIMPL OR app creates opportunities for more frequent, regular feedback to be shared with trainees that can then be applied to their practice.

Unlike many performance evaluation and feedback systems, the SIMPL OR app provides timely information through a low-stakes approach. Many teacher evaluation systems, for example, rely on classroom observations that take place once or twice a year. Given that these evaluations are often tied to compensation or certification decisions, the lack of frequency creates a high-pressure environment for teachers and also reduces the usefulness of the feedback provided. Illustrating key characteristics of practical measures, the SIMPL OR app promotes the use of many brief, frequent evaluations to better understand a trainee’s performance and provide more actionable feedback.

Insights from Using the Measure: Embedding measures into the system and daily practice

Even with a relatively simple tool, a common hurdle to measurement in improvement efforts is actually getting people to use the measure. The SIMPL OR app is trainee driven — after completing a procedure, the trainee requests feedback from their supervisor in the application. As might be expected, the team found that while some trainees frequently requested feedback, others requested very little. This reluctance to seek feedback was often influenced by social and cultural norms in a particular setting.

The team noticed that hospitals that had higher engagement or utilization of the SIMPL OR app had found ways to embed the use of the tool into their daily practice. For example, training surgeons are required to log their procedures for certification purposes, but there is not typically a record of how well they performed the procedure. Some hospitals mandated using the SIMPL OR app for logging cases. This created a uniform system for tracking the procedures each trainee performed. Additionally, because logging a procedure in the SIMPL OR app triggers a request for feedback, requiring its use supports a cultural shift in the institution regarding the value of feedback and makes it part of its normal way of operating. The team identified this type of integration within institutional structures and processes as critical in their efforts to scale the use of the SIMPL OR app.

Impact of Measurement for Improvement

The work of improvers is never done, and collecting key data can point a team toward other areas to focus on. Beyond just increasing the feedback that trainees are receiving, the use of the SIMPL OR app has launched other initiatives aimed at improving the training of physicians and their resulting practice. For example, given the variability of ratings across supervising physicians utilizing the SIMPL OR app, the team began to wonder what was driving this variation. This has led to new improvement opportunities focused on the following questions: As raters, do we agree on what “good” performance is? What are we actually measuring? A clear understanding of the expectations for performance can help shape training experiences and the type of feedback that trainees receive. Additionally, a teaching and learning committee is selecting and formalizing strategies that supervising surgeons will do with trainees; they are using the SIMPL OR app as an outcome measure for their interventions.

“Sometimes this data is hard when it’s about you – the teacher. It’s much easier when it’s about the school. But when it’s about you, the teacher, there are emotions involved. One of the things we’ve learned… is to spend a little more time setting the groundwork for engaging in the data.”

— Jacobē Bell, Director, Teaching Matters NSI Network

Introduction

One feature of practical measurement that makes it distinct from measures that are used for other purposes is that it is “up close and personal” – measures are tied to the actual processes that are the focus of change efforts. This means that if you are trying to improve instruction, you’ll likely want to identify measures that will help you understand the effectiveness of specific classroom-level practices. However, in accountability-focused environments, practitioners can be apprehensive about collecting and sharing data that is so closely tied to their work. To counteract these sentiments, improvement leaders need to be intentional about creating an organizational culture and the associated social processes to foster an environment that encourages honest reflection and learning from data.

The Need for Measurement

The Teaching Matters Network for School Improvement (NSI) is a network of middle schools focused on integrating culturally responsive practices. Utilizing continuous improvement methods, the network’s aim is to increase the number of students who end 8th grade on track for high school and beyond. To meet this goal, the network is focusing on practices that build community, affirm identity, and deepen learning in order to improve literacy (see Figure 1).

Figure 1. Teaching Matters NSI Driver Diagram

The Teaching Matters NSI wanted to understand the extent to which such practices were enacted in classrooms from the students’ perspective. Incorporating student voice would be critical for helping the network as sess the impact of their changes in this area and make progress toward their overall aim.

The Measure: Inclusive Classrooms Survey

After considering a few different options, the network decided to utilize questions from a survey developed by Panorama Education – Panorama Student Survey. The Panorama is a nationally normed survey instrument focused on school climate, teaching and learning, relationships, and belonging. The Teaching Matters NSI selected Panorama because the vendor was already approved by the New York City Department of Education, reducing hurdles to implementing it in network schools. To ensure that this measure met their goals, network leaders selected survey items across six domains related to inclusive classroom practices. Network leaders deliberately chose questions focused on classrooms, not schools (e.g., rigorous teacher expectations instead of schoolwide expectations), and students are asked to report their perceptions of their language arts (ELA) teacher and classroom. 

Two key features of measurement for improvement are that the measure is 1) tied to a working theory of improvement and 2) specific to the processes that are the focus of change efforts. Utilizing the Panorama allows the Teaching Matters NSI to track the impact their efforts are having in fostering inclusive classroom practices. Additionally, Panorama provides for the delivery of the survey and displays an analysis of the results in a dashboard tailored to specific roles (e.g., teacher, school leader, and network leader). This reduces the burden on network and school staff to administer the measure and process data, which allows for results to be reported in a timely manner and frees up the hub team to invest in additional analyses, such as identifying bright spots. Finally, and perhaps most importantly, to ensure that data from the Panorama actually transforms practices within schools and classrooms, the network has developed social learning routines and resources that help network members learn from each other.

Insights from Using the Measure: Creating a culture that fosters learning from data 

The network intentionally creates a social environment that encourages honest reflection and learning from the Panorama data among teachers. Network leaders noted that because Panorama data is reported based on individual teachers and their classroom practices instead of an entire school, it can be difficult for teachers not to take this feedback personally regarding their instruction and how students perceive classroom culture. “It’s not saying that anyone is a bad teacher,” Network Director Jacobē Bell reflected on the challenge of using teacher-level data to drive improvement efforts. “How do you take the ‘but I work so hard’ out of it? Yes, you work hard, and you’re lovely, and this is what your kids are saying.”

Network leaders needed to shift the culture and lay the groundwork for teachers to collectively engage in data analysis by creating a supportive environment focused on two key questions: What are our students telling us? and What can we learn from each other? At the school level, teachers and school leaders work with coaches to analyze Panorama data using a protocol that helps teams determine actionable next steps and change ideas to support areas of need. To support learning across the network, the hub uses a P-chart model to analyze variation across each domain in order to discover bright spots and unearth best practices (see Figure 2). 

Figure 2. Sample P-chart for “Classroom Belonging” Domain

The hub team reviews these charts with coaches, focusing their conversations on specific schools or teachers that are getting positive results. The hub team visits identified schools and classrooms and interviews teachers to unearth what is working well. They capture these strategies in videos and blog posts that are shared on the network’s online platform and during convenings. (Check out this video and summary of a bright spot visit in one of the schools in the Teaching Matters NSI.) By integrating knowledge management practices focused on elevating effective strategies, the hub team hopes to accelerate learning across the network around best practices for building inclusive classrooms.

Impact of Measurement for Improvement

Using Panorama as a measure to evaluate the Teaching Matters NSI’s efforts to advance inclusive classroom practices encourages network members to keep students at the center of their work. Students’ perceptions of their classroom environment guide the creation of change ideas and prompt deep reflection from teachers on their practice. Use of this measure has also shifted mindsets that teachers in the network had around using data. As conversations related to Panorama data focus on learning and not on accountability or evaluation, The measure has fostered a culture that is supportive of improvement work. Teachers have moved beyond the initial apprehension they might have felt about sharing data about their teaching practices and classroom culture and are building mental muscles to meaningfully engage with the data and allow it to inform their practice.

“We were trying to move data to the center of our practice.”

“The approach feels very accessible…I am trying an alternate, countercultural approach”

—  Annie Wilhelm, Associate Professor of Mathematics Education, Department of Teaching and Learning, Southern Methodist University

Introduction

For a measure to be considered practical, it must be minimally burdensome to users and attend to the social processes of use that help activate a strong culture of improvement. While technical considerations are certainly important when developing and testing any practical measure, it is just as critical to build and nurture social routines that engage an improvement community of diverse expertise. It is also integral that we remain intentional and proactive about empowering all team members to stay present as active agents of change.

The Student Engagement Exit Ticket measure was developed out of such commitment to create an improvement-focused culture that enables a generative, trusting community of improvers to come together to openly share insights and ideate solutions through a collaborative design process. This rich practice of collective brainstorming not only allows for greater alignment among individuals focusing on a shared problem of practice but is also instrumental to designing a minimally burdensome, user-friendly measure that best serves the needs of front-line practitioners.

The Need for Measurement

The Student Engagement Exit Ticket measure is a simple survey designed to provide teachers with direct feedback on student engagement around specific problems of practice. Annie Wilhelm, a professor at Southern Methodist University, collaborated with Dallas Independent School District leaders and community members to develop a new STEM-focused PK-8 school. To support this work, the design team worked with a local elementary school to improve teacher learning around using data and continuous improvement methods in their practice. However, shortly after they began this work, COVID-related school closures shifted their focus to helping teachers better understand important classroom dynamics like student engagement during remote and hybrid learning. From student feedback data, the design team and teachers hoped to not only learn about what was working for whom and under what conditions, but also to identify critical areas of unmet need and use that data to inform changes in practice. Testing this measure in the context of the pandemic provided timely learning opportunities for teachers grappling with the adjustment to remote instruction.

The Measure: Student Engagement Exit Ticket

The Student Engagement Exit Ticket is a short survey that collects high-level data on student engagement with content. It is intended to be regularly administered at the end of class (see Figure 1) so students can communicate their needs to teachers by providing feedback and educators can take the next steps to adapt their instructional practices.

Figure 1. Screenshot of Student Engagement Problem of Practice Measure

When considering measures for improvement, the social routines and processes guiding how a measure is developed and used are just as critical as the data it produces. It is important to note the unique nature behind the development of this measure, which saw teachers taking a lead not just in the implementation but also in the design of the survey tool itself. The teacher-led design of this measure speaks to the team’s intentionality towards nurturing social learning routines that energize diverse stakeholders, particularly teachers as front-line practitioners, to actively drive and engage in the improvement process. Additionally, because the measure utilizes an online survey platform (Google Forms), the data is immediately available and teachers can easily review the results, thus providing actionable information that teachers can use to improve their instructional practices.

Further, to ensure that the measure was truly practical for teachers, the professional learning design team was committed to minimizing the burden of implementation on teachers – from creating a low-lift survey that is easy to incorporate into their daily work routines to facilitating the creation and data aggregation process for them. Following the data aggregation, the PD sessions provided a low-stakes, safe environment for data to be shared and for important peer-to-peer learning to occur.

Insights from Using the Measure: Co-creating measures with teachers

The Student Engagement Exit Ticket survey, though not currently in use, offers improvement practitioners important considerations for the development and implementation of practice-based measures. The collaborative design and use of this measure spotlights a countercultural, simple approach that is accessible and can alleviate the burden on practitioners while also elevating their voice in the development process. To develop this measure, Annie and several other collaborators met virtually with the school principal and 15-20 teachers to identify problems of practice they wanted to work on, ranging from the use of technology for instruction to relationship building with students. Based on shared problems of practice, teachers worked together on a virtual whiteboard (Jamboard) to develop look-fors and brainstorm statements they wanted students to say at the end of a lesson with respect to their focal problem areas.

Figure 2. Screenshot of Brainstorm Session Jamboard

Building on the various problems of practice and insights gathered from their brainstorming exercise, the professional learning design team generated multiple short surveys in English and Spanish using Google Forms. Using Google Forms, which was already a familiar tool to teachers, made it easier for the measure to be built into their practice. In so doing, the design team aimed to develop initial measures that could be iterated on as necessary, with hopes of making the measures as close to the teachers’ practice and needs as possible. The teachers then took these surveys to pilot with their students. Though they did not employ a formal protocol for data analysis, the team is aware of their blind spots and would like to use this routine again to revise the measures and to better understand if they can provide more information around variability. The entire group came back together for a PD session following their first testing cycle, where Annie and her team synthesized emergent findings from the data analysis in an effort to facilitate discussion among teachers and engage them in a collective sense-making process. The integration of research, professional, and improvement knowledge in this measure’s development also helped foster a sense of shared responsibility around the improvement aim and collective learnings. 

So, what key takeaways can practitioners glean from this measure? Collective engagement and robust social processes are necessary for maximizing the potential of a practical measure to accelerate change.

Impact of Measurement for Improvement

While the Student Engagement Exit Ticket is not currently in use and has yet to be iterated on, it provides a strong example of a counterculture approach to measure development, as front-line improvers took an active role in designing measures to support their learning and improvement efforts. This collaborative development process has the power to influence a shift in mindset about what it looks like to design and use measures for improvement. By expanding who is involved in measure development, this process also helps build collective will among front-line practitioners and drives their enthusiasm to improve their practice. At the core of improvement is a shared commitment to learning. Improvement demands the active, full engagement of teams and its success is predicated on the ability of a team or network to manage both the technical and human sides of change. This team implemented intentional practices to engage the diverse community of practitioners, researchers, and leaders in their collaborative effort, and we can learn from their best practices as we attend to the social routines that support the development of an improvement culture at large.