“During the pandemic it was very difficult for students to log onto classes that were synchronous because they were in parts of Chile that are very rural and they didn’t have access to the internet, so the foundation looked for different ways to measure how students were participating in learning.”

— Pablo Muñoz, Head of Continuous Quality Improvement and Evaluation for the UBC Improvement Network

Introduction

In order to be “practical,” a practical measure for improvement must take into account the specific context in which it is being used – the time, the place, and the situation. It must be relevant and provide data that help practitioners address the challenges they are currently facing. Effective measures are often born out of practitioners reflecting on questions like What do we need to know right now to do our job well? and What information will help us better support our students or families in the current context? These kinds of measures reflect an unspoken urgency around improving the target outcomes. The Participation Spreadsheet/Tracker was designed to address specific challenges related to student participation during the COVID-19 pandemic. It highlights the importance of creating measures that are responsive to the needs of front-line improvers while taking into account their context so that practitioners have access to the information they need to improve their practice and processes.

The Need for Measurement

The Un Buen Comienzo (UBC) Improvement Network, supported by Fundación Educacional Oportunidad, focuses on improving early childhood education in Chile. Since its founding in 2018, the UBC Network has sought to create a space for collaboration, professional development, and pedagogical innovation among its network members representing over 150 schools across two regions in Chile.

Prior to the COVID-19 pandemic, the network focused on improving language outcomes for first- and second-year students in Early Childhood and Care programs (NT1 and NT2), which includes pre-kindergarten- and kindergarten-aged children from 4 to 6 years old. One of the primary drivers identified to achieve this goal was reducing the percentage of students who were chronically absent during the school year – missing more than 10% of school days (see Figure 1). To support this effort, the network hub leveraged data that was submitted by schools to the Chilean Ministry of Education through the General Student Information System to develop a platform that allowed them to track absenteeism across schools and specific classrooms in their network.

Figure 1. UBC Network’s Driver Diagram (2018-19) with primary driver focusing on attendance

[Image source: 2022 Summit Poster: A new perspective of school attendance in the Un Buen Comienzo Network during the pandemic]

However, in 2020, as the COVID-19 pandemic dramatically altered the way schools were operating across Chile, the Ministry of Education no longer required schools to submit classroom attendance data. With the General Student Information System no longer in use, schools lacked a way to monitor student attendance and needed the ability to understand whether students were engaging in the new structures for remote learning. Schools needed better visibility regarding which students were falling through the cracks. 

This dramatic change in context prompted the UBC Network to shift its efforts and theory of improvement to focus on student participation (see Figure 2), and they developed a measure and new data-collection tools to monitor their progress. A key challenge the network encountered as they shifted their overall aim was to clearly define what was meant by “student participation” as opposed to “student attendance.” Most people share a common understanding of the term “attendance” as it relates to schooling – it is a measure that indicates how often a student comes to school or attends a specific class. However, the UBC Network recognized that this measure was no longer relevant as schools were implementing various forms of remote learning experiences. With this new perspective, the network turned its attention to “student participation,” which they defined as the interaction of students with the school through different learning activities such as worksheet packets, synchronous online class sessions, and face-to-face instruction. (These activities varied widely by school depending on the student population, school location (rural vs. city), and other factors). 

Figure 2. UBC Network’s Driver Diagram (2020-21) with aim focusing on student participation

[Image source: 2022 Summit Poster: A new perspective of school attendance in the Un Buen Comienzo Network during the pandemic]

The Measure: Participation Spreadsheet/Tracker

The network hub created a Participation Spreadsheet/Tracker (sample spreadsheet translated to English) that allowed teachers to enter the activities that each student engaged in each week. The Participation Spreadsheet/Tracker includes a tab that provides clear instructions regarding how teachers should categorize each learning activity (which was important given the broad range of activities at each site) and tabs for each month where teachers track each students’ participation. Under the data table for each month, the spreadsheet also includes a summary table that automatically displays which students met the participation target (three or more activities) each week.

The core purpose of a practical measure for improvement is that it provides usable, relevant information to front-line practitioners to inform their practice and their improvement efforts. The Participation Spreadsheet/Tracker that the UBC Network developed, although relatively simple, responded to a key need that schools and educators faced, particularly as their work shifted during the pandemic. The student participation data collected with this tool allowed teachers, school administrators, and network leaders to track progress toward meeting their revised aim of 80% of children participating in at least three learning activities each week. Additionally, most teachers found the data collection form easy to use (when internet connectivity was not an issue), and the summary table provided timely data that indicated which students school staff might need to follow up with regarding low participation. The hub team also embedded social processes and routines in network meetings to support collective learning around the data to improve outcomes across sites, which are discussed further below.

Insights from Using the Measure: Developing routines for data collection and network learning

In order for a measure to provide actionable information and guide learning across a network, it is important that clear routines regarding data collection are established. To support use of the Participation Spreadsheet/Tracker, the hub created a spreadsheet for each classroom in the network. Each week, teachers were asked to complete the spreadsheet by indicating each student’s participation. These spreadsheets were linked to DataStudio, which generated reports for individual classrooms, an entire district, and for all students in the UBC Network. Every Monday, the hub team shared weekly classroom- and school-level reports with teachers and school leaders (see Figure 3). School staff used this information to evaluate how they were doing and consider strategies to improve participation.

Figure 3. Participation report for one school in the network

This report displays the weekly participation data for a classroom in the network. In the top-left corner, the two circles on the first row report the number of students who have not participated in any activity from May to the date of the report. The two circles in the second row indicate the number of weeks the classroom has met their local participation goal. The bar chart displays the percentage of students who have engaged in each type of learning activity (asynchronous, on-site, or synchronous) for each week that month. The run chart at the bottom of the report shows the percentage of students who achieved the participation goal each week over the course of the school year. Image source: Screenshot from recorded interview on 5/23/22.

To support learning and improvement across the network, one person from each district was assigned as the point person focusing on the participation improvement effort. This participation/attendance manager received additional training from the hub regarding strategies to improve attendance and participation. They also led monthly meetings in their district to encourage school leaders to share instructional and leadership practices. During these meetings, the participation/attendance manager shared their district’s participation data and led school teams to analyze the data and share best practices. Based on these discussions, the teams then identified strategies to test using Plan-Do-Study-Act cycles in order to improve students’ participation in the different learning experiences. At the end of the year, representatives from all the districts in the network met with the hub team to discuss the participation data and to identify lessons learned.

Impact of Measurement for Improvement

As schools shifted to new models during the pandemic, many students in Chile and around the world experienced educational exclusion because of difficulty accessing remote learning activities. Utilizing the Participation Spreadsheet/Tracker allowed schools and classroom teachers in the UBC Network to identify children with low participation and implement targeted strategies to prevent them from experiencing further educational exclusion. Additionally, analyzing participation data at monthly district meetings promoted knowledge sharing between school teams and supported them in developing timely strategies that responded to local realities.

Based on the data recorded in the Participation Spreadsheet/Tracker, the UBC Network made significant progress toward achieving their aim from May to December 2021 (26 weeks). 75% of students participated in at least three learning activities per week in 22 out of the 26 weeks. They also reduced the percentage of students not participating in any learning activities each week from 16% in Week 1 to 0% in Week 26 (see Figure 4).

Figure 4. UBC Network Participation Data from May to December 2021

Society for Improving and Measuring Procedural Learning in the Operating Room (SIMPL OR) app as a Practical Measure

Characteristics of Practical Measures	Descriptions of the SIMPL OR App
Is closely tied to a theory of improvement	The SIMPL OR app is used to support multiple improvement efforts with a common aim to increase practice-readiness among surgery trainees. When it is used as a measure for the aim of “every trainee achieves practice-readiness on one specialty-defined index procedure within each postgraduate year,” observations within individuals are transformed into readiness scores using a prediction model. The readiness score is also used as an outcome measure for testing teaching and learning strategies. Another metric of interest is the number of times trainees request feedback. This metric is used as a PDSA measure to test change ideas designed to improve trainee engagement in feedback-seeking behaviors. (See network improvement project and article on readiness)
Provides actionable information to drive positive changes in practice	The SIMPL Collaborative learned from the users that the SIMPL OR app has face validity. In other words, it is not difficult for users to understand what gets measured. On the one hand, trainees can use the SIMPL OR app to reflect on their performance and to receive formative feedback from faculty, which can guide their efforts in improving their practices. On the other hand, faculty can use aggregated SIMPL OR app data to personalize their teaching and supervision in the operating room. Moreover, program directors can use the data to identify procedures that are particularly challenging and design change ideas to support faculty and trainees. Recently, to maximize the actionability of the data, the Collaborative has invested in optimizing and differentiating data visualizations.
Captures variability in performance	The SIMPL OR app captures variability in trainee performance across operating procedures over time. It sounds straightforward, but when implemented in practice, various implementation challenges may emerge and can add noise to the data — this can potentially make the results less interpretable and hence less actionable. For example, in programs with a smaller faculty size, trainees may not be able to gather feedback from multiple faculty members.
Demonstrates predictive validity	A statistical prediction model has been developed by the SIMPL Collaborative to make sense of observations. Once a data point is entered into the system, a readiness score is generated to estimate the likelihood of success in the next operation. The Collaborative is in the process of understanding how trainees and faculty make sense of readiness scores. In addition, the SIMPL OR app results are expected to predict end-of-rotation performance evaluation results. However, because the latter lacks variability, the Collaborative is planning on shifting focus to studying the predictive relationships between SIMPL OR app results and post-graduation/career outcomes.
Is minimally burdensome to users	Being a phone application makes the SIMPL OR app highly accessible and minimally burdensome. On top of that, the SIMPL OR app embraces user autonomy. Trainees can choose whether to request feedback after a case closes, while faculty can choose whether to accept a feedback request and if they are to dictate a qualitative response. This design helps both trainees and faculty perceive the use of the SIMPL OR app as something they want to do instead of something they have to do.
Functions within social processes that support improvement culture	The SIMPL OR app is used as an assessment for improvement, not an assessment for accountability. Trainees are allowed to request feedback as little as one time or as many times as needed to inform their learning. Given the voluntary nature of asking for feedback, the SIMPL Collaborative designed change ideas to encourage trainees to request more feedback.
Is reported on in a timely manner	Trainees are only required to be formally evaluated through direct observation twice a year. The SIMPL OR app makes the evaluation process simpler and thus encourages more frequent feedback. Once a case closes, the trainee can request feedback from the supervising surgeon. After the surgeon provides feedback by answering three questions on the app, the system will instantly notify the trainee about the survey responses.

Question on Practical Measures Inspired by the SIMPL OR App

Are practical measures synonymous with quick-and-dirty measures?
Practical measures like the SIMPL OR app can often be implemented swiftly and easily. Yet, the process of developing and refining practical measures is usually far from easy. “In diving into the technical assessment properties of [SIMPL OR], we have opened up interesting improvement opportunities,” said Dr. Andrew Krumm, a member of the SIMPL Collaborative. What contributes to a surgeon’s rating of trainee performance? The Collaborative found that an attending surgeon’s rating reflects not only trainee competence, but also a number of other factors, such as the rarity of the procedure and, most of all, the attending surgeon’s own perception of good performance — a factor that contributes a majority of the variance in attending surgeons’ ratings. To make sure that the SIMPL OR app provides accurate assessment of trainee performance to inform improvement efforts, the Collaborative tested change ideas (e.g., specifying key procedures to be assessed, retraining raters) to minimize unwanted sources of variation. Examining the psychometric properties of a measure, identifying problems, and testing potential solutions take expertise, time, and resources; however, these efforts allow improvers to be confident about the efficacy of change being measured using the app’s data.

Student Engagement Exit Ticket as a Practical Measure

Characteristics of Practical Measures	Descriptions of the Student Engagement Exit Ticket
Is closely tied to a theory of improvement	Although the Student Engagement Exit Ticket measure was not tied to any formal theory of improvement, it was in alignment with teachers’ hypotheses of how changes they made would lead to improvements. The measure is associated with a problem of practice (i.e., difficulty motivating students) shared by a group of teachers, and it assesses ideal outcomes identified by the group of teachers (e.g., students’ intrinsic interest in lessons). Teachers who created the measure together used it to test their individual change ideas and learned from one another.
Provides actionable information to drive positive changes in practice	Because the measure was primarily designed by teachers, the data collected using the survey could provide a window for teachers into aspects of their students’ engagement that they cared about most and worked hardest to improve. This should increase the likelihood of them acting on the data meaningfully and constructively.
Captures variability in performance	For the group of teacher-designers involved, the measure served as a common measurement for improvement. By examining variation in the common measure across classrooms, teachers could answer questions about what works, for whom, and under what conditions.
Demonstrates predictive validity	The measure was constructed in the midst of the COVID-19 pandemic to solve urgent problems of practice that arose as schools transitioned from in-person to remote learning. Hence, the predictive validity of the measure was not studied.  In the process of developing this measure, teachers provided look-fors related to their problem of practice and identified success indicators. Though not formally studied, this approach to the design enabled teachers to make informal predictions about variation in student engagement.
Is minimally burdensome to users	During the outbreak of the COVID-19 pandemic, students were asked to fill out many surveys. A lot of them experienced survey fatigue. To make it easier on students, the professional learning design team led by Annie Wilhelm, Associate Professor at the Southern Methodist University, kept the measure short – only two close-ended items were included. Also, the survey was built using Google Forms (a tool that the teachers and students had previous experience using) and was administered during class time. As a result, students should be less likely to perceive completing the measure as “one more thing” to do.
Functions within social processes that support improvement culture	The measure is a product of group ideation. As teachers brainstormed outcomes worth tracking, they engaged in collaborative visioning of what success could look like. Once a shared vision was established, teachers could then 1) backward-map the changes they would make individually to actualize the vision, 2) implement the changes, 3) collect data using the measure to learn about the implementation, and 4) participate in collective sense-making by going through a notice-and-wonder data protocol. All of these processes together transformed the measure from a conventional exit ticket to a learning tool for improvement.
Is reported on in a timely manner	Teachers who tested their change ideas using the measure received reports at a professional development session following testing. Such a delay might make it harder for teachers to make connections between student experience and practices they tried as part of their change ideas, especially if they did not also document those practices.

Question on Practical Measures Inspired by the Student Engagement Exit Ticket Measure

What can improvers do to make the design process of practical measures more practical?
It is not uncommon for improvers to find the process of developing practical measures laborious. Sometimes, even after a lot of hard work has been put into designing a practical measure and making it “rigorous,” those who are closest to the work, such as teachers, still find it difficult to see how using the measure can help them improve their practice, and the measure may end up sitting on the shelf.

One way to address this challenge is to involve practitioners early on in the design process of practical measures. Annie Wilhelm, who built the design process for the Student Engagement Exit Ticket, said, “I was trying to make the [design process] as close to [teachers’] practice and their needs as possible.” For Annie, it’s important for teachers to have a sense of agency over their own learning. Although the teacher-driven process that she crafted was “possibly wrong and definitely incomplete,” it “got something out there that would give [teachers] something to work with.”

Annie’s example demonstrates that when it comes to bringing in teachers to create practical measures, it is okay to start small. In addition, modeling and scaffolding might be needed to help teachers better understand not only the steps but also the value of the process. As their capacity builds, teachers will be more prepared to iterate on practical measures to optimize their rigor, impact, and practicality.

“Just by virtue of tracking this information, teachers were pressed to attend to an important aspect of their instruction that they may not have been thinking about.”

— Sola Takahashi, Senior Research Associate, Improvement Science, WestEd

Introduction

The Student Independent Writing Time measure is relatively simple and straightforward. It does what the name suggests — tracks the number of minutes students spend in independent writing. However, the power of this measure comes not just from the data collected but how it is used. It provides a strong example of how the social processes built around a measure can encourage users to attend to new information, challenge assumptions, and work collaboratively toward instructional improvement.

The Need for Measurement

The Regional Education Lab West (REL West) at WestEd worked with two schools in Reno, Nevada, as part of the K-6 Literacy Improvement Project to increase student achievement in writing. As they developed their theory of improvement, they identified four student learning experiences to support this aim: (1) engagement in writing, (2) collaboration with peers, (3) independent writing time (20-30 minutes daily), and (4) opportunities to reflect on and improve writing. Although students’ independent writing time is a critical classroom practice that paves the way for improvement in writing, the hub leadership team found that it was not consistently implemented at the two schools and identified it as an important measure to track.

The Measure: Student independent Writing Time

The team prepared a Google spreadsheet template that teachers could use to record the number of minutes students spend in independent writing time as well as the total number of minutes for the writing lesson.

As times are added to the spreadsheet, two charts are populated automatically, showing the total lesson time compared to the amount of time spent in independent writing and the time students spend writing each day (see Figures 1 and 2).

Figure 1. Total lesson time compared to independent writing time

Note. “Getting Ready” is the short form of “Getting Ready to Write,” and in this case can be understood as whole group instruction.

Figure 2. Time spent writing per day

Measures for improvement are intended to support learning within a system. The Google spreadsheet templates provide a simple way for teachers to record their data regarding student independent writing time each day. The charts generated in the spreadsheets also make it possible for teachers to analyze the data quickly, identify and implement changes in instruction, and determine whether the changes led to the improvement they were hoping to see. Additionally, because this measure is tied to one of the key levers in the theory of improvement, it can be used to refine and test the theory while allowing teachers to track improvement. The student independent writing time measure was part of a larger system of measures that evolved over time and helped the team learn about their theory of improvement.

Insights from Using the Measure: Social processes that support using data for improvement

In order to spur instructional improvements based on this data, the partnership team needed to build effective data analysis routines among teachers. Taking advantage of a professional learning structure that already existed in the school sites, the team regularly met with teachers during their grade-level professional learning community (PLC) meetings. Every grade-level team also had a “team lead” who received additional coaching about team facilitation and the inquiry process and met with other leads to share change ideas across grade-level teams.

To kick off the improvement project with grade-level teams, the research team focused on building trust with and among teachers. They utilized a discussion protocol in grade-level team meetings that prompted teachers to share their observations about what was happening in their teaching of writing. Teachers reflected on the successes and challenges they were currently experiencing and unpacked common challenges together.

After a few weeks of establishing these learning routines, the partnership team then introduced the idea of collecting independent writing time data, and teachers began recording their data on the provided template. It’s important to note that initially the data was not shared with other teachers in the group. This encouraged teachers to honestly report on their classroom practices, and many were surprised to discover that they actually gave students less time to write than they intended.

After trust was established within the group, teachers began sharing their data with one another. In grade-level meetings, the team utilized a “Learning Huddle Protocol” to reflect on their recent teaching experiences, examine the data, discuss challenges, and determine next steps. These huddles were not only critical for moving the improvement project forward but also encouraged a shift in mindsets in many teachers as they became more comfortable inquiring about their practice. Teachers started having open conversations about their writing instruction and many teachers left the meetings with change ideas to try.

By centering the conversations around the Student Independent Writing Time measure, one member of the partnership team shared that it helped teachers “get away from just their gut feeling about how things were going — pushing them to attend to certain features of the classroom that they might not have been attending to otherwise.” Partnership lead Kim Austin explains, “This process of collecting practice data not only made instruction visible, it made instructional practice visible over time. Seeing the graphics and visual displays of how long the lessons were taking — and how much writing time students were getting, or in many cases, not getting — raised the teachers’ awareness of the instructional decisions they were making every day and how these decisions were impacting their students’ learning opportunities.”

Impact of Measurement for Improvement

The Student Independent Writing Time measure enabled teachers to really take note of their instructional practices around writing. However, this recognition alone might not yield instructional improvement. The social processes and collaborative inquiry routines embedded within the grade-level team meetings built trust and promoted honest reflection as teachers made sense of the data together. These conversations led to collaborative problem solving and equipped teachers with ideas and strategies to improve writing instruction in their classrooms. While testing new ideas, teachers captured data along the way and had time to step back to observe their own practice. In addition to the overall improvement in student writing, perhaps most importantly, these collaborative inquiry routines, utilized alongside the measure, supported a more intentional and disciplined implementation of the curriculum and shifts in teachers’ mindsets around how to improve their practice.

“We’ve had a lot of success helping people transform [data from the state agency] to something that is more actionable and useful to the work they are trying to do.”

— Ben Sanoff, Director of Data Analytics for CARPE College Access Network

Introduction

Practical measures aren’t necessarily about collecting more data or designing something totally new. Most school systems have more data than they know what to do with. Instead, what if we took data that was already available and made it useful? The Financial Aid Application Completion measure does just that. Using extant data from the state education agency, this measure provides information to school staff that is timely and actionable, fueling their improvement efforts.

The Need for Measurement

The CARPE College Access Network,which includes over 30 high schools in Southern California, is focused on increasing the percentage of students who are Black, Latinx, indigenous, or from low-income backgrounds who apply, enroll, and ultimately go to colleges they are likely to graduate from. To meet this aim, the network identified three primary areas to focus on: 1) financial access, including the completion of financial aid applications; 2) college application and enrollment; and 3) decreasing “summer melt” (ensuring that students enroll in college once admitted). 
As the improvement effort was getting started, the network analytics team discovered that most schools lacked tools to track college applications and enrollment as well as summer melt. Collecting this information proved to be challenging for schools, because it required following up directly with students. To circumvent these logistical challenges, the network analytics team began to wonder, What data related to our theory of improvement is already available? This led them to data regarding the submission of college financial aid applications that was reported by the California Student Aid Commission (CSAC). The team decided to find a way to share this information back to the schools in their network so that it could inform their processes for supporting students.

The Measure: Financial Aid Application Completion

Without financial support, many students are less likely to attend college. This measure tracks a key lever in the network’s theory of improvement, and ensures that this potential barrier to college attendance is addressed. Based on data reported by the CSAC, the network analytics team developed an online dashboard for each school that displays the number and percentage of students who have completed financial aid applications. The dashboard also includes a run chart that shows the number of financial aid applications completed each week and as compared to the prior year (see Figure 1). Each school is asked to set a goal for the percentage of students they want to complete financial aid applications. Using this target, the dashboard displays the number of applications that are still needed and how many applications will need to be submitted each week (prior to the deadline) in order for the goal to be met. 

Figure 1. Sample run chart from the school dashboard displaying number of applications completed each week compared to the previous year

One of the key features of a measure for improvement is that it serves as a signal for actions users can take. To this end, the data collected using the Financial Aid Application Completion measure is organized and displayed in multiple formats to support different users engaged in the improvement effort. For example, school administrators might refer to the dashboard to identify the percentage of students who have already completed applications and the number of students completing applications each week to determine if the school is on track to meet its aim (see Figure 2). Based on this information, the administrator can assess their current processes and determine if new processes or strategies might be needed to promote further progress. Additionally, a school counselor can review the by-student data to determine which students have not completed their application and use that information to guide their  outreach to students and families. Having this information allows school staff to more effectively design processes to support students as they apply for financial aid, illustrating another feature of measurement for improvement: it is seen as valuable to the target users. 

Figure 2. Sample table from the school dashboard displaying the percentage of completed applications and progress toward aim

Insights from Using the Measure: Transforming extant data into actionable information

To develop this measure, the network analytics team started with determining what data to pull and where. Their first guiding question was: What do schools need? The answer was surprisingly simple: Schools needed to know which students had not yet completed financial aid applications.

Next, they asked: What is missing or needed from public data reported by the state? The state report only provided aggregate data regarding the number of financial aid applicants from a school and listed the names of students who had completed the application. However, what schools really needed to know was the names of students who had not completed the application, so that school staff could clearly identify the students to follow up with.

The third critical question was: How do we make the state data useful so that it can support our improvement efforts? The network analytics team developed a weekly routine for downloading the data from the state website and uploading it to the hub’s data system to populate the dashboard, which allows network leaders and school staff to easily monitor progress that teams are making toward their goal. Additionally, a data lead from each school uploads the by-student data file to the network’s database platform every other week (or provides the file to the hub to do so). This information is used to create a table for each school that includes the names of students who need to complete a financial aid form or who have an error that needs to be addressed. The hub also supports schools in determining which students to prioritize for follow-up and assigning tasks to appropriate staff.

Impact of Measurement for Improvement

Using the Financial Aid Application Completion measure has encouraged school teams to shift their mindset about how to best support students in their college journey. Instead of waiting to provide assistance to students who ask for it, college counseling teams review the data and take a more proactive approach by providing support for students who need it at that moment. Additionally, this measure effectively transforms state-level aggregate data into practical information that school teams can use to design interventions that increase college access for traditionally underrepresented students. According to CARPE College Access Network’s website, schools in their network have measurably increased college access and improved financial aid completion rates by 11 percentage points for schools that joined the network in 2019 and 18 percentage points for schools that joined in 2021.

Figure 3. Chart showing percentage of students who applied for financial aid utilizing the FAFSA or California Dream Act Application (CADAA) at network schools from 2018-2023

Student Independent Writing Time as a Practical Measure

Characteristics of Practical Measures	Descriptions of the Student Independent Writing Time Measure
Is closely tied to a theory of improvement	The Student Independent Writing Time measure — and other measures such as the Writing Conference Tracker — is a part of a system of measures developed to test a theory of improvement with an aim to increase student proficiency in writing and with drivers focusing on student learning opportunities and instructional practices. It is a driver measure that aligns specifically with the “write independently for 20-30 minutes every day” driver. To address this driver, teachers have created different change ideas, including planning for no more than two “teachable moments” during the direct instruction part of the lesson to save time for independent writing.
Provides actionable information to drive positive changes in practice	The measure provides information for teachers to answer the question, “How much independent writing time is happening during writing lessons?” Without the measure, teachers might gauge the impact of their change ideas by intuition. Through use of the measure, teachers can discern patterns of their instructional practice and make adaptations to ensure independent writing time happens consistently. In the Literacy Improvement Partnership, some teachers leveraged the measure results to fine-tune their work in their classrooms and observed increases in student independent writing.
Captures variability in performance	As teachers log students’ independent writing time, the data entered are used to populate run charts automatically. Each run chart captures variation in independent writing time within a classroom over time. A benefit of using run charts to display data is that it allows practitioners to distinguish between common cause variation (e.g., variation to be expected from a process) and special cause variation (e.g., variation induced by change ideas). If a teacher can pinpoint when a change is induced on a run chart, comparison of the data pattern pre- and post-change can be done to assess empirically whether the change is an improvement.
Demonstrates predictive validity	Although the Literacy Improvement Partnership team did not conduct an analysis of predictive validity of the measure, research from the field suggests that student writing time matters for students’ improved writing.1
Is minimally burdensome to users	Although the logging of independent writing time is not time-consuming (the process takes less than a few minutes), it is one more thing that teachers have to do. Teachers in the Literacy Improvement Partnership found the daily documentation burdensome when sustained for about two months. Using the measure in learning sprints that last for 4-5 weeks and are spaced apart was preferred.
Functions within social processes that support improvement culture	The teacher-report data is not used in teacher evaluation. Yet, because it is not anonymous, teachers doing the reporting may be concerned about who has access to the data. This can have implications on teacher willingness to be transparent about their practice. Efforts directed toward building a culture of social learning and a sense of psychological safety around innovation and experimentation should be prioritized to invigorate teacher reflection and collaborative inquiry. In the case of the Literacy Improvement Partnership, the hub supports schools in establishing a learning huddle structure to do just that.
Is reported on in a timely manner	Teachers are expected to log the independent writing time daily. Some teachers enter data into the designated Google Sheet after a writing lesson or at the end of a school day. Other teachers choose to document daily using paper and pencil and engage in online data entry once at the end of each week. In either case, they can use the automatically populated run charts to reflect on their practice any time they deem it valuable.

¹Graham, S., Bollinger, A., Booth Olson, C., D’Aoust, C., MacArthur, C., McCutchen, D., & Olinghouse, N.(2012).Teaching elementary school students to be effective writers: A practice guide (NCEE 2012- 4058). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/ wwc/publications_reviews.aspx#pubsearch.

Questions on Practical Measures Inspired by the Student Independent Writing Time Measure

What are the functions of practical measures in improvement beyond working as a tool to test a theory of improvement?
In the process of identifying a practical measure to test a change, practitioners discover along the way what the “must-have” and “nice-to-have” elements of the change are. Measurement in many ways shapes intervention. In some cases, measurement is even a major part of the intervention. Sola Takahashi, a researcher who led the development of the Student independent Writing measure, said in an interview, “The recording of the measure was a change idea in itself…Just by virtue of tracking this information, teachers were pressed to attend to an important aspect of their instruction that they may not have been thinking about.”

What contributes to the effective use of practical measures?
Although practitioners are increasingly trained to use data to inform practice (as shown by the increased adoption of data-driven instruction), routine reflection and learning from practice-based data may be hampered by suboptimal school and district context (e.g., a top-down decision-making culture). To maximize what practitioners get out of using practical measures to improve the student learning experience, system leaders should not only focus on building practitioner analytical capacity, but also invest in building coherent structures, processes, and norms that constitute a scientific professional learning system. The learning huddle used in the Literacy Improvement Partnership is an example of such an investment.

“This is a familiar problem… How do we formally assess and monitor the development of different teaching skills? We need to collect data from the front lines of practice in an efficient way.”

— Andrew Krumm, Assistant Professor, University of Michigan, and Chief Data Scientist for the SIMPL Collaborative

Introduction

Key characteristics of practical measures are that they should be minimally burdensome and relatively easy and quick to use. However, whether a measure is seen as “practical” depends somewhat on who you are asking. In particular, front-line practitioners are often very constrained for time and reluctant to take on “one more thing.” To collect essential data from practitioners to inform improvement efforts, it can help to get a little creative. The Society for Improving and Measuring Procedural Learning’s (SIMPL) performance assessment app (SIMPL OR) is an example of how technology can be used to extend the possibility of what is considered feasible when collecting data from busy front-line practitioners like attending, or supervising, surgeons.

The Need for Measurement

As a quality improvement collaborative focused on investigating and developing tools, curriculum, and policy to improve the training of physicians, SIMPL first addressed gathering data on surgical trainee performance over time. Most assessment in surgical training is too infrequent and does not capture the full range of procedures and learning experiences a trainee engages in. This gap highlighted a key area to address in order to improve overall trainee performance: increasing the data and feedback trainees received on their developing skills. Given the time constraints that both training and attending surgeons faced, the SIMPL Collaborative faced a critical and common challenge for those involved in change efforts focused on improving practice: How can we assess and provide feedback to trainees in a rapid, easy way?

The Measure: SIMPL OR (Smartphone-Based Performance Assessment App)

The team developed a smartphone-based formative assessment tool that consists of three questions. Using three separate rating scales, attendings rate the overall complexity of the case, amount of autonomy given to the trainee, and how well they performed the focal part of a procedure. If the supervisor would like to provide information beyond the three questions, he or she can also dictate feedback that the trainee can later listen to. Based on the data collected from evaluations, the measure provides simple data displays so that trainees can quickly see how many procedures they have completed and their performance, and it also allows trainees to review feedback from supervising physicians.

Figure 1. Screenshots of SIMPL OR app

Whether utilized in a hospital with training surgeons or in a school with beginning teachers, measures for improvement must be minimally burdensome and timely. SIMPL OR utilizes what most people have readily available — their phone — and provides a quick and simple way for trainees to request and review feedback. Supervisors are only required to complete three questions, which they can do on a short elevator ride or over lunch. This system allows trainees to receive feedback quickly and signals changes that they might make to improve their performance in future tasks. Another key feature of measures for improvement is that they should be perceived as valuable to the target users. Both training physicians and their supervisors are invested in seeing trainees develop key skills and improve performance over time. The SIMPL OR app creates opportunities for more frequent, regular feedback to be shared with trainees that can then be applied to their practice.

Unlike many performance evaluation and feedback systems, the SIMPL OR app provides timely information through a low-stakes approach. Many teacher evaluation systems, for example, rely on classroom observations that take place once or twice a year. Given that these evaluations are often tied to compensation or certification decisions, the lack of frequency creates a high-pressure environment for teachers and also reduces the usefulness of the feedback provided. Illustrating key characteristics of practical measures, the SIMPL OR app promotes the use of many brief, frequent evaluations to better understand a trainee’s performance and provide more actionable feedback.

Insights from Using the Measure: Embedding measures into the system and daily practice

Even with a relatively simple tool, a common hurdle to measurement in improvement efforts is actually getting people to use the measure. The SIMPL OR app is trainee driven — after completing a procedure, the trainee requests feedback from their supervisor in the application. As might be expected, the team found that while some trainees frequently requested feedback, others requested very little. This reluctance to seek feedback was often influenced by social and cultural norms in a particular setting.

The team noticed that hospitals that had higher engagement or utilization of the SIMPL OR app had found ways to embed the use of the tool into their daily practice. For example, training surgeons are required to log their procedures for certification purposes, but there is not typically a record of how well they performed the procedure. Some hospitals mandated using the SIMPL OR app for logging cases. This created a uniform system for tracking the procedures each trainee performed. Additionally, because logging a procedure in the SIMPL OR app triggers a request for feedback, requiring its use supports a cultural shift in the institution regarding the value of feedback and makes it part of its normal way of operating. The team identified this type of integration within institutional structures and processes as critical in their efforts to scale the use of the SIMPL OR app.

Impact of Measurement for Improvement

The work of improvers is never done, and collecting key data can point a team toward other areas to focus on. Beyond just increasing the feedback that trainees are receiving, the use of the SIMPL OR app has launched other initiatives aimed at improving the training of physicians and their resulting practice. For example, given the variability of ratings across supervising physicians utilizing the SIMPL OR app, the team began to wonder what was driving this variation. This has led to new improvement opportunities focused on the following questions: As raters, do we agree on what “good” performance is? What are we actually measuring? A clear understanding of the expectations for performance can help shape training experiences and the type of feedback that trainees receive. Additionally, a teaching and learning committee is selecting and formalizing strategies that supervising surgeons will do with trainees; they are using the SIMPL OR app as an outcome measure for their interventions.

“Sometimes this data is hard when it’s about you – the teacher. It’s much easier when it’s about the school. But when it’s about you, the teacher, there are emotions involved. One of the things we’ve learned… is to spend a little more time setting the groundwork for engaging in the data.”

— Jacobē Bell, Director, Teaching Matters NSI Network

Introduction

One feature of practical measurement that makes it distinct from measures that are used for other purposes is that it is “up close and personal” – measures are tied to the actual processes that are the focus of change efforts. This means that if you are trying to improve instruction, you’ll likely want to identify measures that will help you understand the effectiveness of specific classroom-level practices. However, in accountability-focused environments, practitioners can be apprehensive about collecting and sharing data that is so closely tied to their work. To counteract these sentiments, improvement leaders need to be intentional about creating an organizational culture and the associated social processes to foster an environment that encourages honest reflection and learning from data.

The Need for Measurement

The Teaching Matters Network for School Improvement (NSI) is a network of middle schools focused on integrating culturally responsive practices. Utilizing continuous improvement methods, the network’s aim is to increase the number of students who end 8th grade on track for high school and beyond. To meet this goal, the network is focusing on practices that build community, affirm identity, and deepen learning in order to improve literacy (see Figure 1).

Figure 1. Teaching Matters NSI Driver Diagram

The Teaching Matters NSI wanted to understand the extent to which such practices were enacted in classrooms from the students’ perspective. Incorporating student voice would be critical for helping the network as sess the impact of their changes in this area and make progress toward their overall aim.

The Measure: Inclusive Classrooms Survey

After considering a few different options, the network decided to utilize questions from a survey developed by Panorama Education – Panorama Student Survey. The Panorama is a nationally normed survey instrument focused on school climate, teaching and learning, relationships, and belonging. The Teaching Matters NSI selected Panorama because the vendor was already approved by the New York City Department of Education, reducing hurdles to implementing it in network schools. To ensure that this measure met their goals, network leaders selected survey items across six domains related to inclusive classroom practices. Network leaders deliberately chose questions focused on classrooms, not schools (e.g., rigorous teacher expectations instead of schoolwide expectations), and students are asked to report their perceptions of their language arts (ELA) teacher and classroom. 

Two key features of measurement for improvement are that the measure is 1) tied to a working theory of improvement and 2) specific to the processes that are the focus of change efforts. Utilizing the Panorama allows the Teaching Matters NSI to track the impact their efforts are having in fostering inclusive classroom practices. Additionally, Panorama provides for the delivery of the survey and displays an analysis of the results in a dashboard tailored to specific roles (e.g., teacher, school leader, and network leader). This reduces the burden on network and school staff to administer the measure and process data, which allows for results to be reported in a timely manner and frees up the hub team to invest in additional analyses, such as identifying bright spots. Finally, and perhaps most importantly, to ensure that data from the Panorama actually transforms practices within schools and classrooms, the network has developed social learning routines and resources that help network members learn from each other.

Insights from Using the Measure: Creating a culture that fosters learning from data 

The network intentionally creates a social environment that encourages honest reflection and learning from the Panorama data among teachers. Network leaders noted that because Panorama data is reported based on individual teachers and their classroom practices instead of an entire school, it can be difficult for teachers not to take this feedback personally regarding their instruction and how students perceive classroom culture. “It’s not saying that anyone is a bad teacher,” Network Director Jacobē Bell reflected on the challenge of using teacher-level data to drive improvement efforts. “How do you take the ‘but I work so hard’ out of it? Yes, you work hard, and you’re lovely, and this is what your kids are saying.”

Network leaders needed to shift the culture and lay the groundwork for teachers to collectively engage in data analysis by creating a supportive environment focused on two key questions: What are our students telling us? and What can we learn from each other? At the school level, teachers and school leaders work with coaches to analyze Panorama data using a protocol that helps teams determine actionable next steps and change ideas to support areas of need. To support learning across the network, the hub uses a P-chart model to analyze variation across each domain in order to discover bright spots and unearth best practices (see Figure 2). 

Figure 2. Sample P-chart for “Classroom Belonging” Domain

The hub team reviews these charts with coaches, focusing their conversations on specific schools or teachers that are getting positive results. The hub team visits identified schools and classrooms and interviews teachers to unearth what is working well. They capture these strategies in videos and blog posts that are shared on the network’s online platform and during convenings. (Check out this video and summary of a bright spot visit in one of the schools in the Teaching Matters NSI.) By integrating knowledge management practices focused on elevating effective strategies, the hub team hopes to accelerate learning across the network around best practices for building inclusive classrooms.

Impact of Measurement for Improvement

Using Panorama as a measure to evaluate the Teaching Matters NSI’s efforts to advance inclusive classroom practices encourages network members to keep students at the center of their work. Students’ perceptions of their classroom environment guide the creation of change ideas and prompt deep reflection from teachers on their practice. Use of this measure has also shifted mindsets that teachers in the network had around using data. As conversations related to Panorama data focus on learning and not on accountability or evaluation, The measure has fostered a culture that is supportive of improvement work. Teachers have moved beyond the initial apprehension they might have felt about sharing data about their teaching practices and classroom culture and are building mental muscles to meaningfully engage with the data and allow it to inform their practice.

Financial Aid Application Completion as a Practical Measure

Characteristics of Practical Measures	Descriptions of the Inclusive Classrooms Survey Measure
Is closely tied to a theory of improvement	The Financial Aid Application Completion measure is used as a driver measure to monitor shifts in financial access, which is one of the three drivers included in the CARPE College Access Network’s theory of improvement. The theory is that improvement in Free Application for Federal Student Aid (FAFSA) application completion rate will contribute to improvement in college enrollment rate (i.e., percentage of students enrolled in tertiary institutions where they are most likely to graduate).
Provides actionable information to drive positive changes in practice	By disaggregating the data by network, by school, by student — in some cases, by counselor and by targeted student group — the hub has transformed publicly available but hard-to-use data into accessible and actionable information for diverse stakeholders. It turns out that the aggregate dashboards are the most helpful for school leaders, such as assistant principals, to make data-driven decisions to create conditions that maximize the impact of change ideas targeting FAFSA application completion. Meanwhile, the by-student data is the most pivotal to the work of counseling team members, as they can use the information to prioritize and keep their efforts consistent with the Multi-Tiered System of Supports (MTSS) framework.
Captures variability in performance	By building dashboards, the CARPE hub provides means for schools to understand variation among student groups and among individuals, as well as variation over time (e.g., 2019-20 data plotted against 2020-21 data). In addition, the CARPE hub visualizes the data aggregated by school using 1) small multiples ordered by the number of students in each school and 2) frequency-table-sorted based on percentage of application completers. These visuals capture variation across schools and allow the hub to be agile in strategizing coaching plans to meet the schools where they are at.
Demonstrates predictive validity	The CARPE hub has built dashboards for both Financial Aid Application Completion and Verified College Enrollment (data pulled from the National Student Clearinghouse) and has examined how changes in one are related to changes in the other. The hub found that among schools that had the largest increases in FAFSA application completion in 2018-19, there were increases in college enrollment — more at two-year colleges than four-year institutions. This finding was promising, but given that not as much variance in college enrollment was accounted for by FAFSA application completion as expected, the hub concluded that focusing solely on FAFSA application completion was not enough to achieve the network’s aim. Therefore, the hub expanded the improvement work to other drivers like college application completion.
Is minimally burdensome to users	One of many advantages of using extant measures like the Financial Aid Application Completion measure is that users do not have to spend extra time on collecting and documenting data. The time saved can be used in implementing social learning routines (e.g., data huddles) that facilitate knowledge consolidation and cross-pollination.
Functions within social processes that support improvement culture	The CARPE hub encourages the schools to interact with the Financial Aid Application Completion data system at least bi-weekly so the measure is more of a tool for learning and less of a tool for evaluation. This aligns with the hub’s emphasis on building a network culture that prioritizes measurement for improvement over measurement for accountability. All data displays and routines are set up in ways that facilitate collaboration and not competition.
Is reported on in a timely manner	The by-network and by-school data are updated weekly by the CARPE hub while the by-student data is updated bi-weekly by a data team lead from each school. The processes are highly standardized, and the frequency of data pulls may increase during critical periods, particularly in October and from January to March, as it is important to both start strong and end strong. Timely information can be provided to improvers because the hub invests time and resources not only in designing and testing data infrastructure and routines but also in building the capacity of network members to organize, visualize, and understand data so they can up their data game when the time calls for it.

Question on Practical Measures Inspired by the Financial Aid Application Completion Measure

How might we deal with a lack of consensus over how something should be measured?
The improvement principle “make the work user-centered” is applicable here. For the CARPE College Access Network, the operational definition of the primary driver, financial access, is FAFSA completion, CalGrant awardance, and paying for college.

With such a definition, one would expect the measures of the driver to be straightforward. However, as the CARPE hub members dove into calculating the FAFSA completion rate, they realized that there was little agreement over which denominator should be used. Should it be the total number of senior students enrolled in a school? Or, should it only count the number of senior students who have expressed interest in going to college?

To address this challenge, the CARPE hub encouraged schools to submit their own calculations. The hub then used that data to calibrate the data pulled from the California Student Aid Commission Race to Submit Dashboard and the Department of Education websites. The advantage of using an approach like this is that it could potentially help elevate user voice and build trust in the process and the data, which is crucial to continuous improvement.

As the school-reported FAFSA completion rates started to go up, the CARPE hub needed to address another key improvement question: Were those changes really improvements? After auditing the changes using denominators consistent with publicly available enrollment data, the CARPE hub could confidently conclude that the changes they were observing were real improvements and not merely changes in the school-reported measure. 

Participation Measure as a Practical Measure

Characteristics of Practical Measures	Descriptions of the Participation Measure
Is closely tied to a theory of improvement	Following the increase in educational exclusion during the pandemic, the Un Buen Comienzo (UBC) Improvement Network shifted its focus from attendance to participation and developed an aim for 80% of preschool and kindergarten children participating in at least three learning activities each week. The Participation Spreadsheet/Tracker is a tool to track a leading outcome measure. It monitors student engagement in learning experiences and helps teachers, school administrators, and network leaders identify individual students who are falling behind or at risk of not finishing successfully so they can develop personalized interventions for them. This measure is tied to the network’s theory of improvement, and tracking student engagement in learning activities supports their efforts to prevent students from experiencing further educational exclusion.
Provides actionable information to drive positive changes in practice	This measure relies on teachers to collect weekly data on each student’s participation. Using this information, the hub team generates weekly classroom- and school-level reports to share with teachers and school leaders. School teams can then review the disaggregated data to better understand patterns in participation and discern which classroom practices are helping enable student engagement and which are hindering engagement. The participation data are also shared in monthly district meetings, led by trained participation/attendance managers, to engage different school teams in collective sense-making, surface bright or blind spots, and identify timely strategies for improving participation.
Captures variability in performance	Once teachers record the student participation data, the Participation Spreadsheet/Tracker automatically feeds and produces weekly Data Studio reports that record student participation rates across classrooms, schools, districts, and the entire network over time. The data is used to populate a run chart, included in the weekly reports shared with teachers, that monitors the percentage of students who have achieved the participation goal each week and those with a 0% participation rate. This visualization tool captures the different participation rates by school and classroom so teachers and school leaders can then use the available data to attend to the variability in student performance.
Demonstrates predictive validity	According to the UBC Network’s theory of improvement, tracking student participation through a more holistic approach focused on engagement in different types of learning activities (synchronous, asynchronous, or onsite) can help school teams prevent educational exclusion (children with 0% weekly participation) and attain their goal of 80% of children participating in three learning activities a week. Additionally, the student participation data could function like an on-track indicator. While related outcome data is not available, we would expect the participation data to predict other variables like course completion and academic achievement.
Is minimally burdensome to users	For most teachers, recording participation on the prepared Google Sheet was easy and not very time consuming. However, some teachers were less familiar with using Google Apps and needed initial support as they transitioned from recording participation/attendance in the SIGE database to utilizing this new tool. After completing the Tracker/Spreadsheet, weekly data reports are automatically generated and sent to teachers for review. This process of collecting and documenting data was therefore systematized for teachers and integrated into their daily routines.
Functions within social processes that support improvement culture	The UBC Network had strong structures and processes in place to support peer-to-peer learning around their improvement effort. Monthly meetings were held in each district to share the best pedagogical and leadership practices. These meetings were facilitated by the district’s assigned attendance manager who was trained by the Network in strategies for improving attendance and participation. During these meetings, the attendance managers shared the district participation results, analyzed the data, and provided the team an opportunity to engage in collective sense-making. During these monthly meetings, teams were able to come together in a safe, low-stakes environment to study and learn from the participation data so they could improve their outcomes. In addition, representatives from each district met with the hub team to hear from each other and identify lessons learned.
Is reported on in a timely manner	Teachers were expected to log student participation data into the Spreadsheet/Tracker each week. On the following Monday, weekly data reports were made available for school teams, which then used that data to evaluate how they were doing and identified students at risk of educational exclusion. The monthly district meetings, in particular, provided a structure for the team to make timely decisions based on their data that directly responded to the needs of their local contexts.

Question on Practical Measures Inspired by the Participation Measure

How do you maintain standardization across the network, while ensuring local needs are sufficiently addressed?
A core improvement principle insists that “variation in performance is the core problem to address.” This process requires us to center data, elevate different voices, and be disciplined about standard work processes in order to ensure that the change idea can be carried out reliably by various people, even working under different conditions. To do so, we must use practice-based evidence to inform standard work processes.

Across the UBC network, 14 districts shared this participation measure and data collection tool. What was unique about this process is that the network did not assign the learning activities for participating districts. Instead, they empowered the schools to select the activities that were most suitable for their students given the available resources and unique circumstances of the communities they served, and provided guidance to ensure quality learning experiences were implemented as needed. To support the data collection and use of the participation measure, the network did define key terms related to participation (synchronous, asynchronous, etc.), so that data was recorded consistently across sites. This approach also allowed network leaders to analyze participation data across districts and build processes to support the overall improvement effort.