{"id":639,"date":"2025-08-01T12:00:58","date_gmt":"2025-08-01T19:00:58","guid":{"rendered":"https:\/\/carnegie25live.wpenginepowered.com\/?p=639"},"modified":"2025-12-16T10:58:15","modified_gmt":"2025-12-16T18:58:15","slug":"simpl-or","status":"publish","type":"post","link":"https:\/\/www.carnegiefoundation.org\/simpl-or\/","title":{"rendered":"SIMPL OR"},"content":{"rendered":"
\n
\n\n

Society for Improving and Measuring Procedural Learning in the Operating Room (SIMPL OR) app as a Practical Measure<\/strong><\/strong><\/h2>\n\n\n\n
<\/div>
<\/div>
<\/div><\/div>\n\n\n\n
Characteristics of Practical Measures<\/strong><\/th>Descriptions of the SIMPL OR App<\/strong><\/th><\/tr><\/thead>
Is closely tied to a theory of improvement<\/td>The SIMPL OR app is used to support multiple improvement efforts with a common aim to increase practice-readiness among surgery trainees. When it is used as a measure for the aim of \u201cevery trainee achieves practice-readiness on one specialty-defined index procedure within each postgraduate year,\u201d observations within individuals are transformed into readiness scores using a prediction model. The readiness score is also used as an outcome measure for testing teaching and learning strategies. Another metric of interest is the number of times trainees request feedback. This metric is used as a PDSA measure to test change ideas designed to improve trainee engagement in feedback-seeking behaviors. (See network improvement project<\/a> and article on readiness<\/a>)<\/td><\/tr>
Provides actionable information to drive positive changes in practice<\/td>The SIMPL Collaborative learned from the users that the SIMPL OR app has face validity. In other words, it is not difficult for users to understand what gets measured. On the one hand, trainees can use the SIMPL OR app to reflect on their performance and to receive formative feedback from faculty, which can guide their efforts in improving their practices. On the other hand, faculty can use aggregated SIMPL OR app data to personalize their teaching and supervision in the operating room. Moreover, program directors can use the data to identify procedures that are particularly challenging and design change ideas to support faculty and trainees. Recently, to maximize the actionability of the data, the Collaborative has invested in optimizing and differentiating data visualizations.<\/td><\/tr>
Captures variability in performance<\/td>The SIMPL OR app captures variability in trainee performance across operating procedures over time. It sounds straightforward, but when implemented in practice, various implementation challenges may emerge and can add noise to the data \u2014 this can potentially make the results less interpretable and hence less actionable. For example, in programs with a smaller faculty size, trainees may not be able to gather feedback from multiple faculty members.<\/td><\/tr>
Demonstrates predictive validity<\/td>A statistical prediction model has been developed by the SIMPL Collaborative to make sense of observations. Once a data point is entered into the system, a readiness score is generated to estimate the likelihood of success in the next operation. The Collaborative is in the process of understanding how trainees and faculty make sense of readiness scores. In addition, the SIMPL OR app results are expected to predict end-of-rotation performance evaluation results. However, because the latter lacks variability, the Collaborative is planning on shifting focus to studying the predictive relationships between SIMPL OR app results and post-graduation\/career outcomes.<\/td><\/tr>
Is minimally burdensome to users<\/td>Being a phone application makes the SIMPL OR app highly accessible and minimally burdensome. On top of that, the SIMPL OR app embraces user autonomy. Trainees can choose whether to request feedback after a case closes, while faculty can choose whether to accept a feedback request and if they are to dictate a qualitative response. This design helps both trainees and faculty perceive the use of the SIMPL OR app as something they want to do instead of something they have to do.<\/td><\/tr>
Functions within social processes that support improvement culture<\/td>The SIMPL OR app is used as an assessment for improvement, not an assessment for accountability. Trainees are allowed to request feedback as little as one time or as many times as needed to inform their learning. Given the voluntary nature of asking for feedback, the SIMPL Collaborative designed change ideas to encourage trainees to request more feedback.<\/td><\/tr>
Is reported on in a timely manner<\/td>Trainees are only required to be formally evaluated through direct observation twice a year. The SIMPL OR app makes the evaluation process simpler and thus encourages more frequent feedback. Once a case closes, the trainee can request feedback from the supervising surgeon. After the surgeon provides feedback by answering three questions on the app, the system will instantly notify the trainee about the survey responses.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
<\/div>
<\/div>
<\/div><\/div>\n\n\n\n

Question on Practical Measures Inspired by the SIMPL OR App<\/strong><\/h2>\n\n\n\n

Are practical measures synonymous with quick-and-dirty measures?<\/strong>
Practical measures like the SIMPL OR app can often be implemented swiftly and easily. Yet, the process of developing and refining practical measures is usually far from easy. \u201cIn diving into the technical assessment properties of [SIMPL OR], we have opened up interesting improvement opportunities,\u201d said Dr. Andrew Krumm, a member of the SIMPL Collaborative. What contributes to a surgeon\u2019s rating of trainee performance? The Collaborative found that an attending surgeon\u2019s rating reflects not only trainee competence, but also a number of other factors, such as the rarity of the procedure and, most of all, the attending surgeon\u2019s own perception of good performance \u2014 a factor that contributes a majority of the variance in attending surgeons\u2019 ratings. To make sure that the SIMPL OR app provides accurate assessment of trainee performance to inform improvement efforts, the Collaborative tested change ideas (e.g., specifying key procedures to be assessed, retraining raters) to minimize unwanted sources of variation. Examining the psychometric properties of a measure, identifying problems, and testing potential solutions take expertise, time, and resources; however, these efforts allow improvers to be confident about the efficacy of change being measured using the app\u2019s data.<\/p>\n\n<\/div>\n<\/div>\n\n\n

<\/div>
<\/div>
<\/div><\/div>\n\n\n
\n\n
<\/div>
<\/div>
<\/div><\/div>\n\n\n
\n\n

You Might Like<\/h2>\n\n\n
\n
\n\n
\n
<\/a>\n
\n
\"\"<\/figure>\n<\/div>\n\n
\n

Carnegie Foundation and ETS Release Skills Progressions for Collaboration, Communication and Critical Thinking\u00a0<\/h2>\n\n

Read more<\/a><\/p>\n<\/div>\n<\/div><\/div>\n<\/div>\n\n<\/div><\/div>\n\n

\n\n
\n
<\/a>\n
\n
\"\"<\/figure>\n<\/div>\n\n
\n

Rising Demand for Career Education Prompts College Board to Expand Its Footprint<\/h2>\n\n

Read more<\/a><\/p>\n<\/div>\n<\/div><\/div>\n<\/div>\n\n<\/div><\/div>\n\n

\n\n
\n
<\/a>\n
\n
\"\"<\/figure>\n<\/div>\n\n
\n

Opportunity Colleges and Universities Series: University of Illinois Chicago Profile<\/h2>\n\n

Read more<\/a><\/p>\n<\/div>\n<\/div><\/div>\n<\/div>\n\n<\/div><\/div>\n\n

\n\n
\n
<\/a>\n
\n
\"\"<\/figure>\n<\/div>\n\n
\n

Podcast: The Next 50 Years of Higher Ed \u2013 What Leaders Need to Know<\/h2>\n\n

Read more<\/a><\/p>\n<\/div>\n<\/div><\/div>\n<\/div>\n\n<\/div><\/div>\n<\/div>\n<\/div>\n\n\n

<\/div>
<\/div>
<\/div><\/div>\n\n<\/div>","protected":false},"excerpt":{"rendered":"

Society for Improving and Measuring Procedural Learning in the Operating Room (SIMPL OR) app as a Practical Measure Characteristics of … Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"ghostkit_customizer_options":"","ghostkit_custom_css":"","ghostkit_custom_js_head":"","ghostkit_custom_js_foot":"","ghostkit_typography":"","footnotes":""},"categories":[8],"tags":[9],"client_controls":[],"class_list":["post-639","post","type-post","status-publish","format-standard","hentry","category-practical-measurement-for-improvement","tag-technical-paper"],"acf":[],"yoast_head":"\nSIMPL OR - Carnegie Foundation for the Advancement of Teaching<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.carnegiefoundation.org\/simpl-or\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"SIMPL OR - Carnegie Foundation for the Advancement of Teaching\" \/>\n<meta property=\"og:description\" content=\"Society for Improving and Measuring Procedural Learning in the Operating Room (SIMPL OR) app as a Practical Measure Characteristics of ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.carnegiefoundation.org\/simpl-or\/\" \/>\n<meta property=\"og:site_name\" content=\"Carnegie Foundation for the Advancement of Teaching\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/carnegie.foundation\/\" \/>\n<meta property=\"article:published_time\" content=\"2025-08-01T19:00:58+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-12-16T18:58:15+00:00\" \/>\n<meta name=\"author\" content=\"DerekC\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@CarnegieFdn\" \/>\n<meta name=\"twitter:site\" content=\"@CarnegieFdn\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DerekC\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.carnegiefoundation.org\/simpl-or\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.carnegiefoundation.org\/simpl-or\/\"},\"author\":{\"name\":\"DerekC\",\"@id\":\"https:\/\/www.carnegiefoundation.org\/#\/schema\/person\/2c01d29bdf9103121a2c02b3b5a30fcd\"},\"headline\":\"SIMPL OR\",\"datePublished\":\"2025-08-01T19:00:58+00:00\",\"dateModified\":\"2025-12-16T18:58:15+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.carnegiefoundation.org\/simpl-or\/\"},\"wordCount\":899,\"publisher\":{\"@id\":\"https:\/\/www.carnegiefoundation.org\/#organization\"},\"keywords\":[\"Technical Paper\"],\"articleSection\":[\"Practical Measurement for Improvement\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.carnegiefoundation.org\/simpl-or\/\",\"url\":\"https:\/\/www.carnegiefoundation.org\/simpl-or\/\",\"name\":\"SIMPL OR - Carnegie Foundation for the Advancement of Teaching\",\"isPartOf\":{\"@id\":\"https:\/\/www.carnegiefoundation.org\/#website\"},\"datePublished\":\"2025-08-01T19:00:58+00:00\",\"dateModified\":\"2025-12-16T18:58:15+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.carnegiefoundation.org\/simpl-or\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.carnegiefoundation.org\/simpl-or\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.carnegiefoundation.org\/simpl-or\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.carnegiefoundation.org\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"SIMPL OR\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.carnegiefoundation.org\/#website\",\"url\":\"https:\/\/www.carnegiefoundation.org\/\",\"name\":\"Carnegie Foundation for the Advancement of Teaching\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.carnegiefoundation.org\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.carnegiefoundation.org\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.carnegiefoundation.org\/#organization\",\"name\":\"Carnegie Foundation for the Advancement of Teaching\",\"url\":\"https:\/\/www.carnegiefoundation.org\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.carnegiefoundation.org\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.carnegiefoundation.org\/wp-content\/uploads\/2025\/12\/Logo-Edit-MC.svg\",\"contentUrl\":\"https:\/\/www.carnegiefoundation.org\/wp-content\/uploads\/2025\/12\/Logo-Edit-MC.svg\",\"width\":249,\"height\":46,\"caption\":\"Carnegie Foundation for the Advancement of Teaching\"},\"image\":{\"@id\":\"https:\/\/www.carnegiefoundation.org\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/carnegie.foundation\/\",\"https:\/\/x.com\/CarnegieFdn\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.carnegiefoundation.org\/#\/schema\/person\/2c01d29bdf9103121a2c02b3b5a30fcd\",\"name\":\"DerekC\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.carnegiefoundation.org\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/569f1fc418ff9e5f872a030fc28b67243e00ad2d18ad9f8438675ee99b35d02c?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/569f1fc418ff9e5f872a030fc28b67243e00ad2d18ad9f8438675ee99b35d02c?s=96&d=mm&r=g\",\"caption\":\"DerekC\"},\"url\":\"https:\/\/www.carnegiefoundation.org\/author\/derekc\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"SIMPL OR - Carnegie Foundation for the Advancement of Teaching","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.carnegiefoundation.org\/simpl-or\/","og_locale":"en_US","og_type":"article","og_title":"SIMPL OR - Carnegie Foundation for the Advancement of Teaching","og_description":"Society for Improving and Measuring Procedural Learning in the Operating Room (SIMPL OR) app as a Practical Measure Characteristics of ... Read more","og_url":"https:\/\/www.carnegiefoundation.org\/simpl-or\/","og_site_name":"Carnegie Foundation for the Advancement of Teaching","article_publisher":"https:\/\/www.facebook.com\/carnegie.foundation\/","article_published_time":"2025-08-01T19:00:58+00:00","article_modified_time":"2025-12-16T18:58:15+00:00","author":"DerekC","twitter_card":"summary_large_image","twitter_creator":"@CarnegieFdn","twitter_site":"@CarnegieFdn","twitter_misc":{"Written by":"DerekC","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.carnegiefoundation.org\/simpl-or\/#article","isPartOf":{"@id":"https:\/\/www.carnegiefoundation.org\/simpl-or\/"},"author":{"name":"DerekC","@id":"https:\/\/www.carnegiefoundation.org\/#\/schema\/person\/2c01d29bdf9103121a2c02b3b5a30fcd"},"headline":"SIMPL OR","datePublished":"2025-08-01T19:00:58+00:00","dateModified":"2025-12-16T18:58:15+00:00","mainEntityOfPage":{"@id":"https:\/\/www.carnegiefoundation.org\/simpl-or\/"},"wordCount":899,"publisher":{"@id":"https:\/\/www.carnegiefoundation.org\/#organization"},"keywords":["Technical Paper"],"articleSection":["Practical Measurement for Improvement"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.carnegiefoundation.org\/simpl-or\/","url":"https:\/\/www.carnegiefoundation.org\/simpl-or\/","name":"SIMPL OR - Carnegie Foundation for the Advancement of Teaching","isPartOf":{"@id":"https:\/\/www.carnegiefoundation.org\/#website"},"datePublished":"2025-08-01T19:00:58+00:00","dateModified":"2025-12-16T18:58:15+00:00","breadcrumb":{"@id":"https:\/\/www.carnegiefoundation.org\/simpl-or\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.carnegiefoundation.org\/simpl-or\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.carnegiefoundation.org\/simpl-or\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.carnegiefoundation.org\/"},{"@type":"ListItem","position":2,"name":"SIMPL OR"}]},{"@type":"WebSite","@id":"https:\/\/www.carnegiefoundation.org\/#website","url":"https:\/\/www.carnegiefoundation.org\/","name":"Carnegie Foundation for the Advancement of Teaching","description":"","publisher":{"@id":"https:\/\/www.carnegiefoundation.org\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.carnegiefoundation.org\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.carnegiefoundation.org\/#organization","name":"Carnegie Foundation for the Advancement of Teaching","url":"https:\/\/www.carnegiefoundation.org\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.carnegiefoundation.org\/#\/schema\/logo\/image\/","url":"https:\/\/www.carnegiefoundation.org\/wp-content\/uploads\/2025\/12\/Logo-Edit-MC.svg","contentUrl":"https:\/\/www.carnegiefoundation.org\/wp-content\/uploads\/2025\/12\/Logo-Edit-MC.svg","width":249,"height":46,"caption":"Carnegie Foundation for the Advancement of Teaching"},"image":{"@id":"https:\/\/www.carnegiefoundation.org\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/carnegie.foundation\/","https:\/\/x.com\/CarnegieFdn"]},{"@type":"Person","@id":"https:\/\/www.carnegiefoundation.org\/#\/schema\/person\/2c01d29bdf9103121a2c02b3b5a30fcd","name":"DerekC","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.carnegiefoundation.org\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/569f1fc418ff9e5f872a030fc28b67243e00ad2d18ad9f8438675ee99b35d02c?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/569f1fc418ff9e5f872a030fc28b67243e00ad2d18ad9f8438675ee99b35d02c?s=96&d=mm&r=g","caption":"DerekC"},"url":"https:\/\/www.carnegiefoundation.org\/author\/derekc\/"}]}},"_links":{"self":[{"href":"https:\/\/www.carnegiefoundation.org\/wp-json\/wp\/v2\/posts\/639","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.carnegiefoundation.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.carnegiefoundation.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.carnegiefoundation.org\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.carnegiefoundation.org\/wp-json\/wp\/v2\/comments?post=639"}],"version-history":[{"count":0,"href":"https:\/\/www.carnegiefoundation.org\/wp-json\/wp\/v2\/posts\/639\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.carnegiefoundation.org\/wp-json\/wp\/v2\/media?parent=639"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.carnegiefoundation.org\/wp-json\/wp\/v2\/categories?post=639"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.carnegiefoundation.org\/wp-json\/wp\/v2\/tags?post=639"},{"taxonomy":"client_controls","embeddable":true,"href":"https:\/\/www.carnegiefoundation.org\/wp-json\/wp\/v2\/client_controls?post=639"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}