2023 SGEA Annual Conference

Written comments are widely used and serve important functions in medical education: as an element in assessment to aid making judgements about students’ readiness to progress, as an essential communication to students to guide their development, as communication to others about students (e.g., Medical Student Performance Evaluation or MSPE) and to provide details or justification for numeric ratings.  The impact and utility of comments are directly related to the quality of the written comments and may be quite variable depending on the skill of the author.
If “words provide novel information that numbers cannot” (Ginsberg, 2021), the quality of a narrative matters. Yet, written narrative comments are often vague and non-specific, providing limited value to trainees and reviewers. Key features of quality comments used for feedback and decision-making purposes are specificity and actionability. For instance, comments shown to be more helpful are those that are lengthier, more detailed, and provide specific examples of appropriate and/or inappropriate behavior (Gulbas, 2016). Importantly, comments that are actionable provide trainees with developmental feedback.
Language used in narrative comments has received notable attention in the recent literature, particularly with respect to gender bias. Much of what has been written focuses on letters of recommendation and gendered language. Work by Turrentine et al (2019) revealed that descriptions of leadership characteristics and the use of superlatives or standout adjectives more often appeared in descriptions of male applicants, and that “a thematic difference attributing performance with men and caring with women” was seen. Similar findings were reported by Hoffman and colleagues (2018) for fellowship letters of recommendation with male applicants reported as superb, intelligent, and exceptional vs. female applicants who were described as compassionate, calm, and delightful. In the undergraduate medical education setting, Gorth (2021) found similar gender-focused language in clerkship evaluations: “wonderful, eager, helpful, team” showing up more often for comments about persons using female pronouns, and “communication, plan, presentation, skills, and performance” associated with persons with male pronouns. These examples suggest that our language choices, even in professional settings, may show bias towards certain characteristics. Raising awareness about this through examples and discussion is essential to begin to change the culture.
The diversity and subjectivity of narratives from different evaluators can be a challenge to providing meaningful evaluation comments. Interrater reliability is essential so that written evaluation narratives may be appropriately used for assessment and decision-making about learners. Following concerns of the quality of written narrative feedback at our institution, we implemented a faculty and staff development workshop focused on quality written narrative feedback. We adapted Kelly MS et al.’s rubric to analyze comments evaluating student performance in pre-clerkship and clerkship experiences. Through individual and small group activities, we calibrated narrative analyses and had participants re-write quality comments. This workshop will use a similar approach to analyze and re-write narrative comments based on student performance in medical education instructional activities.