Is it an art or a science? Assessment mixology in reflective practice.

Abstract

Introduction/Background

Reflective practice is central to the commitment to improve standards of clinical practice. In Phase 2 (years 4 and 5) of the MBChB programme at the University of Auckland, a portfolio of reflective pieces is used as a must-pass assessment for the personal and professional skills (PPS) curriculum domain. A descriptive rubric is used to mark and moderate students’ portfolios along eight numerically scored categories, resulting in a nominal grade (i.e., fail, borderline, pass, distinction).

Methods

Following a blind bench-marking exercise, the reliability and defensibility of the PPS portfolio assessment was examined. Phase 2 portfolio results from 2021 – 2023 were submitted to statistical analyses.

Results

Numerical scores for portfolios awarded a distinction grade were significantly higher than those awarded a pass grade. However, numerical scores overlapped for the high end of pass grades and low end of distinction grades. Where numerical scores overlapped, distinction grades were reliably associated with higher scores in key rubric categories, including all categories assigned heavier weighting.

Discussion

Despite the lack of a rigid cut-point between groups, overall scores for our ‘pass’ and ‘distinction’ groups were reliably different. Assessing the overall narrative against the rubric allows a little ‘art’ into the mix, capturing the richness of portfolio content. This requires assessors to be both flexible and considered in their approach with preparation and experience key to reliability of assessment.

Keywords

Reflective portfolio; Qualitative assessment; Medical education; Professionalism

Watch recording here: https://youtu.be/1iz017poDC8