Limiting Gender Bias in Simulation Assessment

Today’s piece is written by Dr. Lall. She is an Associate Professor and Associate Residency Director of Emergency Medicine at Emory University in Atlanta, GA. She is also the current president of the Academy for Women in Academic Emergency Medicine. Dr. Lall’s research focuses include physician wellness and gender bias and inequity in medicine. The following is a summary of her recent publication on this issue.

You can find the publication here:

Jeffrey N. Siegelman, Michelle Lall, Lindsay Lee, Tim P. Moran, Joshua Wallenstein, and Bijal Shah (2018) Gender Bias in Simulation-Based Assessments of Emergency Medicine Residents. Journal of Graduate Medical Education: August 2018, Vol. 10, No. 4, pp. 411-415.

Background:

There is a paucity of studies on gender differences in milestone assessment. One recent large multi-site cohort study of EM residents evaluated bias in end-of shift evaluations and found a significant bias based on resident gender (Dayal A et al, 2017).  Shift evaluations usually represent subjective assessments and residents are evaluated only on cases seen during a particular shift, resulting in considerable variation with respect to which competencies are assessed across residents and rated by faculty. Simulation allows for a more structured, consistent evaluation environment in which residents can be tested on identical clinical problems, and in which specific competencies can be assessed. We hypothesized that simulation, being a more objective assessment tool, may mitigate gender disparities in resident assessment.

In our three year experience with biannual milestone-based simulation assessments of all our EM residents, no significant gender bias was observed in contrast to other forms of resident assessments, such as end-of-shift evaluations.

Tips for SIM Educators:

1. Training the standardized patient is key to successful simulation assessment.
   1. Pilot test the scenarios to ensure the case plays as expected and appropriately elicits the opportunity for the resident to perform the desired critical behaviors.
   2. Evaluate for potential bias introduced by the standardized patient script or actions as the scenario plays out.
   3. Ensure that standardized patient responses are the same every time.
      1. Same response in the same tone of voice with the same facial expressions whether the physician is male or female.
   4. Standardized patient script cues should be written with binary language.
      1. If the resident does not introduce themself to the patient, prompt the resident with “Doctor, what is your name?”
      2. Avoid language like miss, ma’am or sir
2. Education and training of the rater is of critical importance.
   1. Raters should be instructed that evaluation in these cases is not subjective.  Evaluation is binary and based on observable behavior only.
3. Convert milestone language into binary, observable behaviors
   1. Assessment items should avoid language that may introduce bias including subjective assessments.
      1. Agenic adjectives: typically used to describe men and when used to describe women carry a negative connotation.  Examples include assertive, autonomous, independent, confident, intellectual.
      2. Communal adjectives: typically used to describe women and when not demonstrated by women carry a negative connotation.  Examples include kind, compassionate, sympathetic, warm, helpful.
   2. Focus on action based assessment items, for example:
      1. Resident introduced themself to the patient
      2. Resident updated the family using lay terminology
      3. Resident ordered magnesium without prompting

Checklist for Limiting Bias in Simulation Assessment

• Standardize the Scenario
  • Standardized patient/Confederate scripting and training is crucial
  • Simulation operator training
  • Pilot the case
• Create an Objective Rating Tool
  • Focus on observable behaviors rather than subjective assessments
    • Observed/ Not Observed/ Unable to Assess
  • Train the raters
  • Use language that avoids bias
• Monitor for bias
  • Analyze data after an assessment for validity evidence, reliability, and evidence of bias
  • Make adjustments to the case as needed