Author Archives: cegrossman

Three CHSE special- Evaluating Impact of Medical Simulation to Patient Outcomes, Translational Science?

3 up for CHSE and 3 certified — I’d like to think of myself as a nice Brie cheese perhaps….

Anyhow –  we started talking about this last week:


From McGaghie et al – Translational Medicine, February 2010, Vol2, issue 9

——————————–                T1                                         T2                                   T3

increased/improved    KSA, profesionalism          Patient Care                     Patient outcomes


Target                            Individ. And teams          Individ. And teams             Individ. & public health


Setting:                           simulation Lab                 clinic and bedside              clinic and community

Other views:

Kirkpatrick  Reaction/Learning Behavior  Results
Gaba So you taught them something, can they do it when you give them the simulation? You taught them something in simulation, when they go out on the wards, do they really do it – the way you taught them?Ability to measure this.. Does it really change patient outcome?Does it save money?

Just because you show something works in one study, does it mean it can be disseminated and it can be done by others?

Does it really change the outcome of populations as a whole to do these things?



Cognitive Load and Overload… 4_10_2015


Interesting articles, trying to figure out how all of these ideas  apply to medical simulation

(truth be told we didn’t read all of them – just some…)

1 . Manipulation of cognitive load variables and impact on auscultation test performance Ruth Chen • Lawrence Grierson • Geoffrey Norman Received: 20 January 2014 / Accepted: 20 November 2014

2. 4C/ID in medical education: How to design an educational program based on whole-task learning: AMEE Guide No. 93

3. Mental load: helping clinical learners Geoff White, Clinical Education and Professional Development Unit, School of Primary Health Care, Monash University, Australia

4. Cognitive load theory in health professional education: design principles and strategies. Medical Education 2010: 44: 85–93

5.  Cognitive Load Theory: Implications for medical education: AMEE Guide No. 86 (more overview wtih examples)

Extraneous Load – load not essential to the task
Intrinsic Load – load associated with the task
Germane Load – available working memory to learn and to deal with the extraneous and intrinsic load —  these are elements that allow cognitive resources to be put towards learning/problem solving i.e. assist with information processing.

We should think about each of these as we tailor our learning experiences at different levels of learners.  In health care – as opposed to powerpoint slides – there is always going to be a lot of extraneous load – and we do need to teach learners how to deal with this part of the tax on our working memories.

Nice pictures from article # 4 that show how overloading the intrinsic or extrinsic load can leave not potential germane load left for learners to work with in particular situation

A nice alternate way to think about cog load in reference to slides in powerpoint (but I can abstract this to similar “noise” in a simulation scenario)

Have to vary your learning instructional strategies for novices vs. experts…

Article #2 provides a detailed look at how to provide whole task learning – using cognitive load theory at it’s backbone. This is not just simulation but multiple ways of learning – that take a scaffoldign approach of single or simple task to more complex or multiple task – while at the same time (if I am interpreting this right) allowing for more direct coaching at the beginning and then more hands off coaching at the end – as learners go from novice to expert or at least more advanced. Contained within this schema is continual assessment, feedback and reflection.

Interesting approach – starting with a worked example – “show them what you want them to do” then break this down into sizabel chunks – that progressively get more difficult as the learning continues.

I had to read this article several times. I will probable need to read if a few more times.

Article #4 talks about strategies to minimize extraneous load and minimize intrinsic load for novice learners, and how to increase intrinsic load (and thus germane load) as learner expertise increases.
Minimizing extraneous load for novices:
– starting with goal free strategies (generate a list of as many diagnostic possibilities as you can) and moving to goal directed (what is the most likely diagnosis).
-starting with worked examples (i.e. demonstrations of how to do the task) then moving to completing more and more of a task – I liken this to learning a new computer software by trial and error – clicking button sequences until you finally get it right. It’s inefficient and frustrating. Much better to show the completed task and then let the learner complete a similar one
-Reinforce material by using multimodal presentation
A deconstructed roadmap for moving from unconscious incompetence to unconscious competence








Assessing, assessments,and feedback…

Boulet JR, Jeffries PR, Hatala RA, Korndorffer JR, . Feinstein DM, Roche JP. Research Regarding Methods of Assessing Learning Outcomes Sim Healthcare 6:548-551,2011
Simulation based assessments are used for both formative and summative assessments of healthcare providers. The objective of this article was to look at how their use is supported and to provide direction in the form of consensus recommendations for research. . They delve into “the four components of Kane’s inferential chain –Scoring, Generalization, Extrapolation, and Decision/Interpretation.” The importance of the need for reliability as well as validity of simulation based assessments is emphasized within a brief review of existing research (as of 2011) and opportunities for additional work. Their recommendations target five areas for research which include measurement error, developing scoring rubrics, supporting the underlying theories for learning with simulation, translation of simulation based learning to the clinical environment, and the impact of implementing these assessments for healthcare.

Scenario Design – 2/27/15

Photo Credit: Umpqua via Compfight cc

Photo Credit: Umpqua via Compfight cc

Comparative effectiveness of instructional design features in simulation-based education: Systematic review and meta-analysis. DAVID A. COOK. Medical Teacher. 2013; 35: e867–e898

Simulation in healthcare education: A best evidence practical guide. AMEE Guide No. 82 IVETTE MOTOLA. Medical Teacher. 2013; 35: e1511–e1530
This best evidence practical guide for simulation in healthcare education is highly recommended reading for educators. It is part 2 of the Guide, and is focused on “the educational principles that lead to effective learning.” Curriculum integration, feedback and debriefing, deliberate practice, mastery learning, capturing clinical variation, individualized learning, and finally approaches to team training are discussed. Each topic is defined and explored in relation to its effective use in simulation with practical implementation points and challenges that may be encountered


Developing High-Fidelity Health Care Simulation Scenarios: A Guide for Educators and Professionals Guillaume Alinier. Simulation & Gaming
2011. 42(1) 9–26

This is a conversational piece on important considerations in scenario design that concludes with a sample scenario design template. It’s a good article for prompting new simulation educators (or just disorganized people like myself (-:) to think through how we build scenarios to address specific objectives and challenge learners in specific ways. Figure 1 depicts general courses of action and prompts the educator to think about actions or conditions that result in a change in patient condition as the scenario unfolds. It’s a good synthesizing graphic.
The section on “Preparing Scenarios in an Organized Manner” is particularly helpful as a practical guide to planning a smooth running scenario.  The Salas article (see SMARTER 2 articles down) and the TEACH Sim article resonate better with me and reflect the human factors and observational assessment orientation, but this is easily digestible and says much of the same stuff in different ways.

The Template of Events for Applied and Critical Healthcare Simulation (TEACH Sim) A Tool for Systematic Simulation Scenario Design. Benishek & Salas et al. Sim Healthcare 10:21-30, 2015.

The authors review and compare five existing scenario design templates and then feature their own design TEACH Sim. This is a great evidence- based tool for designing simulation scenarios. In fact, we were so impressed that we have already started using it (with permission from Eduardo Salas, PhD). It has a logical flow that is easy to understand and with the addition of our own simulation operation flow chart for high fidelity scenarios includes all the components for effective scenario design. Instructions for their template are also included.

 A Measurement Tool for Simulation-Based Training in Emergency
Medicine: The Simulation Module for Assessment of Resident Targeted
Event Responses (SMARTER) Approach.  Rosen, MA et al, Sim Healthcare 3:170–179, 2008

An event based approached to training and measurement, assesment and feedback. Systematic fashion to do several things: “1. Develop and maintain links between simulation scenario events, performance measures, and ACGME core competencies, 2. generate diagnostic measurement that feeds the processes of providing corrective feedback to accelerate kill acquisition and provide learning outcomes data rooted in the ACGME core competencies, 3. to provide opportunities to perform that are structured to maximize learning and assessment opportunities”

8 step process.

Each scenario designed to sample a specific part of the core competencies’ content (can’t measure everything at once — but can do multiple sims or have multiple different observations not using sim that can triangulate on the components of the core competency that you are trying to grade.)

A practical guide – with well delineated examples at multiple areas of the process of scenario creation – that are focused on EM and maybe critical care cases – but could be abstracted to other disciplines in formation/generation of scenarios.

Personal thoughts: Why didn’t I read this 7 years ago.. Check out for some nice examples from the Academic Life in Emergency Medicine crew of simulation cases linked to ACGME core competencies.

Debriefing and Simulation Trail Mix – a little bit of a bunch of things

Photo Credit: CapsLK via Compfight cc

Photo Credit: CapsLK via Compfight cc

Alternative Educational Models for Interdisciplinary Student Teams. Judy L. LeFlore: Sim Healthcare 4:135–142, 2009

13 teams – randomized to Self Directed  (described in this paper as allowing students to proceed through the scenario with little or no input from the instructor until after the scenario) learning with facilitated debriefing vs. instructor modeled learning (defined in this paper as instructors model the appropriate responses and interventions during a team simulated clinical scenario while the students or participants observe – with the experts verbalizing what they are doing and why it works in the scenario — this act then followed by students participating in a simulation) with modified debriefing afterwards. Team content (all seemingly novice students – NP, RN, RT, and social work). Reported debriefing was performed by someone trained in debriefing through a comprehensive simulation workshop and debriefing workshop – not stated what exact question styles were used.

Results: Instructor Modeled learning (IML) – had no effect difference on knowledge test vs. Self directed learning with facilitated debriefing. However – the IML groups had higher satisfaction score, and higher overall teamwork and CRM scores (behavioral asssessment checklist); IML team also had quicker times to intervention based on the Technical Evaluation Tool used.

Questions from me? People are generally happier when the explicitly know what is expected of them.Did the IML just teach to the test/evaluation? Is that necessarily a bad thing? Does this work better for novice vs. expert learners? How do we model what we want students to do?  Video, other fashions, simprov.  Personally I have had our residents watch videos of teams prior to coming into our simulations – (shoutout! to the Stanford AIM LAB )

Directed self-regulated learning versus instructor regulated learning in simulation training. Ryan Brydges. Medical Education 2012: 46: 648–656

First – What is  Directed Self Regulated Learning?  Well ” It requires a knowledgeable educator to design practice conditions using validated learning principles. A trainee then steps into this structured setting and is given a limited control of a specific aspect of practice and therefore is metacognitively, behaviorally, and motivationally active in the learning experience.”

Instructor regulated learning as described in their paper is similar to what many centers do for procedural group training 1 instructor, 4 residents, everybody gets a turn/gets to demonstrate skill  – then you move on.

45 PGY1 Internal Medicine Residents – Standard pre-training curriculum /standard testing/questionnaires pre/post educational intervention – assigned randomly to Directed Self Regulated Learning (DSRL – solo participant could choose to progress from easy to hard LP simulator models on their own time in a 35 minute time frame, could play pre training video ad lib during training session – then take the post test – then get 15 minutes for feedback)) vs. Instructor regulated learning (IRL) for teaching lumbar puncture using simulation (instructor ratio 1:4, did not have access to video during session – as instructor was resource, instructor and students collectively decided when to progress from easy to hard – still within  a 35 minute time frame for the 4 students (as opposed to 1) – then take post test – then get 15 minutes feedback as a group).  Then everybody gets a 3 month retention test.

23 residents completed the study. Trained/blinded experts rated videotaped procedures.

Pre and post test findings were similar (although IRL post test trended higher) — but at the 3 month time frame – better retention trend in checklist score, and significant score increase  in the GRS (global rating score) of the DSRL group.

Maybe this speaks to something about being responsible adult learners, and being invested in their own learning… Multiple moving parts here, small study, food for thought.

Comparison of Postsimulation Debriefing Versus In-Simulation Debriefing in Medical Simulation. Jon N. Van Heukelom, MD; Sim Healthcare 5:91–97, 2010

Both post simulation and in-simulation debriefings have been used in medical simulation, the question remains which method is more effective? The authors compared these two styles of debriefing with 161 third year medical students participating in ACLS simulations. A retrospective pre-post assessment was completed by the students on “self reported confidence, level of knowledge related to medical resuscitation and the simulation itself.”

The post simulation debrief group gave higher rankings for “effectiveness of the debriefing style, debriefing leading to effective learning, and understanding of correct and incorrect actions.” These results were statistically significant, and students rated this method as more effective overall.

The discussion noted advantages and disadvantages of each method and its effect on the quality of learning. Their students however “did not feel that interruptions during a simulation significantly altered the realism of the simulation.”

Limitations of the study noted that this is one level of learners in one type of simulation, thus the inability to generalize to other levels of learners or other types of simulations. It did not include clinical outcome data nor did it specify the amount of time spent in simulation versus debriefing for each group (total length of time =20 min).

When Things Do Not Go as Expected: Scenario Life Savers: Dieckmann, et al. Sim Healthcare 5:219–225, 2010

Dieckmann et all discuss the need to build into scenarios the possibility that learners will not proceed down the paths that the instructor considers possible or likely. This might result from altered learner comprehension of the meanings and clues within a scenario, inability to accept the scenario as plausible, or mismatch between the scenario difficulty and the learner’s ability.   These situations might compromise the potential for learning in a simulation.

“Life savers” attempt to rescue the scenario and make it meaningful and relevant for the learner. These life savers can be brought into the scenario from within or outside the scenario. Examples of altering the scenario from within would include having a confederate administer a drug that is necessary to the progress of the scenario when the learners have failed to do it, or manipulating vital signs on the fly from the control room to make manikin status changes more or less obvious. Examples of altering the scenario from outside would include speaking via the overhead speaker into a scenario to stop participants from doing an action that would be harmful to themselves or to the manikin, or stopping and restarting a scenario to recover from control room or technology mishaps.

Life savers that are employed by a role player from within a scenario must follow the logic of the scenario. They must make sense to the learner in the context of the situation. By contrast, life savers brought from outside the scenario do not, and in some cases can intentionally disrupt the scenario.

This has implications for both scenario design and prebriefing. During design of scenarios, instructors should consider the potential need for life savers, and how they might be implemented if necessary. The learners should be prepared for this possibility during the prebrief.

Serious Games

Serious Games for Education and Training. De Gloria, A et al. International Journal of Serious Games. Vol 1, (1), 2014

“Contextualizing the player’s experience in challenging realistic environments, supporting situated cognition.” Allow players to be in a play environment to explore formal learning in a wide variety of situations, even to mimic work in “complex/costly environments or dangerous/critical situations.”

This is a great soup to nuts review (for a non-serious gamer) of the state of the art of serious gaming.

Interesting concepts:
1. Constructivist learning theories in gaming – knowledge created through experience while exploring the world and performing activities. But have to be mindful of cognitive load theory
* game has to balance the inherent challenge and the players ability to address and overcome the challenge
2. “Flow” to measure engagement in an educational game (has 8 components)
concentration, challenge, skills, control, clear goals, feedback, immersion, and social interaction.
3. Assessment and feedback through the serious game
– different learning analytics through which to provide feedback (may also include cameras, eye trackers, vital signs), as well as assessment to figure out what the leaner has learned.
* this is not mentioned in this article – but check out (play an interview – and see the feedback you get…)

figure 4 – reminder of Bloom’s taxonomy and Kolb’s learning theory – also brought up was the Nonaka SECI model (socialization, externalization, combination, internalization)

From Content to Context: Videogames as Designed Experience. Squire, K. Educational Researcher Vol 35, no 8, pp 19-29, Nov 2006
“videogames as a designed experiences, in which participants learn through a grammar of doing and being.”
Table 1: Interesting aspects of components of the learner and game as interface – that easily apply to learning goals in bloom’s taxonomy and Kolb’s learning stages.

Interesting side tales about the Grand Theft Auto game, as well as Civilization III, and a game called Supercharged! (to help learn about electrostatic physics.) Not everything in serious games is overtly on the military front.

Possibility for anonymity as a member of a game, being able to experiment, learn through successes, learn through failure, redo and improve, to become expert problem solvers.
Learning from a game is only as good as the work that goes into the designed experience for the learner.

A Comprehensive Review of Serious Games in Health Professions.  Ricciardi F, De Paolis LT. International Journal of Computer Games Technology. Vol 2014, Article ID 787968

In their review of serious games developed for health professions, the authors described the “state of the art of serious games” and sought to understand if they are useful tools for training as well as the benefits and the current issues related to them.  The review was grouped by the area of application ( surgery, odontology, nursing, cardiology, first aid, dietician and diabetes, psychology and other.) They concluded that serious games for health professions are not as widespread as expected and the highest number of games was found in the field of first aid.   The focus of the papers was mostly on development of serious games and how they were used in a particular area of health.

***In the serious games review – only a few included an evaluation component. ( no standardized way to do this in serious games.)

Another issue identified was the lack of a multiplayer component to many of these games; Since healthcare now emphasizes the importance of a teamwork approach, the authors thought this to be a shortcoming.

Four factors were identified that highlight the differences between serious games from traditional simulators: entertainment factor, developmental costs, developmental time, and deployment costs.  The entertainment factor was the obvious advantage and each of the other factors were claimed to be lower in cost for the serious games.

Questions from the reading:  Who develops the games? Who states the game are valid for the skills trying to be learned? Time/Cost analysis for development/testing and use of serious games vs. simulation training?

 de Wit-Zuurendonk L, Oei S. Serious gaming in women’s health care. BJOG 2011;118 (Suppl. 3): 17–21.

A clipped quote from the article “Systemic search in pubmed – this search revealed zero hits for serious gaming in women’s health care.”

Learning background theory: Four elements important to adult learning – autonomy, past experience, goal oriented (or orientated), problem based learners rather than content oriented.

People enjoy gaming, some evidence that serious games are associated with increased technical skills. Male students like gaming more than female students (from data in the supplement).

In a non-technical skills pathway: One study that showed that triage was better after serious games vs. a card based game.

Cons of serious games – solo work; not able to fully replicate team dynamics;

Take Away thoughts from the morning:

Not a ton of data yet about efficacy of serious gaming in relationship to medicine (but taking the leap of faith from the aviation community that we have to medical simulation…)

Could serious games be the pre-reading/preparation work of the future for in person simulation training?

Interesting applications are out and about, they are not all military based anymore.

The VCUHS Geriatrics Division along with the school of medicine developed a serious game of sorts – more inter professional interaction, with faculty facilitation – using simulated patient information, not in real time, not in virtual reality, but I think this counts.