Researchers at Carle Illinois College of Medicine and Chonnam National University (CNU) in South Korea are building a high-tech framework that could transform how future doctors are trained worldwide. They’re leveraging artificial intelligence (AI) and extended reality (XR) to automatically track and analyze physical indicators of how medical students learn during immersive simulations that mimic the clinical environment. The research could help establish new international protocols for collecting data on the effectiveness of immersive learning.
Extended Reality (XR) Developer Cedric Dumas at CI MED’s state-of-the-art JUMP Simulation Center and Professor Jeeheon Ryu, the director of CNU’s Center for Immersive Learning Technology, are leading the team, which pairs CNU’s learning science expertise with CI MED’s advanced simulation development and applied research. With funding from a six-year grant from the National Research Foundation of Korea (South Korea’s equivalent of the National Science Foundation), their work expands on current studies that leverage AI to improve medical simulation and training by automatically tracking and measuring learners’ activity.
The research is grounded in embodied cognition theory, which focuses on learning as a physical exercise shaped by how a person interacts with their environment. The team is using technology to track physical indicators, such as eye movements, speech, attention, and body language, and provide feedback that adapts in real time to how a learner functions within an immersive clinical simulation.
Dumas says CI MED’s research partnership with CNU is unique in two ways: it leverages multiple AI technologies to calculate markers indicating how people work and learn together, and it analyzes those social signals to provide real-time understanding and feedback. “It is meant to serve educators during or after teaching activities to improve their understanding of the learning process and how their students have learned. Ultimately, it would help to detect issues, adverse events, etc.,” Dumas said.
The goal is early intervention for both individual learners and groups when the system detects physical indicators of roadblocks in the learning process. “If we can have more sophisticated information about the learners available to the professors and instructors, it could have benefits in giving some prioritized help for the student, before they’re at the end of the process,” Ryu said.
Tapping into CI MED’s expertise in XR development, the team also aims to design and develop new extended reality simulations using digital avatars and digital twins – a virtual representation that ‘stands in’ for a human. The new simulations will be designed to build strong decision-making and teamwork skills, even when teams collaborate remotely rather than being in the same room. The new simulations will be incorporated into CI MED’s curriculum and could serve as a model for future simulation content.
Ryu says the team’s most ambitious goal is to work across disciplines and national boundaries to establish global protocols for how researchers and educators use AI to collect multimodal data for assessing learner behavior. CI MED’s leadership among engineering-based medical schools worldwide may be important in achieving that goal. The college leads the Global Consortium on Innovation and Engineering in Medicine, which provides a forum to exchange best practices to advance the curricula of medical schools that incorporate engineering principles.
Ryu says he chose to partner with the Jump Simulation Center on the project because of CI MED’s demonstrated expertise in not only running complex group simulations but also in XR research and development. He says CI MED’s engineering-informed medical curriculum is also an advantage in developing the new framework.
For decades, simulations have played a key role in educating future physicians, allowing students to practice clinical skills and problem-solving in a safe environment. CI MED’s Jump Simulation Center offers high-tech equipment, ranging from task trainers for practicing specific clinical skills to high-fidelity manikins, virtual reality, and now extended reality suites that allow future doctors to practice clinical interactions and group problem-solving.