Advertisement

Clinical assessment during a global pandemic – Transitioning to a COVID safe hybrid OSCE

Published:October 14, 2021DOI:https://doi.org/10.1016/j.ijosm.2021.10.004

      Abstract

      Objective structured clinical examinations (OSCEs) are often used to assess the clinical competence of students in preprofessional osteopathy training. During the COVID-19 global pandemic, the final year OSCE in the RMIT University osteopathy program was redeveloped leveraging online technologies within COVID-19 guidelines such as hygiene and occupancy limitations. Final year osteopathy students were assessed using a hybrid ten station OSCE, comprising both online and face-to-face components. The examination was led by a pre-recorded narrated PowerPoint video. The video contained instructions, case information for five cases and prompts for the practical stations. A student model stepped into the room as needed for practical stations. The examiner assessed students from another room via video streaming, with limited interaction with students. The hybrid OSCE was conducted safely during Stage 4 restrictions adhering to COVID Safe guidelines, allowing robust competency assessment of final year students, enabling timely graduation and transition to practice. Institutional support, technology infrastructure, clear communication and stakeholder collaboration are key to successful implementation. The hybrid OSCE format offers a potential solution for institutions delivering high-stakes assessment in the continuing challenges of clinical assessment in the post COVID landscape. Adopting hybrid assessment formats may facilitate remote assessment of students in clinical placements.

      Keywords

      Implications for practice

      • It is possible to apply a COVID-19 Safe approach to conduct high stakes clinical examinations safely and effectively.
      • Amendments to the assessment approach requires collective support from the institution, staff, students, examiners, and regulatory bodies.
      • Prioritise frequent, clear communication with all stakeholders.
      • Allocate specific staff to the planning and communication roles to ensure consistent, clear messaging.
      • An adaptable, reflexive, and agile stakeholder group is needed to ensure success of a hybrid OSCE.
      • Sound institutional infrastructure is key to delivery of a hybrid OSCE model.

      Background

      RMIT University (RMIT) is a publicly funded university in Melbourne, Australia. RMIT is one of three education providers of pre-professional osteopathy training accredited in Australia. The program is a five-year, double bachelor's degree, utilising several evaluation strategies including a high stakes Objective Structured Clinical Examination (OSCE). OSCEs assess the competence and safety of students throughout a course and prior to graduation and registration as a healthcare professional. OSCEs enable synchronous assessment where students progress through a series of timed stations and are graded on clinical knowledge and skills against standardised criteria [
      • Zabar S.
      • Kachur E.
      • Kalet A.
      • Hanley K.
      ]. OSCEs usually involve 6–12 stations with either continuous or multiple cases to assess the domains of clinical competence in osteopathic practice. OSCEs in osteopathy have been shown to offer a psychometrically sound assessment of clinical competence [
      • Vaughan B.
      • Florentine P.
      The OSCE in a pre-registration osteopathy program: introduction and psychometric properties.
      ].
      In March 2020, Australia was placed under Stage 3 (Stay at Home) restrictions, which was escalated to Stage 4 (Lockdown) restrictions in Melbourne until late October 2020. During this time, all learning and teaching activities in universities in the Melbourne metropolitan area shifted rapidly to online delivery as campuses were forced to close. Delivery and assessment of specialised practical skills were deferred until they could be delivered in accordance with COVID-19 Safe guidelines. In vocational health profession courses like osteopathy, the ceasing of face-to-face practical classes and clinical placements created significant challenges and concerns for all stakeholders. The implications for both education providers and students were significant, including delays in course progression and risks to ongoing skill acquisition and development.
      Despite these challenges, RMIT pledged students in their final year would be able to complete their program on time. In a practical sense, this meant the osteopathy faculty team needed to develop alternative approaches to face-to-face clinical assessments to ensure course progression for this cohort. We needed to rapidly develop a hybrid (online with face-to-face elements) OSCE approach to enable timely delivery, while meeting the requirements of the institution's assessment policies and the accreditation authority. In this commentary we discuss the impacts, challenges, and opportunities we experienced in adapting our face-to-face fifth year OSCE into a COVID-19 Safe hybrid OSCE in 2020. In sharing our experience, we hope this may be of assistance to osteopathic stakeholders should ongoing adaptations be needed as we continue to navigate the uncertainty of the COVID-19 pandemic.

      Impact: need to change high stakes assessment

      Previously, the final year OSCE required students to move between multiple rooms in the RMIT Health Clinic, spending between 5 and 10 minutes in each room. Students, examiners, and invigilators would share physical spaces such as consultation rooms and corridors. Assessments were recorded on paper and collected by organisational staff at the end of the day. There was a high likelihood that participants would be exposed to different surfaces and items and interact with multiple student models and examiners using this traditional approach. For 2020 we had to reduce the risks of transmission outlined above while adhering to institutional and government guidelines.

      Actions: planning and delivery

      To inform our approach, we consulted with university safety personnel and professional staff, and performed a brief review of recently published literature [
      • Boursicot K.
      • Kemp S.
      • Ong T.H.
      • Wijaya L.
      • Goh S.H.
      • Freeman K.
      • et al.
      Conducting a high-stakes OSCE in a COVID-19 environment.
      ,
      • Vivekananda‐Schmidt P.
      • Lewis M.
      • Coady D.
      • Morley C.
      • Kay L.
      • Walker D.
      • et al.
      Exploring the use of videotaped objective structured clinical examination in the assessment of joint examination skills of medical students.
      ]. We identified several necessary additional measures that could be applied, and the need for sufficient training of safety protocols to ensure adherence when attending the examination.

       List of additional measures for COVID safe hybrid OSCE

      • 1.
        Mandatory RMIT COVID Safe Training module to ensure safe return to campus for assessment activities for involved personnel.
      • 2.
        Victorian state government COVID-19 infection control training modules for involved personnel.
      • 3.
        Strict infection control and personal hygiene on the day of the OSCE.
      • 4.
        Physical distancing of individuals and physical isolation of different groups (student/patients, examiners, faculty/professional organisational staff).
      • 5.
        Electronic data gathering via online (Canvas hosted) rubrics.
      • 6.
        The RMIT Critical Incident Management Team (CIMT) required detailed plans outlining the above protocols for minimising exposures, contact tracing and infection control.

      Challenges of COVID-19 impacts

      There were several challenges we faced when redesigning and implementing the COVID Safe OSCE in 2020.

       Challenge 1: managing examination spaces

      Scheduling of students and staff needed to comply with institutional and government regulations and limit contact between people wherever possible. We were required to limit the number of people on location, control people entering and exiting the building, limit the movement of people between rooms and comply with maximum room capacities set by the CIMT.

       Solution: single room with remote examiner

      Each student was assigned to a single room for the duration of the examination. They were assigned a student model who would wait in the hallway outside the examination room and assist only when required. Models consisted of 4th year osteopathy students and 5th year students who had already completed their assessment. The use of multiple exam versions and sequestering ensured students had no prior exposure to the examinable cases before their scheduled time. The examiner was assigned to an adjacent room and observed via a videoconferencing platform built into Canvas. All personnel were scheduled according to strict building entry and exit times.

       Challenge 2: standardisation of assessment

      It was critical that our novel high-stakes examination was fair, valid, and defensible to satisfy requirements set by the Australian Osteopathic Accreditation Council (AOAC), the regulatory body for osteopathy education in Australia. Standardisation of examination content, conditions and grading were our key concerns.

       Solution: benchmarking, peer validation and evidence-based design

      A process of benchmarking and peer validation from experienced osteopathy educators was utilised when developing the examinable content and grading rubrics. Validity of the OSCE was achieved by benchmarking against the 2019 Osteopathic Graduate Capabilities []. Examinable cases were standardised through peer examiner validation ensuring multiple versions were comparable. The rubrics used a matrix describing the clinical skills assessed in each station and associated marking criteria. Criteria were then aligned with the graduate capabilities for each station.
      The OSCE was led by a 65-min continuous pre-recorded PowerPoint presentation with narration which contained the examinable cases. Each station consisted of written cases or instructions describing which clinical skills students were required to demonstrate. The video included set timings for each station, verbal instructions, a countdown timer, and audio effects to signal the beginning and end of each station. Use of a video as both a guide and prompt ensured consistency in the delivery of the cases and the examination environment.
      The use of asynchronous grading was considered to further reduce personnel on site during the examination. However, a review of the literature and peer discussion facilitated our decision to use synchronous grading. Evidence suggests asynchronous examiners grade with lower scores to synchronous examiners when applying global rating scales in OSCEs [
      • Vivekananda‐Schmidt P.
      • Lewis M.
      • Coady D.
      • Morley C.
      • Kay L.
      • Walker D.
      • et al.
      Exploring the use of videotaped objective structured clinical examination in the assessment of joint examination skills of medical students.
      ].

       Challenge 3: student and examiner apprehension

      In a challenging year, students and examiners reported to staff that they were apprehensive about the uncertainty, which we needed to address. Managing concerns and expectations of stakeholders is a key component of managing change during a stressful, uncertain period.

       Solution: communication and training

      A sample of the video was made available to students and examiners in advance. This sample video included examples of cases and the audio-visual cues and station timings to be used in the formal examination. Students and examiners were instructed to use this video as a study tool to develop familiarity and reduce apprehension.

       Challenge 4: digital literacy & technology infrastructure

      Due to state government-imposed restrictions, academic and professional staff were not permitted access to RMIT campuses in the lead up to the OSCE, and first access was permitted 3 days prior to the assessment taking place. Therefore, technology infrastructure in the RMIT Health Clinic had not previously been tested for this activity. Consideration and testing of available bandwidth, functionality of software and associated hardware settings was essential to ensure successful delivery of an online hybrid OSCE approach.

       Solution: using familiar software and testing

      The OSCE was conducted using a videoconferencing platform that had been used by clinical educators during virtual clinic throughout 2020. Those educators who were recruited as examiners had undertaken 16 sessions with students throughout semester 1 and 2 using this software. Prior to the OSCE, staff tested the planned setup and adjusted computer settings to ensure webcam definition, microphone sensitivity and speaker volume were optimal for accurate observation and assessment of students. The video, stored on a USB-drive, was played through VLC Media Player to ensure stability and fidelity, which cannot be guaranteed when livestreaming via an internet connection.
      Due to concerns that unexpected connectivity disruptions may impact the assessment, several low-tech contingency plans were developed. If connection difficulties occurred, examiners were instructed to revert to manual timing and observation of the students through an open door to maintain appropriate physical distancing. If a more significant computer system disruption had occurred, hard copies of the cases were available to examiners who could then have presented them to students while manually timing and observing from the doorway of the consultation room. Fortunately, these contingencies were not needed, but the existence of these plans were reassuring to the students, examiners, and faculty.
      Where possible, we would recommend trial of the hybrid assessment format in advance through use of mock-exams to test the design and infrastructure.

      Opportunities created due to redesign of high-stakes assessment

       Improved alignment with clinical practice

      Removal of the examiner from the room gave an opportunity to better simulate the professional clinical environment. Students were instructed to direct all communication to their patient model rather than the examiner. They did so with the knowledge there would be no probing questions throughout the examination, and no additional case information provided beyond what was contained in the video. In doing so, we prioritised standardisation and authenticity of the assessment over the ability to ask probing questions, which may not be suitable for all assessment scenarios. These examination conditions enabled an authentic assessment of a simulated clinical encounter that was not influenced by the examiner.

       Development of assessment literacy through deliberate exam practice

      The examination format enabled repeated opportunities for structured development and guidance for clinical educators and students. Fifth year clinical educators met online weekly with their student group in the eight weeks leading up to the OSCE. Within these sessions, all groups undertook targeted exam practice by revising and practicing the clinical skills assessed in the OSCE and applying the scoring rubrics during peer assessment. Firstly, the clinical educator would explain the specific criteria and lead a discussion on achieving the different levels of performance. Learners were placed in virtual breakout rooms in pairs to practice novel case scenarios, with one student responding to the case and the other acting as a remote patient and peer assessor. Feedback was reviewed by each pair against their performance on the rubric before swapping roles. Clinical educators moved between the virtual rooms and offered additional feedback. This approach enabled repeated opportunities to practice the exam format, using the cases and rubrics in a similar online format to the actual exam. This deliberate approach to exam practice is likely to have developed the assessment literacy of all users.

       Development of remote assessment capabilities within the profession

      By responding and adapting to an unusual situation we have harnessed an opportunity to develop novel assessment capabilities within the osteopathy program at RMIT. By using online examiners, we gained the capability of running high-stakes examinations remotely when students or examiners are not able to attend campus. Two students were unable to attend campus for their OSCE due to participation in external clinical placements interstate. Both were examined using the same OSCE design with the video live streamed rather than played from the USB drive. Several students on campus in turn were assessed in their supplementary exams by a remote examiner with no issues reported. The capability to use remote examiners may be relevant to education providers reconsidering their delivery of high-stakes assessments during the COVID-19 pandemic.
      The use and further development of these capabilities may have broader implications for high-stakes assessment beyond the current pandemic. Hybrid assessment formats with remote examiners may be utilised for work-integrated learning (WIL) in osteopathy programs, allowing online synchronous assessment of students during their placements in regional and remote clinics. Additionally, there is great potential for the use of simulated patients in a video led OSCE. Pre-recorded footage of models performing the case scenarios could be included in the video, which may increase the authenticity of the assessment.

      Outcome

      In 2020, the OSCE was conducted over three consecutive days, 86 students were examined (another 3 completed deferred assessments), 36 students participated as patient models and 18 staff participated in grading or invigilation. To date, there have been no reports of COVID-19 infections in any of the students or staff involved in the OSCE. 86 out of 89 students (97%) successfully completed the OSCE on their first attempt. However, 3 students were not successful on an individual station on their first attempt, but successfully completed the supplementary OSCE.
      We conducted an informal evaluation of student and staff experiences following the OSCE. We asked a series of questions via Microsoft Forms where responses were marked on a 5-point Likert scale. The questions sought informal feedback on the exam content, structure and management, safety, and the student and staff experiences. Students overwhelmingly reported positive experiences, particularly the use of the video cue and the removal of the examiner from the room, with both interventions reducing their perceived stress levels. Examiners also reported positive experiences and commented they believed students were able to perform to their full potential with the examiner in a separate room. The format removed the requirement to monitor the time of each station permitting examiners to focus on observation of the student resulting in more considered grading and meaningful feedback. Due to the additional pressures on staff and resources resulting from multiple lockdowns during this time, formal evaluation was not conducted.
      Embracing an opportunity for change has resulted in a robust high-stakes hybrid OSCE within the RMIT osteopathy programme. As the university sector continues to harness technology and embed online teaching and learning pedagogies, the osteopathic profession will benefit from continuing to revise traditional education practices and adopt contemporary approaches to clinical education. Further development of this assessment format and the digital literacy of both staff and students will see more opportunities for positive impact on the osteopathic education.

      Declaration of competing interest

      I hereby declare that there exist no conflicts of interest on behalf of all authors.

      Ethical approval

      Not applicable.

      Funding

      This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

      Acknowledgments

      We would like to acknowledge the staff within the Discipline of Osteopathy at RMIT University for their input and feedback surrounding the hybrid OSCE.

      References

        • Zabar S.
        • Kachur E.
        • Kalet A.
        • Hanley K.
        Zabar Se Kachur Ee Kalet Ae Hanley Ke Objective structured clinical examinations 10 steps to planning and implementing OSCEs and other standardized patient exercises. first ed. Springer, New York, NY2013 (New York : Imprint: Springer; 2013)
        • Vaughan B.
        • Florentine P.
        The OSCE in a pre-registration osteopathy program: introduction and psychometric properties.
        Int J Osteopath Med. 2013; 16: 198-206
        • Boursicot K.
        • Kemp S.
        • Ong T.H.
        • Wijaya L.
        • Goh S.H.
        • Freeman K.
        • et al.
        Conducting a high-stakes OSCE in a COVID-19 environment.
        MedEdPublish. 2020; 9
        • Vivekananda‐Schmidt P.
        • Lewis M.
        • Coady D.
        • Morley C.
        • Kay L.
        • Walker D.
        • et al.
        Exploring the use of videotaped objective structured clinical examination in the assessment of joint examination skills of medical students.
        Arthritis Care Res. 2007; 57: 869-876
        • Osteopathy Board of Australia
        Capabilities for osteopathic practice.
        2019 (Available from:)