Volume 16 - Issue 1, 2016 - Best Practice

SOCRATES: An Electronic Evaluation System to Advance Medical Trainees

Developing thorough competency in both specialty and professionalism is of pivotal importance for radiology trainees. Residents and fellows are challenged by uncertainty and limited experience early in their career, by volume overload in daily routine and by hierarchical structures within departments. At the same time trainees are expected to develop clinical competency and an adequate level of professionalism that allows them to interact with supervisors, colleagues, staff and patients respectfully  and effectively. While teaching and learning form the essence of education, formalised evaluation by consultants (U.S. terminology: attending physicians) and self-assessment has not been standardised in many European educational medical institutions.

Objective feedback and assessment during education and support of this development process may be hampered by absence of precise directives, uncertainty about how to apply assessment criteria and by lack of objectivism and professionalism of consultants. While feedback is defined as the immediate informal response to an action or behaviour, evaluation represents a formalised way of assessment. A structured evaluation process for medical trainees might be supported and recommended by national educational authorities; the implementation of a standardised system with defined criteria applicable specifically to an adequate assessment of trainees however may vary greatly between institutions, if implemented at all. Objective tracking of personal development might be jeopardised by lack of motivation of consultants, communication problems and lack of efficient standardised evaluation processes.

Evaluations may thus result in non-objective, irregular and non-representative assessments that do not aid the advancement of trainees. In the end personal professional education may become deficient with detrimental effects on patient care, personal development and the public’s view of healthcare professionals.

Project Objective

Training for radiology and nuclear medicine residents at our institution, currently numbering 13 radiology residents, has been a well-structured programme with precisely defined rotations by modalities, daily teaching sessions and formalised lectures, yielding resident satisfaction consistently at or slightly above average according to the Swiss Institute for Medical Education (SIWF). As the supervising authority the SI WF conducts annual surveys on the quality of the teaching institutions by sending out standardised questionnaires to all residents. The forms are filled out voluntarily. Individual answers of participating residents are not published; the results for each institution however are posted publicly on the website siwf.ch in German or French.

While our department consistently scored above national average for the years 2011, 2012 and 2013 on a six-point scale in the categories “overall assessment” and “implementation of evidence-based medicine”, in other disciplines such as “teaching culture”, “learning culture and feedback”, “leadership”, “error and safety”, “decision-making” and “communication” our programme reached average or only barely above average marks. Further internal observations revealed a lack of regular consultant-to-trainee feedback. No regular and standardised evaluation system was in place that would be based on direct input by more than one supervising physician; a formalised self-assessment tool was not available. In addition no formal support or guidance for professional behaviour and development existed.

Our goal was to develop a digital evaluation system that allows regular, structured, and transparent evaluations of residents, with the aim to improve self-reflection and objective assessment of trainee performance and professionalism.

System Development and Concept

A configurable intranet-based evaluation system was developed that allows consultants and trainees to rate performance and professional behaviour using a standardized scoring system. We named the system SO CRATES after the ancient Greek philosopher (469 BC – 399 BC), whose method of dialectic inquiry and logical conclusions for the purpose of problem-solving is considered fundamental to the approach of philosophical and scientific thinking in European and other cultures. The ancient philosopher is also cited in Plato’s Apology with his famous statement “The unexamined life is not worth living”, by which he stresses the importance of reflection and critical examination of man’s own life.

At the same time we used “SOCRATES ” as an acronym for “Standardized Online Competency Review and Advancement of Trainee Evaluation System”, which also describes the approach and purpose of the tool.

The system is highly configurable and may be used for various medical specialties and other healthcare professions.

We have configured the program to fit the needs of a radiology residency programme, but many other configurations are possible. Our version includes 7 categories for assessment:

  • Knowledge;
  • Patient care;
  • Report creation;
  • Technical skills;
  • Professionalism;
  • Personal effort;
  • On-call performance.

These categories contain several criteria, each pertinent to the field. Each is to be addressed by a six-point rating scale and additionally with free text comments that all department consultants are encouraged to give (Fig. 1). Specific justifying comments are not only appreciated, but also expected, if exceptionally poor or good marks are given. Consultants give scores anonymously, while free text comments are identifiable. However, this is one of many options where transparency can be modified.


A self-assessment module, containing the same categories and criteria, is displayed next to the consultants’ assessment for comparison of results, once trainees and consultants have completed the evaluation independently of each other. User access can be adjusted to allow for adequate transparency and confidentiality at the same time. For example modality (area)-supervising radiologists have access to results of their area only, while the residency programme director and institutional director have access to results of all areas. A dashboard allows for a quick overview of evaluations to be completed (Fig. 2).


We chose to use a rotation schedule and review process based on areas, which are mostly imaging modalities, but also modality-independent specialties such as emergency or paediatric radiology. However, the evaluation form is the same for all rotations. We determine the rotation schedule for the entire year upfront, during which residents switch rotations on a quarterly basis; monthly, semi-annual or annual evaluation periods are possible. Evaluations are available electronically to be filled out via a personalised account by accessing the institutional intranet during or after each rotation. Additional email reminders are sent out at the end of each quarter, and a deadline is set shortly after each quarter by which all entries have to be completed, in order to allow for timely analysis of results.

Horizontal tracking of individual results in comparison to all other residents allows a quantitative and qualitative assessment of individual performance and is plotted in diagrams (Fig. 3). Longitudinal comparison of personal performance and development over time can be displayed for one modality and in comparison to other areas, when rotations are repeated over time (Fig. 4 and 5). Charts also show the comparison between self-assessment and consultants’ review (Fig. 6).




Quarterly discussions between modality supervisors and residents are conducted individually and confidentially to ensure timely communication of results, provide support and give directions for further development and improvement of performance. Annual reviews, which are mandatory by the SIWF, occur between residency programme director and the individual resident to integrate results of the year in the overall assessment and plan within the training programme and to determine a potential need for adjustment in the rotation schedule or for other modifications.

Evaluations can be saved electronically or printed out by the supervisors along with an automatically generated date and time stamp, signed by resident and supervisor and filed to the resident’s institutional record. Another copy may be signed for the resident’s personal record.

Impact

One year after implementation of SOCRATES there was uniform agreement amongst residents and consultants that the system was reliable and easy to use. No technical failures occurred. A “save for later” function allows for the evaluation process to be paused in a busy routine with no loss of information. However, the process is usually quick, taking between 3 and 15 minutes for each trainee, depending on the length of the individual comments given.

The annual Swiss national satisfaction survey 2014 showed a noticeable increase in the rating for “regular feedback” from 5.0 in 2013 to 5.9 points in 2014 on a six-point scale (max. 6.0), compared to 5.1 points national average, raising our department in the entire category on “learning culture” from 5.1 points in 2013 to 5.6 points in 2014, compared to 5.2 points nationally. No other measures were implemented during the year other than the evaluation system, along with instructions on how to use it, so we relate at least part of this improvement to the use of the new system.

Also other categories such as “Decision Making” and “Error and Safety” were rated higher: Marks for “Error and Safety” increased from 5.2 to 5.6 points, compared to 5.0 and 5.2 points nationally between 2013 and 2014. Even though it is probably not as self-explanatory as “regular feedback” is for the explanation of the improvement of results, the system might have had an influence on how trainees are being informed of errors and how they reflect on them once errors are documented and analysed in a standardized system. However, SOCRATES is not a documentation system for logging operational errors, nor are we using it to rank residents; all results are kept confidential. For its second year of use in 2015 the repeated annual survey revealed steady results, which encouraged us to continue use of the programme in the future.

At the time of submission of this manuscript the system is being implemented at several radiological institutions in Switzerland. We are currently also evaluating the results of the optional consultant evaluation module for 2015, which allows residents to assess all consultants anonymously in order to improve teaching based on constructive feedback without the fear of negative impact for the residents. The consultant evaluation module contains a different set of evaluation criteria than the resident assessment tool, including teaching effectiveness, motivation to teach, resident support, professionalism and other. It is filled out annually, although the evaluation period as well as all the criteria may be modified as well. To guarantee confidentiality and prevent misuse of conduct, this survey, in contrast to the resident assessment tool, is completely anonymised including comments given by the residents, so that no user, including users with a system administrator’s function, may be able to unveil the identity of residents or timeline of submitted evaluations. Consultants have access to their individual results, compared to the group average. Access to these results may also be customised for others, such as institutional or programme directors.

Both the resident and consultant assessment tool are accompanied by separate help pages that provide guidance to the user on how to fill out the evaluation forms. The resident page also gives suggestions on how to conduct the summative periodical discussion and demonstrates with screenshots mocked up examples what the result pages will look like (Figs. 3-6). The help pages can be edited as well to include for example institutional conventions or to post rules or agreements on the conduct of the surveys etc.

During its two years of use SOCRATES has become an integrated part of our residency programme and is being received as a constructive feedback system that is easy to use with very acceptable effort for both residents and consultants. The system may be easily configured to suit the requirements not only of medical specialties other than radiology, but also for other healthcare professions such as nursing or technical staff.

References:

Gardner AK, Scott DJ, Choti MA et al. (2015) Developing a comprehensive resident education evaluation system in the era of milestone assessment. J Surg Educ, 72(4):618-24.

 

Reisdorff EJ, Carlson DJ, Reeves M et al. (2004) Quantitative validation of a general competency composite assessment evaluation. Acad Emerg Med, 11(8): 881-4.

 

Plato, Apology; 38a, Vol. 1, translated by Harold North Fowler; Cambridge, MA, Harvard University Press;London, William Heinemann Ltd. 1966.



Print as PDF
SOCRATES: An Electronic Evaluation System to Advance Medical Trainees SOCRATES: An Electronic Evaluation System to Advance Medical Trainees

No comment


Please login to leave a comment...