Figure 19.1 Key components of best practice in assessment.
MORE ATTENTION IS NOW PAID TO ASSESSMENT OF CLINICAL COMPETENCE AND ON-THE-JOB ASSESSMENT
A range of reliable, valued and practical tools are now available to assess a student’s clinical competence.
Clinical competence is the bedrock of safe, efficient and effective patient care. Support for this statement is documented in the literature: a well-conducted interview and physical examination lead to the correct clinical diagnosis in more than 80 per cent of patient encounters (Peterson et al. 1992). This holds true even in the face of advanced technology offering a wide range of sophisticated diagnostic tools. Clinical competence is, therefore, the most important outcome of any medical training programme. The responsibility to provide evidence that such expertise has been acquired, and is practised in the workplace, lies with universities and medical licensing bodies.
Over the past 40 years a range of tools for the assessment of clinical competence, in both test settings as well as the workplace, have been developed. This chapter does not provide detailed descriptions of each of these tools since there are many in the literature; three useful references are included for the reader who wishes to pursue the topic in more detail (Norcini and Burch 2007; Kogan et al. 2009; Norcini 2010).
For the purpose of this chapter it is essential to recognise that ‘best practices’ are always located in real-life (authentic) settings that are subject to context-specific needs and limitations, i.e. ‘best practices’ are not located in an ‘academic vacuum’ free of limitations and needs. For the purpose of this chapter the author identifies three key components of ‘best practices’, as shown in Figure 19.1:
1 the efficient and innovative use of locally available resources;
2 a clear focus on local health needs;
3 optimal use of favourable educational climates to introduce innovations and improve practice.
In this chapter, examples of ‘best practices’ in the assessment of clinical competence are highlighted using case studies and examples from the literature. The chapter aims to distil the key principles demonstrated by the examples drawn from around the world.
Tools used to assess clinical competence
Over the past four decades a variety of tools have been developed to assess clinical competence. In 1990 George Miller used the figure of a pyramid to describe four categories of assessment of clinical competence: the lowest level of the pyramid tests knowledge (knows) followed by competence (knows how).
The upper levels of the pyramid focus on what trainees are able to demonstrate in a test or examination (shows how) and what trainees actually do in the workplace (does). The literature makes a distinction between these two settings by referring to ‘competence’ as behaviour observed in a test setting and ‘performance’ as behaviour in the workplace. This confuses clinicians, who refer to clinical proficiency as ‘competence’ and only differentiate between the assessment of competence in an artificial setting (test or examination) or in an authentic setting (the workplace). In order to avoid confusion, the author uses the term competence to refer to observed clinical proficiency, which is either assessed in a test setting or in the workplace, more recently known as workplace-based assessment (WPBA).
Strategies assessing clinical competence in a test environment include the objective structured clinical examination (OSCE) and the objective structured long case examination record (OSLER). Tools used for WPBA can be grouped into four broad categories:
1 records reflecting work experience, such as log books or clinical encounter cards (CECs);
2 observed single-patient encounters, such as the mini-clinical evaluation exercise (mini-CEX), direct observation of procedural skills (DOPS) and clinical work sampling (CWS);
3 discussion of clinical cases managed using chart-stimulated recall (CsR), also known as case-based discussions (CbD) in the UK;
4 feedback on routine clinical practice from peers, colleagues and patients collected by survey and collated to provide multi-source feedback (MSF). Information derived from peers is usually captured using the mini-peer assessment tool (mini-PAT) while team feedback and patient surveys provide complementary information to build a holistic picture of the trainee in the workplace.
Designing an assessment system
Before addressing the selection of appropriate assessment instruments the reader should appreciate that any one instrument cannot assess all the competencies required of trainees. A compilation of instruments spanning the spectrum of required competencies is needed (Schuwirth and van der Vleuten 2010). This is the foundation of a balanced, comprehensive assessment system, which limits the impact of error variance due to sampling (number and scope of test items), assessors (leniency, halo effect and shrunken scope of scores), context specificity and other assessment biases embedded in gender, age, ethnicity, language and culture.
The UK case study by Capey and Hays describes a comprehensive assessment system, which forms part of the Foundation Programme used to train new medical graduates in the UK.
Case study 19.1 The use of workplace-based assessment in the UK Foundation Programme
The UK Foundation Programme (UKFP) was reviewed and initiated in 2005 as a response to the Modernising Medical Careers (MMC) initiative, whose purpose was to make postgraduate specialty training in the UK both more flexible and potentially shorter (Department of Health 2004).
The UKFP is a 2-year supervised training period undertaken immediately after basic medical education that consolidates and translates learning (knowledge and skills) to the workplace environment, ensuring that entrants to postgraduate speciality training are well prepared for a shorter and more specialty-specific phase. Key to this preparation are the assessments to determine readiness to progress, provide feedback to the trainees (Davies et al. 2009) and identify any ‘doctors in difficulty’ who require further training and assessment. The UKFP assessments were designed to maximise the amount of feedback that each trainee would receive during the first stages of postgraduate training, through using frequent assessments from multiple perspectives and multiple methods. The onus is on trainees to collate a portfolio of assessments as evidence that they are ready to progress. The individual assessment tools used are WPBA, including the mini-CEX, DOPS, CbD and mini-PAT. All of these individual instruments have been demonstrated to provide a valid measurement of elements related to competence in clinical medicine and were used formatively to provide feedback after each encounter to foster professional development of the trainee.
Despite the theoretical advantages of this approach, some practical issues have arisen during implementation, mainly due to the scale of the change and the substantial logistic challenges involved. A report into the implementation of the UKFP in 2010 found that the WPBA assessments ‘have many attractions but [are] time consuming and faculty training . . . essential’ (Collins 2010: 95). The magnitude of the task of completing the individual assessments was significant, with 250,000 clinical assessments being carried out between August 2008 and August 2009. Clinicians conducting the assessments needed training in the new WPBA methods, as providing mostly constructive feedback to colleagues, with at times decisions that might affect progression, is a complex task that requires specific training in educational supervision and use of the WPBA tools (Carr 2006). Achieving this level of training across such a large system took much longer and consumed more resources than predicted.
Trainees themselves have expressed mixed views about the value of the WPBA instruments used in the UKFP (Bindal et al. 2011). Amidst busy service workloads for all involved, the main issues cited were problems with finding assessors and assessments being completed when required. It was reported that many assessments were completed retrospectively or by more junior and relatively untrained colleagues. The view of the trainers was that they had become an onerous tick-box exercise that had little value (Collins 2010).
In conclusion, the individual assessment elements used in the UKFP all have individual validity. The use of these tools as a whole programme of assessment should have provided a well-situated and valid picture of the trainee’s clinical competence. However significant implementation issues were encountered and the programme of assessments was trivialised by logistics, training issues and a misunderstanding of the purpose of the assessments by trainees, assessors and employers.
While the assessment system is laudable in many ways, three major limitations were identified when the system was implemented:
1 the extent of the required scale of implementation was not appreciated; for example more than 250,000 assessments were conducted in the first 12 months;
2 there was insufficient resource allocation for faculty training because the scale of implementation was underestimated;
3 there was insufficient time to conduct the assessment events because of a reduction in consultant staff working hours (new European Union regulation) and a failure to allocate time in staff work schedules to perform these assessments.
This resulted in delays in trainee assessment, with a tendency to conduct assessment events retrospectively as well as the need for trainees to ask junior staff to conduct their assessments. Not unexpectedly, trainees’ scepticism of the programme is now widely recognised. While assessor training and timely completion of assessment events can be addressed, the issue of adequate time to do so is more complex because the external directive is beyond UK control. Furthermore, these additional demands on working time may impact on clinical service delivery and patient care. How the UK addresses this issue is of international relevance because similar training programmes for junior doctors have been implemented elsewhere, for example, in Australia.
Selecting an assessment tool
The selection of an appropriate assessment tool should be guided by the nine simple questions included in Table 19.1. The order in which these questions are addressed is of secondary importance, because the issues are interrelated and the real task is to strike a balance between assessment purpose and need, available resources, the required quantitative (psychometric) properties of the test scores and the prevailing educational climate. The remainder of this chapter describes, and reflects on, case studies and examples from the literature that describe ‘best practices’ in the assessment of clinical competence in context-specific settings.
Successes and challenges of achieving ‘best practices’ in local contexts
The introduction of ‘best practice’ assessment strategies in novel settings is always a challenge. The process can be likened to using a newly released drug, which has been shown to improve patient outcomes in double-blind randomised controlled trials. In the trial setting all potential dangers and biases are addressed by excluding participants who may negatively influence the study results. Once benefit has been shown and new ‘best practice’ has been established, the average doctor is faced with the challenge of using the new drug in real practice, which is fraught with risks, biases and limitations that were specifically excluded during the drug trials. The challenge, to introduce the new drug into routine clinical practice, is akin to the process of introducing novel educational strategies and adapting them for use in local conditions. Indeed, it is at the interface between the adoption and adaptation of assessment practices in new settings that context-sensitive practices emerge, i.e. ‘best practices’. The following examples illustrate the point.
Table 19.1 Questions to guide the selection of tools to assess clinical competence
1 What dimension(s) of clinical competence is/are to be assessed?
2 What are the consequences of the decision to be made?
3 What are the required quantitative properties of the anticipated test scores?
4 What measure of clinical authenticity is required?
5 What resources are available or what can be accessed?
6 Are there identifiable biases to be addressed or avoided?
7 Is the prevailing educational climate open to change and innovation?
8 What is the expected impact on trainee learning and professional behaviour?
9 Are local/regional/national health needs being addressed?
Implementing the OSCE: Brazil vs Argentina
The Objective Structured Clinical Examination (OSCE) is widely accepted as a valid, reliable and effective means of assessing clinical competence in a controlled test environment. While OSCEs have been ‘best practice’ in affluent countries for many years, the use of this resource-intensive assessment instrument is not universal in the developing world. Furthermore, such ‘best practice’ assessment strategies may not be accepted if introduced in unfavourable educational climates. Two examples from South America are useful to consider.
In 1995 an attempt was made to introduce OSCEs at the Faculty of Medicine of Ribeirão Preto of the University of São Paulo, Brazil (de Almeida Troncon 2004). The initiative was met with much resistance from both faculty and students. A lack of human resources and limited knowledge of contemporary assessment practices were two important reasons why the initiative failed. But, the main reason for the negative reaction was an unfavourable educational climate in a conservative medical school that did not have a tradition of using objective examinations to test clinical skills. Since then the Brazilian government has made major progress in improving population health and modernising medical education (Schmidt and Duncan 2004). Teaching and assessment innovations are now being implemented in many institutions in Brazil, demonstrating the power of changed political will, social reform and economic growth, all of which underpin the educational changes witnessed over the past 25 years (Blasco et al. 2008; De Souza et al. 2008).
In contrast to the Brazilian experience, OSCEs were successfully implemented at the National University of Cuyo in Mendoza, Argentina, in 2003 (Vargas et al. 2007). The key reasons for the success in Argentina were:
• A favourable educational climate: the medical school had embarked on a process of curriculum revision and modernisation of assessment practices and so faculty were keen to implement OSCEs.
• The creative use of existing resources: locally available pedagogical and assessment experts were consulted; faculty attended locally run workshops on OSCEs and then trained their colleagues at no additional cost; local-community theatrical actors were trained as standardised patients at minimal cost; existing furniture and clinic space were configured to create OSCE stations.
• A clear idea of needs: national priority health needs were identified and stations were designed so that clinical skills could be assessed in these scenarios; the psychometric requirements of the assessment results of such a high-stakes examination were recognised and achieved with appropriate test design and faculty training.
The mini-CEX
The mini-CEX is a valuable tool to assess clinical competence in the workplace. Currently it is recommended that this tool be used for formative assessment with an emphasis on feedback and the development of action plans to improve trainee performance. Two case studies demonstrate some of the challenges of introducing this tool in routine clinical practice.
In a study conducted at 17 postgraduate cardiology training centres in Buenos Aires, Argentina, the mini-CEX was very well received in terms of feedback to the trainees, but most trainees failed to obtain sufficient encounters to yield reproducible results (Alves de Lima et al. 2007). This study emphasised the importance of feedback rather than focusing on psychometric rigour that was not achievable in the workplace. As previously stated, the challenge is to strike a balance among authenticity, learning value and psychometric rigour. Ultimately the issue can be reduced to two of the questions previously listed: What is the key purpose of the assessment event? and What are the consequences of the event? In this example, the assessment tool provided feedback to improve competence in the workplace and not a judgement decision for academic certification.
The case study from Argentina demonstrates a critical issue relevant to directly observed assessment strategies – the impact of examiner-derived inferences on the perceived competence of the trainee.
Case study 19.2 Role of feedback for inference clarification during a mini-CEX encounter at the Instituto Cardiovascular de Buenos Aires, Argentina
This case study took place during my daily teaching rounds with cardiology residents. One morning during teaching rounds, I performed a mini-CEX with a postgraduage Year 3 resident. Initially he presented the patient case in the hallway, followed by an examination (history taking and physical examination). The patient was a 78-year-old female who had presented to the emergency room 24 hours before with symptoms of heart failure. The history was taken appropriately. During the physical examination at the bedside, the resident palpated the lower limbs for the presence of oedema and to assess the femoral and tibial pulses. He did not, however, remove the sheet from the patient. The physical examination technique of the lower limbs was deemed suboptimal by my standards. Before giving my recommendations regarding his examination skills, I asked the resident why he had not removed the sheet during the physical examination. The resident replied that there were six people around the bed and he wished to avoid embarrassing the woman.
High-level inference has the potential to undermine feedback quality because the potential exists for that feedback to be based on faculty assumptions (Kogan et al. 2011). When I, and other doctors, observe a resident during a clinical encounter, we not only make assumptions and inferences about behaviour, but also provide narrative feedback framed as recommendations. Provoking residents’ self-reflection and developing plans to undertake and evaluate residents’ improvements by the teachers are key factors for effective feedback on clinical performance (Norcini and Burch 2007). Feedback should be more than something exclusively trainer-driven. It has to be a two-way process during which trainers provide comments (recommendations) and trainees reflect on their performance and there is ongoing dialogue that enriches both (Archer 2010). In addition to self-reflection (reflection on action) supported by external feedback (trainers’ recommendations), there has to be linkage with personal goals (action plans) in a coherent, coordinated process rather than a series of unrelated events (Archer 2010). Feedback has been shown to reinforce or modify behaviours and to help learners to reconstruct knowledge, change their performance and feel motivated for learning.
Feedback, an integral part of the mini-CEX, has to be specific and based on what was directly observed during the encounter (Alves de Lima 2008). Questioning, discussing and active testing of the inferences drawn from the observation are crucial before starting the feedback process. Conclusions about resident performance require teachers to use real data (actual residents’ actions), to select behaviours and give meaning that can lead to actions (rating and feedback) (Kogan et al. 2011). Feedback that includes recommendations, self-reflections and action plans functions as a coherent process rather than as a series of unrelated events. The value of feedback is determined by the participants rather than by the instruments used. Understanding how different factors can affect an assessor’s judgements and ratings during direct observation and how they relate to a resident´s learning needs is essential and has to be a key element taken into consideration when organising training sessions. The focus is on the participants instead of on the instruments (Alves de Lima et al. 2011; Pelgrim et al. 2012). Now when I observe residents during rounds, I am aware of engaging them in dialogue around feedback rather than telling things to them as recommendations.
An important, often unnoticed, source of observation bias is the inferences made by clinicians while observing trainees in clinical practice. Since experienced clinicians routinely make inferences about information derived from patients during clinical encounters, it is not surprising that they also do so when assessing trainees. In this case study, a trainee’s attempt to preserve patient privacy and dignity in front of a large crowd of clinicians at the bedside was misinterpreted as a lack of clinical competence. The message of the case study is clear – assessors must be aware of this tendency, which is best avoided, or at the very least, recognised and directly addressed with the trainee before making decisions about observed clinical competence. Factors which impact on directly observed assessment strategies, such as the mini-CEX, include: (1) the frames of reference used by assessors when translating observations into judgements and ratings; (2) the high levels of inference made during the assessment process; (3) the methods by which judgements are transcribed into numerical scores, and (4) factors external to trainee performance. The latter include context (complexity of the patient used in the assessment process, trainee’s prior experience, assessor–trainee relationship) and response to feedback by both the trainee and the assessor (Kogan et al. 2011).
Direct observation of procedural skills
The assessment of procedural skills using DOPS is motivated by the need for safe, effective and cost-efficient patient care. Two case studies highlight issues relevant to using the DOPS assessment tool.
The following case study describes simulation training for surgical skills in a UK setting.
Case study 19.3 Organising and running a simulation training workshop for core surgical trainees in the United Kingdom
It is widely acknowledged that full implementation of the European Working Time Directive has had a detrimental impact on surgical training in the UK, in particular elective training opportunities. As a consequence, doctors who have been selected into core surgical training after foundation years have less operative experience and competence relative to their predecessors. In addition, changes in National Health Service (NHS) culture and service expectations are making it increasingly difficult for trainers to take extra time in theatre to facilitate elective training. Therefore there is a need to facilitate surgical training outside of the operating theatre, to help speed up trainees’ learning curves to competence, so they maximise their operating opportunities.
A 1-day workshop was organised for rotational core trainees by a hospital with excellent teaching and training facilities in a purpose-built education centre with a fully accredited clinical skills wet laboratory and full-time support staff. The aim of the workshop was to provide trainees with the opportunity to practise and develop their laparoscopic skills, and become competent in deploying bowel-stapling devices in a safe, controlled environment. The workshop was limited to a maximum of 12 trainees with a high trainer-to-trainee ratio led by consultant surgeons and senior registrars.
To keep the cost to a minimum for trainees, industrial support was provided in the form of complimentary stapling equipment with a company representative available for technical advice during the day, and faculty voluntarily gave their time. The workshop was advertised via Deanery email distribution lists to regional trainees with an electronic flyer.
Six weeks prior to the workshop, a pre-course questionnaire was emailed to participants. This learning needs assessment served several purposes:
• Gather information on participants’ experience, such as number of operations performed and months in general surgical placements.
• Assess knowledge and encourage relevant study to minimise the requirement for knowledge-based teaching on the day.
• Ask participants why they signed up and to generate their own learning outcomes.
• Enable the faculty to review the participants’ responses within the knowledge-based presentations as a way of reflecting on their learning and providing feedback.
During the workshop the participants were encouraged to complete their own assessment forms (self-assessment and reflection). Further written feedback was added by faculty and signed off as the day progressed.
The workshop began with two interactive PowerPoint presentations covering the principles of bowel anastomosis, stapling techniques and a DVD demonstration of the stapling exercises for the afternoon. After this, the remainder of the day was devoted to supervised simulation training. Participants worked in pairs with laparoscopic box trainers, progressing from simple to more complex tasks. In the afternoon, there were two bowel anastomotic stapling sessions, with all participants performing (and assisting their partner with) three or four different stapling tasks using a realistic fresh cadaveric animal model (porcine bowel). The trainers provided tuition and informal feedback throughout the day. A course certificate was presented to each trainee along with the completed assessment form as structured feedback.
Organising such a workshop was challenging and required excellent communication and coordination with administrative staff, industry representatives, catering staff and faculty. Essential tasks included preparation of course materials, faculty training and briefing, and setting up the clinical skills laboratory. For the first workshop the course organiser did most of the above, including setting up the lab for the morning and afternoon sessions. However, a miscommunication with the manager of the local abattoir led to the organiser having to drive to the abattoir to collect the porcine bowel and then prepare it during a lunch break!
The first workshop was very successful and evaluated extremely positively, but for subsequent workshops a laboratory technician was recruited to assist with setting up. This released the course convenor to facilitate the day more effectively. Arrangements with suppliers must be organised well in advance. Faculty training and briefing are also important, particularly to ensure that formal written reflection and feedback are signed off as the day progresses.
The workshop, as demonstrated in the UK case study, has many good features from an assessment perspective.
• The learning needs of the trainees were identified and expected outcomes were clarified before the workshop – an excellent example of alignment between learning needs, expected outcomes and assessment strategies.
• The workshop provided a balance of theory and practical training opportunities.
• The workshop offered both low- and high-fidelity learning opportunities.
• Trainees were required to provide input about their own performance.
• Written feedback was a formal part of the training programme.
The organisers faced challenges related to logistics and the need for adequate technical and administrative support. These are key issues, which, if overlooked, can mean the difference between success and failure.
In the case study from the Pontificia Catholic University of Chile, the authors provide another example of the innovative use of DOPS in a postgraduate training programme.
Case study 19.4 How to assess trainees’ clinical competence performing endoscopies in a postgraduate residency programme at the Pontificia Universidad Católica de Chile
The Pontificia Universidad Católica de Chile Medical School (PUCMS) implemented several initiatives in order to improve medical education for undergraduate medical students (Sánchez et al. 2008). Those efforts have been recognised in national and international accreditation processes (Sánchez et al. 2010). However, postgraduate medical education is less developed in our institution, and there has been a significant delay transferring successful teaching and learning innovations or assessment instruments developed in undergraduate level to postgraduate residency programmes (PRPs).
The Chilean Society of Gastroenterology carried out a Delphi technique consensus, in order to establish the core competencies of the Chilean gastroenterologist (Riquelme et al. 2010). In this consensus diagnostic upper gastrointestinal endoscopy (UGIE) was considered by an expert panel as the most important and essential procedure for the specialty. This case study focuses on an upper gastrointestinal endoscopy basic training programme (UGIETP) of the gastroenterology PRP at the PUCMS.
The gastroenterology PRP is a 2-year programme and the UGIETP takes place in the first 4 months of the first year of this residency programme. During the first 4 weeks, the trainee receives basic information about the endoscope and training process in a simulated environment. During the next 12 weeks, trainees are trained with real patients in the endoscopic room of an outpatient clinic using low-risk patients and only performing diagnostic procedures under supervision of a personal endoscopic trainer (expert).
Trainees who successfully complete the simulated training stage are allowed to continue their training programme with real patients. They start by taking a brief history and gaining informed consent, explaining to the patient what they are going to do during the UGIE. After sedation the trainee starts the procedure under strict supervision and the trainee’s performance is assessed with a DOPS developed by the trainers and experts, identifying four key features after each procedure and one complete checklist (33 items) at the end of each session. Additionally, efficiency of movements is assessed using the Imperial College Surgical Assessment Device (ICSAD), allowing objective quantification of movements and the path length travelled by each hand (distance measured in metres) (Aggarwal et al. 2006). The educational activity follows the rules of workplace-based assessment and it is really important to keep it authentic (Norcini and Burch 2007). Therefore, the whole teaching activity takes place in a real environment with real patients.
After each procedure the trainer provides effective feedback, including the strengths and achievements of the trainee and weaknesses that should be improved in future procedures. Effective feedback is a key element in this programme and takes place immediately after the UGIE has been performed by the trainee. UGIE could be considered as a meta-competence because the whole procedure involves clinical skills, patient investigation (taking histological samples or urease test to detect Helicobacter pylori infection according to the endoscopic findings), written communication and information-handling skills, ethical aspects and legal responsibilities. For that reason, clinical skills and trainees’ progression through the UGIETP are mainly measured with the four-key-feature DOPS and ICSAD measurement. Other competencies related to UGIE are assessed with the 33-items checklist and portfolio activities. Formative assessment also includes a follow-up process for each patient with biopsies. Residents report the final diagnosis, discuss treatment options and follow the patient, including critical appraisal of their endoscopic reports and reflection about selected patients in their portfolio.
Effective feedback has been identified in a meta-analysis as the most important element in WPBA in terms of the educational impact (Miller and Archer 2010). Effective feedback is immediate, accurate and based on the strengths and weaknesses of the trainee. Moreover, trainees establish an action plan with their trainer for the next session in order to acquire knowledge and improve skills and attitudes related to UGIE performance.
Based on the learning curves of the last ten trainees who followed this UGIETP, they are autonomous after 80 procedures and they significantly reduced their travelled path length of each hand (99.7 m to 60 m; distance measured in metres) measured with ICSAD. Second-year residents and experts (trainers) were also measured and their travelled path lengths were 49.7 m and 17.9 m, respectively (González et al. 2012). We observed different levels of performance because some trainees are more skilled than others. It is important to train them, considering their individual achievements based on their strengths, and explain to them how to improve their performance. All trainees construct their own learning curves and we compared them with cumulative curves of previous cohorts of trainees. Summative assessment is based on their progression through the UGIETP and a reflective portfolio about their learning process. If they successfully complete the UGIETP, they are allowed to continue with the next stage, including therapeutic UGIE and colonoscopy in a more complex inpatient endoscopic unit at the hospital.
Learning objectives related mainly to one dimension of a competence, but sometimes competencies relating to a clinical procedure were more complex than we expected. We needed to combine a wide variety of formative and summative assessment instruments, which were suitable (valid and reliable) for evaluating knowledge, skills and attitudes.
The American Society of Gastrointestinal Endoscopy (ASGE) reported that 130 UGIE are needed to be competent (Adler et al. 2012). Our training programme demonstrated that, if you train residents in a simulated environment followed by sessions with real patients in the endoscopic room, following our training system, trainees are autonomous with only 80 procedures. Moreover, we train residents with low-risk patients (American Society of Anesthesiologists (ASA) classification I or II) and have had no endoscopic complications in more than 1,000 UGIEs performed by trainees.
The activity in the simulated environment and the endoscopy room is aligned with the assessment system and the learning outcomes that the trainees must achieve at the end of the UGIETP. What is more interesting is that, according to the results observed in the first ten residents (two cohorts), we learnt more about what is effective in terms of the training process (teaching activities), assessment process (assessment for learning rather than assessment of what was learnt) and feedback (formative assessment) in our training programme. Based on the results obtained using the Postgraduate Hospital Educational Environment Measure (PHEEM) questionnaire, residents perceived there to be a positive educational environment, highlighting trainers’ quality, protected time for this educational activity and a safe environment in the endoscopy room (Herrera et al. 2012). The four-key-feature DOPS was easy to use after each procedure, and allowed the construction of learning curves to establish trainees’ competence performing UGIE. On the other hand, the 33-item checklist and ICSAD assessed a wider spectrum of competencies and objective quantification of movements, respectively. However, both are time consuming and it is unrealistic to include them after each procedure.
The key learning points of the Chile case study include the following:
• Patient safety is key when teaching procedural skills using real patients. This dilemma often strongly biases training towards simulation settings with little real clinical experience. In this programme, the issue was addressed by allowing trainees to perform endoscopy on low-risk patients identified using an objective risk stratification tool.
• DOPS focused on four aspects of each procedure and a detailed checklist was only completed at the end of each session. This reduced the paperwork between cases and limited the impact on service delivery time.
• DOPS was used in conjunction with another validated procedure-specific rating tool, which improved the quality and objectivity of the assessment process.
The educational impact of workplace-based assessment
The final case study from Saudi Arabia highlights the influence of curriculum change on learning behaviour. At the King Saud University in Saudi Arabia, extensive curriculum innovation was introduced with a focus on integrating WPBA and clinical teaching. Previously, medical students were only allowed to observe patient encounters undertaken by qualified clinical staff and their clinical skills were not assessed in the workplace. Not surprisingly, students placed little value on clinical activities and attendance was poor. Since introducing a suite of WPBA tools, the students have become active participants in the clinical services and appreciate the value of learning and assessment activities in the workplace.
Case study 19.5 Introducing workplace-based assessment in a reformed, undergraduate curriculum at King Saud University, Saudi Arabia
The College of Medicine, King Saud University, was the first medical college in the Kingdom of Saudi Arabia. In line with contemporary changes in medical education, the traditional curriculum of the college has recently undergone many changes to include various innovative strategies in teaching, learning and assessment. For example, clinical teaching has been introduced as early as from the first academic year to make teaching and learning more contextual, interesting and beneficial for the students.
In the past, however, the teachers were more entrenched in the old style of clinical teaching. Usually, clinical teaching took place at the bedside or with students observing clinical teachers at an ambulatory care unit. Thus, objective observation and feedback did not take place. This was contrary to the well-established educational premise that clinical teaching needs good planning, observation of students and continuous feedback (Harden and Laidlaw 2012). Further, the teachers observed that the students gave low priority to attending clinical teaching sessions, as these sessions did not help them in achieving high marks in their final assessment.
In an attempt to find a balance between ensuring that appropriate planning, observation and continuous feedback are practised uniformly throughout all clinical teaching and learning sessions, and that the students take clinical teaching and learning seriously, the college realised that reforming the existing clinical teaching would not be sufficient. In order to remedy this, the college attempted to institutionalise the above good practices of teaching and learning in the clinical setting (i.e. planning, observation and feedback) through assessment. This led to the introduction of WPBA (Carr 2006; Dewi and Achmad 2010).
Three methods of WPBA were agreed by the curriculum committee, to be included in the clinical assessment and teaching: mini-CEX, CbD and DOPS. These methods of assessment and teaching demanded a lot of skills from the tutors to assess and teach their students effectively (Hill and Kendall 2007). Further, the college reckoned that continuous monitoring and evaluation were necessary to prevent the implementation of WPBA being reduced to a tick-box exercise (Bindal et al. 2011).
This situation compelled us to devise methods to improve the faculty skills of applying the above WPBA methods. Many workshops were conducted for faculty to orient them in applying these methods. Feedback from both faculty and students was collected to get their opinion on improving the implementation of WPBA. These feedback and suggestions were applied to upgrade both the ongoing process of workplace-based clinical assessment and teaching, and the faculty development activities on WPBA. The introduction of WPBA also triggered a change in the student perception of clinical teaching and learning. Informal student and staff surveys indicated that students now take clinical teaching and learning much more seriously.
This case study shows how faculty development activities on WPBA, with special emphasis on objective observation and feedback, can be used to institutionalise good practices of clinical assessment and teaching. In addition, it demonstrates that introducing WPBA with the explicit intention of combining assessment with teaching and learning leads to students taking clinical teaching and learning more seriously. Fine-tuning with faculty development activities, based on the continuous evaluation of the implementation of WPBA, is ongoing.
The future of assessment on a global scale
The quantitative properties of assessment scores have been the major focus of attention for far too long. Medical education leaders have made a plea for a broader view of assessment and revision of the traditional measures that are used to determine the quality of assessment practices. In addition, it is also true that the ‘big five’ – the USA, Canada, the UK, Australia and Western Europe – have dominated medical education practices for far too long.
Global economic and political tides are slowly turning and the BRICS member countries (Brazil, Russia, India, China and South Africa) are likely to lead the way. Currently BRICS member countries are home to almost three billion people or about 43 per cent of the world’s population and their combined external trade is worth about $US6 trillion, or 17 per cent of the world total. So, it is plausible that in the coming decade the developing world, led by BRICS, will have a palpable influence on ‘best practices’ in medical education. This chapter already provides insights into current pockets of regional innovation and excellence. Ultimately, the sharing and exchange of local assessment practices that are responsive to culture, language, available resources and health needs may expand our understanding of ‘best practices’ in 21st-century medical education.
Take-home messages
To recap, the transversal principles of ‘best practices’, as demonstrated in this chapter on the assessment of clinical competence, suggest that successful assessment strategies are based on several principles:
• Start planning an assessment event by deciding on the purpose (formative (feedback) or summative (judgement)) and consequence (feedback for improvement or judgement for certification).
• Align assessment practices with learning needs and expected outcomes.
• Attempt to limit biases commonly encountered in WPBA.
• Feedback should:
be based on observed behaviour;
avoid making inferences;
include trainees’ perception of their own performance;
conclude with an action plan to foster further development.
• The quantitative properties of WPBA scores currently limit their use for high-stakes summative purposes.
• ‘Best practices’ are achieved when assessment strategies are adapted to suit local circumstances and address local needs.
Bibliography
Adler, D.G., Bakis, G., Coyle, W.J., DeGregorio, B., Dua, K.S., Lee, L.S., McHenry, L. Jr, Pais, S.A., Rajan, E., Sedlack, R.E., Shami, V.M. and Faulx, A.L. (2012) ‘Principles of training in GI endoscopy’, Gastrointestinal Endoscopy, 75(2): 231–5.
Aggarwal, R., Dosis, A., Bello, F. and Darzi, A. (2006) ‘Motion tracking systems for assessment of surgical skill’, Surgical Endoscopy, 21(2): 339–9.
Alves de Lima, A.E. (2008) ‘Constructive feedback. A strategy to enhance learning’, Medicina (Buenos Aires), 68(1): 88–92.
Alves de Lima, A., Barrero, C., Baratta, S., Castillo Costa, Y., Bortman,, G., Carabajales, J., Conde, D., Galli, A., Degrange, G. and van der Vleuten, C. (2007) ‘Validity, reliability, feasibility and satisfaction of the Mini-Clinical Evaluation Exercise (Mini-CEX) for cardiology residency training’, Medical Teacher, 29(8): 785–90.
Alves de Lima, A., Conde, D., Costabel, J., Corso, J. and Van der Vleuten, C. (2011) ‘A laboratory study on the reliability estimations of the mini-CEX’, Advances in Health Sciences Education: Theory and Practice, 18(1): 5–13.
Archer, J.C. (2010) ‘State of the science in health professional education: effective feedback’, Medical Education, 44(1): 101–8.
Bindal, T., Wall, D. and Goodyear, H. (2011) ‘Trainee doctors’ views on workplace-based assessments: Are they just a tick box exercise?’, Medical Teacher, 33: 919–27.
Blasco, P.B., Levites, M.R., Janaudis, M.C., Moreto, G., Roncoletta, A.F.T., de Benedetto, M.A.C. and Pinheiro, T. (2008) ‘Family medicine education in Brazil: Challenges, opportunities and innovations’, Academic Medicine, 83(7): 684–90.
Carr, S. (2006) ‘The foundation programme assessment tools: An opportunity to enhance feedback to trainees?’, Postgraduate Medical Journal, 82: 576–9.
Collins, J. (2010) Foundation for excellence: An evaluation of the foundation programme, Medical Education England. Online. Available HTTP: http://hee.nhs.uk/wp-content/uploads/sites/321/2012/08/Foundation-for-excellence-report.pdf (accessed 25 March 2015).
Davies, H., Archer, J., Southgate, L. and Norcini, J. (2009) ‘Initial evaluation of the first year of the Foundation Assessment Programme’, Medical Education, 43: 78–81.
De Almeida Troncon, L.E. (2004) ‘Clinical skills assessment: Limitations to the introduction of an OSCE (Objective Structured Clinical Examination) in a traditional Brazilian medical school’, Sa~o Paulo Medical Journal, 122(1): 12–17.
Department of Health (2004) ‘Modernising medical careers: The next steps. The future shape of foundation, specialist and general practice training programmes’, London: Department of Health.
De Souza, P.A., Zeferino, A.M.B. and da Aurelio Ros, M. (2008) ‘Changes in medicine course curricula in Brazil encouraged by the Program for the Promotion of Medical School Curricula (PROMED)’, BMC Medical Education, 8: 54. Online. Available HTTP: http://www.biomedcentral.com (accessed 9 September 2014).
Dewi, S.P. and Achmad, T.H. (2010) ‘Optimising feedback using the mini-CEX during the final semester programme’, Medical Education, 44(5): 509.
González, R., Rodríguez, A., Buckel, E., Hernández, C., Tejos, R., Parra, A., Pimentel, F., Boza, C., Padilla, O. and Riquelme, A. (2012) ‘Systematization of an upper gastrointestinal endoscopy training program for diagnosis in a simulation-based setting and learning curves in real patients’, Revista Gastroenterologia Latinoamericana, 23(4): 191–6.
Harden, R.M. and Laidlaw, J.M. (2012) Essential skills for a medical teacher, Edinburgh: Churchill Livingstone, Elsevier.
Herrera, C., Olivos, T., Román, J.A., Larraín, A., Pizarro, M., Solís, N., Sarfatis, A., Torres, P., Padilla,
O., Le Roy, C. and Riquelme, A. (2012) ‘Evaluation of the educational environment in medical specialty programs’, Revista Médica de Chile, 140: 1554–61.
Hill, F. and Kendall, K. (2007) ‘Adopting and adapting the mini-CEX as an undergraduate assessment and learning tool’, The Clinical Teacher, 4: 244–8.
Kogan, J.R., Holmboe, E.S. and Hauer, K.E. (2009) ‘Tolls for direct observation and assessment of clinical skills of medical trainees. A systematic review’, Journal of the American Medical Association, 302(12): 1316–26.
Kogan, J.R., Conforti, L., Bernabeo, E., Iobst, W. and Holmboe, E. (2011) ‘Opening the black box of clinical skills assessment via observation: A conceptual model’, Medical Education, 45(10): 1048–60.
Miller, G.E. (1990) ‘The assessment of clinical skills/competence/performance’, Academic Medicine, 65(9): s63–7.
Miller, A. and Archer, J. (2010) ‘Impact of workplace based assessment on doctors’ education and performance: a systematic review’, British Medical Journal, 341: c5064.
Norcini, J.J. (2010) ‘Workplace assessment’, In T. Swanwick (ed.) Understanding medical education: Evidence, theory and practice, (pp. 232–45), Oxford: Wiley-Blackwell.
Norcini, J. and Burch, V. (2007) ‘Workplace-based assessment as an educational tool: AMEE guide no. 31’, Medical Teacher, 29(9): 855–71.
Pelgrim, E.A., Kramer, A.W., Mokkink, H.G. and van der Vleuten, C.P. (2012) ‘The process of feedback in workplace-based assessment: Organisation, delivery, continuity’, Medical Education, 46(6): 604–12.
Peterson, M.C., Holbrook, J.H., Hales D.V., Smith, N.L. and Staker, L.V. (1992) ‘Contributions of the history, physical examination, and laboratory investigation in making medical diagnoses’, Western Medical Journal, 156(2): 163–5.
Riquelme, A., de la Fuente, P., Méndez, B., Salech, F., Valderrama, S., Méndez, J.I., Oporto, J., Miquel, J.F., Defilippi, C., Soza, A., Sirhan, N. and Sáenz, R. (2010) ‘Competencies of the Chilean gastroenterologist: Modified Delphi technique’, Revista Gastroenterologia Latinoamericana, 21(4): 437–53.
Sánchez, I., Riquelme, A., Moreno, M., Mena, B., Dagnino, J. and Grebe, G. (2008) ‘Revitalising medical education: The school of medicine at the Pontificia Universidad Católica de Chile’, The Clinical Teacher, 5: 57–61.
Sánchez, I., Riquelme, A., Moreno, R., García, P. and Salas, S. (2010) ‘International accreditation process at a Latin American medical school: A ten-year experience’, Medical Teacher, 32(3): 271.
Schmidt, M.I. and Duncan, B.B. (2004) ‘Academic medicine as a resource for global health: The case of Brazil’, British Medical Journal, 329: 753–4.
Schuwirth, L.W.T. and van der Vleuten, C.P.M. (2010) ‘How to design a useful test: The principles of assessment’. In T. Swanwick (ed.) Understanding medical education: Evidence, theory and practice, (pp. 195–207), Oxford: Wiley-Blackwell.
Vargas, A.L., Boulet, J.R., Errichetti, A., van Zanten, M., Lopez, M.J. and Reta, A.M. (2007) ‘Developing performance-based medical school assessment programs in resource-limited environments’, Medical Teacher, 29: 192–8.