Introduction

Section:

ChooseTop of pageAbstractIntroduction <<Materials and MethodsResultsDiscussionConclusionReferencesCITING ARTICLES

The American Veterinary Medical Association Council on Education (COE), the accrediting body for all North American and some foreign veterinary schools, mandates the demonstration of new graduates' clinical competence as follows:

Competency-based education is a growing trend in human and veterinary health professional training. The general goals of competency assessment are that the student first demonstrates knowledge of clinical medicine and also demonstrates the abilities to integrate knowledge from several sources (history, physical examination, records review, laboratory testing), to make sound decisions, and to relay those decisions to clients. The COE does not dictate how schools demonstrate their students' mastery of competencies. The creation of appropriate assessment tools must take into account the ability of those tools to meet these specific learning objectives and must expose health students to a broad sampling of cases to ensure that clinical competence assessment is reliable across the breadth of the discipline.² Many modalities have been created to try to assess competency in health students, including objective structured clinical examinations (OSCEs) in which the student interacts with a standardized patient or client; case logs or portfolios that may or may not contain descriptive and reflective elements; clinical competency examinations that can be completed either on paper, online, or as practical examinations; assessments by faculty and practitioners as students move through a variety of clinical rotations and externships; skills lists; and students' self-assessments.^3–18

Self-assessment can be used for formative or summative evaluation. Formative evaluation is constructive review for the sole purpose of improving subsequent performance. Summative evaluation also includes some aspect of grading. Self-assessment for educational purposes is a natural offshoot of human behavior in other contexts. As social life has evolved, the need for social comparison—that is, the need to compare one's own attributes to those of others—has done so as well.¹⁹ High-performing students, including veterinary students, have been proven to spend a considerable amount of mental energy evaluating and comparing themselves to others.²⁰ This self-evaluation can be channeled to good use for learning and, in an educational setting, it also has non-cognitive benefits including increased student motivation and increased communication between students and teachers as students strive to understand how they can best achieve mastery.²¹

The primary concern with using self-assessment for learning is that human beings are not naturally good at self-evaluation. The ability to assess one's own competence requires many of the same skills that are required to perform other tasks well. As a result, it is difficult for low-performing individuals to judge their own performance accurately. Low-performing individuals might at times overestimate their abilities because of the ambiguity involved in the process.¹⁹ Almost any individual, for example, if asked to rate him or herself as a companion, can find some aspect of the large skill set that makes up companionship at which he or she excels. Accuracy of self-assessment also varies with knowledge of external factors, the type of ability being rated, and the context in which the self-assessment occurs.^5,22,23 High-performing individuals tend to underestimate their skills, perhaps because of social norms against bragging, unrealistically high personal goals, or unrealistic beliefs about others' comparative skills.^9,24 Finally, one might underestimate or overestimate one's skills based on confidence, which has been defined as the willingness to undertake an activity.²³ Many facets of human health professional training have demonstrated that low-performing students overestimate their abilities when they self-assess.^{5–9,11–13,25–30} Veterinary students, like other elite performers, are generally quite self-confident,²⁰ and this confidence can be reflected in an increased self-evaluated level of competence.²³

Our hypothesis is that veterinary students' self-assessment of their performance on a small–animal, internal-medicine clinical rotation will not be representative of faculty assessment, especially in the cases of the highest and lowest performing students, as has been shown in other health professions.

Materials and Methods

Section:

ChooseTop of pageAbstractIntroductionMaterials and Methods <<ResultsDiscussionConclusionReferencesCITING ARTICLES

The study protocol was approved by the Institutional Review Board (IRB) of the authors' home institution. Fourth-year veterinary students were solicited to participate in the research study and assured that their participation was voluntary, anonymous, and confidential. Students were assured that the collected data would not be accessible to faculty and could in no way affect their grades. The only qualifying criterion was the completion of Small Animal Medicine-B (SAM-B), the students' second two-week block of small–animal, internal-medicine clinical rotation. At the conclusion of the rotation, students completed the same online rotation assessment form used by faculty. This assessment form is a rubric that includes anchors denoting specific standards for A–F grades (see Figure 1). Student self-assessments were coded by staff so that the investigators would not be aware of which students participated. Student self-assessed grades were their own; faculty grades for each student were determined using the combined input of all attending faculty and house officers on that rotation. Grades were reported on a standard A to F scale for knowledge, clinical skills, and professionalism, and an overall grade was assigned. Sensitivity and specificity of the student grade for the determining faculty grade were calculated for each competency separately, as were mean and standard deviation. For the purposes of this analysis, A and B grades denote high competence whereas C and D grades denote low competence.

Results

Section:

ChooseTop of pageAbstractIntroductionMaterials and MethodsResults <<DiscussionConclusionReferencesCITING ARTICLES

One hundred students, including fourth-year students from the graduating classes of 2009 and 2010, completed the self-assessment. Of the 100, 88 were from the home institution and 12 were from other schools. Class rank was available for 88 students. Twenty-one students were in the uppermost quartile of their class, 28 in the second quartile, 21 in the third quartile, and 18 in the lowest quartile. Of the 100 students, 45 received an overall grade of A, 42 an overall grade of B, 12 an overall grade of C, and 1 an overall grade of D. The mean grade was 3.3 on a 4-point scale with A equaling 4 points. By way of comparison, the students of the entire class of 2009 (N=84) earned an average grade of 3.1 on their first completion of SAM-B.

The sensitivity and specificity of using student self-assessment to predict the faculty evaluation of C or D (low competency) are shown in Table 1. The sensitivity is the proportion of students assessed by faculty as low-competency students whose self-assessment grade was also C or D. The specificity is the proportion of students assessed by faculty as high–competency students whose self-assessment grade was also A or B. As an example, the sensitivity for knowledge is 11%. This means that of the 19 students who received a C or D grade from faculty for this competency, only 2 (11%) of them would have been identified by self-assessment. Conversely, of the 81 students who received an A or B grade from faculty for knowledge, 74 (91%) would have been identified by self-assessment.

Table 1 Sensitivity and specificity of student self-assessments

To explore the relationship between student self-assessment and faculty assessment further, the mean and standard deviation were calculated for both types of evaluation in all four areas. As Table 2 shows, the average faculty score was lower than the average student self-assessment score and the standard deviation was higher for faculty scores. For all four categories, these differences were statistically significant at the 0.05 level. Means were compared with paired t-tests and variances were compared with F-tests. This indicates that students were more likely to give themselves a grade toward the middle of the range than the faculty, and suggests that students who perform higher are more likely to give themselves lower grades and students who perform lower are more likely to give themselves higher grades.

Table 2 Mean and standard deviation of student and faculty evaluations on a 4-point scale, reported as M (SD)

This tendency is demonstrated by the difference in overall grades. Of the 30 students with one A grade and one B grade, 18 students gave the lower grade than the faculty (60%). However, of the 13 students with one C or D grade and one higher grade, none gave the lower grade than the faculty grade (0%). Using Fisher's exact test, the difference between these proportions is statistically significant (p=.0005). The differences in the proportions for the knowledge and clinical skills scores are also statistically significant (p=.0075 and p=.019); for professionalism, too few low grades were reported to detect any differences. Figure 2 displays these differences graphically.

Figure 2 Summary of student and faculty scores for each competency. Line width corresponds to number of students who fit that pattern. Black lines show students who had faculty evaluations of C or D but self-assessments of A or B.

Discussion

Section:

ChooseTop of pageAbstractIntroductionMaterials and MethodsResultsDiscussion <<ConclusionReferencesCITING ARTICLES

These data support formerly published studies that document low-performing students' overestimation of skills by self-assessment when compared to expert assessment. This was a voluntary sampling of students and it is possible that this was not a representative population. Objective data, including grade distribution and the number of participating students from each quartile, suggest that this was indeed a representative population. It is nevertheless possible that because participation in this study was voluntary only students who thought they were doing well volunteered, which would explain the low sensitivity. Conversely, it may be that the sample population had greater self-awareness than the larger population, which would have led to even more significant variation between self-assessed and instructor-assessed scores if the entire population had participated.

This study did not evaluate the reported effect of gender. A study evaluating human medical students in a surgery rotation documented the consistent overestimation of abilities by male students and consistent underestimation of abilities by female students.²⁵ This might be the result of social norms suggesting that women are taught not be assertive or boastful.³¹ This effect was not evaluated in this study because students were identified only by number and because odds are very high that most of the participants were female considering the current gender inequity in veterinary medical training.

Expert evaluation has been proven to be most valuable when the student is assessed by a variety of instructors, all of whom are looking for different aspects of the skills described.^7,17 In this study, while one clinician was the instructor of record for each rotation, the students' grades were compiled from individual evaluations of all the faculty and house officers with whom they had worked, a process that increased the accuracy of instructor assessment.

A related concern is the “halo effect,” defined as the tendency to rate students higher based on single attributes that receive undue weight. Because all of these students are chosen for admission from a much larger pool, there might be a tendency to rate them highly due to the instructors' innate belief in the high performance of all students in their program.⁴ This effect is especially significant to the general public, who expect experts to accurately identify individuals in health professional training who are not yet ready to practice. As a result of these same concerns, it is evident that students' self-assessment of competence is unsuitable as the primary or sole evidence of their skills. Again, public safety requires competence to be accurately determined before individuals are licensed to practice veterinary medicine.

Veterinary medical students need to develop the ability to recognize strengths and weaknesses in clinical practice in order to guide their clinical decision-making and career development, the latter of which includes continuing education chosen to enhance students' skills over the course of their entire career. As demonstrated here and elsewhere, self-assessment is unlikely to be an accurate indicator of true ability.

Valid self-assessment requires knowledge of the standards of expected performance and the ability to make informed judgments about how closely one's ability meets those standards.³² Timely instructor feedback coupled with questioning, reflection, and subsequent practice enhances self-assessment skills.^6,13,14 Students should set specific goals against which they can judge their own progress.^6,29 Opportunities for practice must be made readily available as frequency of performance is well correlated with accuracy of self-assessment.¹¹

Conclusion

Section:

ChooseTop of pageAbstractIntroductionMaterials and MethodsResultsDiscussionConclusion <<ReferencesCITING ARTICLES

This study is the first report in veterinary medicine to demonstrate the same inaccuracy of student self-assessment as a measure of ability as has been demonstrated in other health professions. If student self-assessment is to be used as a measure of the levels of competence that are required to meet accreditation standards, certain specific measures to ensure the accuracy of student self-assessment must be employed, including active faculty mentoring of student self-assessment, student goal-setting and reflection, and availability of subsequent opportunities for practice.