Abstract
A retrospective review of the first-year surgical skills competency-based assessment was performed at the Western College of Veterinary Medicine (WCVM) using 6 years of data from 475 students. The cumulative pass rate was 88.2% on first attempt and 99.2% upon remediation. Student gender did not influence overall pass/fail rates, with a failure rate of 11.1% for female students and 10.5% for male students (p = 0.88). Significantly decreased pass rates were associated with identification of the Mayo scissors (p = 0.03), explanation of using Allis tissue forceps (p = 0.002), and performance of a Lembert suture pattern (p < 0.01). An increased pass rate was observed for the cruciate pattern (p < 0.01). No differences were found in pass/fail rates for hand ties (p = 0.80) or instrument ties (p = 0.60). The most common errors occurred with half hitch ties: hand ties (53%) and instrument ties (38%). The most common errors were also recognized for instrument handling (31%) and needle management (20%) during the suture pattern section. The veterinary medical education community may benefit from the evidence-based findings of this research, in terms of understanding student performance across competencies, identifying areas requiring additional mentoring, and determining appropriate competencies for first-year veterinary students.
Surgical skills training,1,2 competency-based assessments,3–5 and entry-to-practice expectations6–8 have evolved and become more clearly defined. The etiology of these changes to veterinary surgical education is multifactorial; the ethical dilemma of using live animals to teach surgical skills in a non-survival laboratory has been a significant factor.9 Cadavers present significant limitations with respect to availability, storage, and decay.9 Biosafety and the physical fidelity of cadavers can be compromised by decay, thereby undermining the education experience.9 Incorporating non-technical competencies such as communication, business skills, and critical thinking into the veterinary curriculum has limited the amount of time available for teaching core surgical skills.3 Finite financial resources to run expensive laboratories, limited faculty time, and the American Veterinary Medical Association Council on Education’s requirement for outcomes-based assessments have also played a role.1,6
Practicing in a surgical skills lab before performing live-animal surgery has also been shown to be associated with significantly reduced anxiety in fourth-year veterinary students.10 Students perceive anxiety as a negative emotion that is counterproductive to learning.11 An optional surgical skills laboratory was documented to be associated with faster total ovariohysterectomy and incision closure times in third-year students compared with those having limited or no exposure with this training opportunity.3,4 In human medicine, short, intensive workshops and modules to provide hands-on surgical experience to undergraduates have been incorporated into the curriculum at different times: first year,12 third year,13 and graduation14; other workshops are specified only as undergraduate.15 Non-surgical clinical and procedural skills training modules have also been reported.16,17 Basic surgical, clinical, and procedural skills laboratories set the foundation for positive learning environments that lead to quantifiable improvements in student outcomes in both veterinary and human medicine curricula.
Most veterinary schools provide basic surgical skills training before performing live surgery. At the WCVM, we teach basic surgical skills in the first, second, and third year of the veterinary curriculum to give students a chance to practice before performing elective surgery. These entry-to-practice competencies, primarily spays and neuters, are performed in the third and fourth years. We conducted this retrospective study of first-year student performance to review our successes and potential opportunities for improvement. The main objective was to establish appropriate surgical skills competencies for first-year veterinary students. No differences in pass/fail rates and common errors were hypothesized to exist for each of the three main sections of our competency-based assessment: surgical instrumentation, knot tying, and suture patterns.
An internal review and the secondary use of student data for research and publication was approved by the University of Saskatchewan’s Behavioral Research Ethics Board (BEH #17–254).
A web-based independent learning formata was introduced in 2010 as part of the first-year curriculum. Three surgical skills labs with learning outcomes are provided on the website for students to review and so they can practice these skills prior to eight weekly optional tutorial sessions (Box 1). A low-cost 20 × 30 cm square foam block with felt overlay and standard basic surgical and suture instruments were provided for self-directed practice at home or during these tutorial sessions. Additional student support included a faculty-to-student ratio of 1:20 for personalized instruction and feedback during tutored sessions. After eight weekly sessions, a mandatory 10-minute pass/fail assessment was administered, with the opportunity for remediation provided 1–2 weeks later for unsuccessful students (Box 2). Successful completion of this basic surgical skills course was required for promotion from the first to the second year of the Doctor of Veterinary Medicine (DVM) program.
Lab 1: Identify common surgical instruments, understand how and when they are used, and be able to hold and use these instruments correctly.
Lab 2: Identify and perform the common surgical knots and be able to choose an appropriate knot to use based on the indications for each knot.
Lab 3: Perform the commonly used suture patterns and understand when their use is appropriate.
Section 1 (Lab 1): Surgical Instruments
Students were asked to identify, demonstrate the appropriate hold of, and explain the use of one surgical instrument out of the following: Adson-Brown forceps, Allis tissue forceps, Backhaus towel forceps, rat tooth forceps, Crile hemostat, Kelly hemostat, Halstead mosquito hemostat, Rochester-Carmalt hemostat, Mayo Hegar needle driver, Olson Hegar needle driver, suture (general purpose) scissors, Mayo scissors, or Metzenbaum scissors. The selection of 1 surgical instrument out of 13 options was at the discretion of the examiner. The criteria for assessment were correct identification, demonstration of appropriate hold, and accurate explanation of use of the surgical instrument.
Section 2 (Lab 2): Knot Tying
Hand tie—Students were asked to perform a hand tie. The selection of right- or left-handed tie, and one- or two-handed tie was at the discretion of the student. The rationale for permitting student selection of the hand tie was to promote development of a consistent, well-executed hand tie for entry-level practice, and to accommodate for differences in handedness. The criteria for assessment were consistent hand motions, efficiency, and creation of a secure square knot.
Instrument tie—Students were asked to perform an instrument tie. The selection of a secure square knot (four throws) or surgeon’s knot (surgeon’s throw and four square throws) was at the discretion of the examiner. The criteria for assessment were correct performance of the instrument tie and a resulting secure (square or surgeon’s) knot.
Section 3 (Lab 3): Suture Patterns
Students were asked to perform one suture pattern out of the following: Connell, Cushing, cruciate, Ford interlocking, horizontal mattress, Lembert, near-far-far-near, simple continuous, and vertical mattress. The selection of the suture pattern was at the discretion of the examiner. The criteria for assessment were proper execution of the pattern, correct instrument holds and use, and proper needle management. Criteria not assessed included spacing, bite size, and suture tension.
Statistical analysis of the data compared student pass/fail rates across types of knot tying (hand and instrument ties); suturing (9 patterns); and instrument recognition, explanation of use, and demonstration of hold. Chi-square (χ2) analyses (α = 0.05 significance) were initially performed, and standardized residuals were used for post hoc analysis when a significant chi-square was found. Descriptive statistics were calculated for the different types of error rates. The potential for student gender differences in pass/fail rates was examined using chi-square analyses (α = 0.05 significance).
An overview of the student data set has been summarized (Table 1). Between 2010 and 2016, 475 first-year students were enrolled in a mandatory competency-based surgical skills lab. The competency exam scoring sheets for the 2013 year were not available due to a clerical error and premature data deletion. The cumulative pass rate was 88.2% on first attempt, 99.2% on second attempt, and 100% on third attempt. This resulted in a total of 535 assessments available for analysis. All sections of the examination had a pass or fail, but some did not record the requested task on the scoring sheets. Therefore, the total number of assessments per section may be smaller than 535.
|
| Year | One attempt | Two attempts | Three attempts | Total students | Total no. of assessments |
| 2010 | 54 | 25 | 2 | 81 | 110 |
| 2011 | 69 | 9 | 1 | 79 | 90 |
| 2012 | 77 | 2 | 1 | 80 | 84 |
| 2014 | 67 | 11 | 0 | 78 | 89 |
| 2015 | 75 | 5 | 0 | 80 | 85 |
| 2016 | 77 | 0 | 0 | 77 | 77 |
| Grand Total | 419 | 52 | 4 | 475 | 535 |
| Cumulative (%) | 88.2 | 99.2 | 100 | – | – |
Successful completion of the competency-based assessment was mandatory (100% attendance). Overall mean student attendance for each voluntary proctored session was 54.2% (21.3%–100%).
Of the 475 students, 85.9% were female (408) and 14.1% were male (67). Female and male students had overall failure rates of 11.1 % and 10.5%, respectively. Student gender was not associated with the overall pass/fail rates (χ2 = 0.023, degrees of freedom = 1, p = 0.88).
An overview of the pass/fail rates for the recognition, demonstration of appropriate hold, and explanation of use for surgical instruments has been summarized (Table 2).
|
| Instrument recognition | Instrument hold | Explanation of instrument use | ||||
| Fail | Pass | Fail | Pass | Fail | Pass | |
| Adson-Brown forceps | – | 67 | – | 67 | – | 67 |
| Allis tissue forceps | 2 | 45 | – | 47 | 3* | 44* |
| Babcock tissue forceps | – | 7 | 1 | 6 | 2 | 5 |
| Backhaus towel forceps | – | 28 | – | 28 | – | 28 |
| Rat tooth forceps | – | 8 | – | 8 | – | 8 |
| Crile hemostat | – | 8 | – | 8 | – | 8 |
| Kelly hemostat | – | 20 | – | 20 | – | 20 |
| Halstead mosquito hemostat | – | 35 | – | 35 | – | 35 |
| Rochester-Carmalt hemostat | – | 5 | – | 5 | – | 5 |
| Mayo Hegar needle driver | – | 168 | – | 16 | – | 16 |
| Olsen Hegar needle driver | – | 13 | – | 13 | – | 13 |
| General purpose scissors | – | 6 | – | 6 | – | 6 |
| Mayo scissors | 7** | 62** | 1 | 68 | 1 | 68 |
| Metzenbaum scissors | 2 | 101 | – | 103 | 4 | 99 |
| > 1 instrument requested | – | 33 | – | 33 | – | 33 |
| Grand Total | 11 | 454 | 2 | 463 | 10 | 455 |
| Percentage | 2.4 | 97.6 | 0.4 | 99.6 | 2.2 | 97.8 |
Note: Bold signifies statistically significant findings.
* standardized residual = −2.03, p < 0.05; χ2 = 37.07, degrees of freedom = 16, p = 0.002
** standardized residual = −5.45, p < 0.01; χ2 = 27.59, degrees of freedom = 16, p = 0.03
The overall pass rate for recognition of examiner requested instruments was 96.7%. Of those students failing this section of the assessment, 63.6% could not identify the Mayo scissors. On post hoc analysis, students were significantly more likely to fail the competency exam when asked to choose the Mayo scissors from a tray of instruments (standardized residual = −5.45, p < 0.01; χ2 = 27.59, degrees of freedom = 16, p = 0.03). According to written examiner feedback, the seven failures associated with misidentification of the Mayo scissors were due to the erroneous selection of general purpose (suture) scissors (4), Metzenbaum scissors (2), or no recorded feedback (1).
The overall pass rate for demonstration of an appropriate instrument hold was 99.6%. Although the chi-square was initially significant, all post hoc standardized residuals were less than 1, indicating a lack of statistical significance (χ2 = 40.23, degrees of freedom = 16, p = 0.0007). The initial significance of the chi-square may be due to the low expected values in failure rates. Despite an absence of failures during this section of the assessment, there were two written examiner feedback comments, including a reminder to use the third (ring) finger not the second (middle) finger in the ring of the Babcock tissue forceps, and a comment that the tips of the curved Mayo scissors should always be directed away from the student’s body during demonstration.
The overall pass rate for explanation of surgical instrumentation use was 97.8%. Of those students failing this section of the assessment, 30% could not explain the use of the Allis tissue forceps. On post hoc analysis, students were significantly more likely to fail the competency exam when asked to explain the use of the Allis tissue forceps (standardized residual = −2.03, p < 0.05; χ2 = 37.07, degrees of freedom = 16, p = 0.002). According to written examiner feedback, the three failures associated with Allis tissue forceps were related to mistaking the use of this instrument with that of the Babcock tissue forceps (2) and to not being able to explain the use of the Allis tissue forceps in general (1).
The overall pass rate for hand ties was 96.4% (Table 3). There were no observed effects on pass rates when students performed their choice of four possible hand ties (χ2 = 0.99, degrees of freedom = 3, p = 0.80).
|
| Fail | Pass | |
| Left one hand | 9 | 264 |
| Left two hand | 1 | 24 |
| Right one hand | 4 | 126 |
| Right two hand | 1 | 28 |
| Grand Total | 19 | 516 |
| Percentage | 3.6 | 96.4 |
According to written examiner feedback, the following errors were identified. Half hitch ties (resulting from unequal tension applied to the two suture ends, typically by pulling vertically upwards on one end of the suture) were the most commonly made error associated with hand ties (53%). Starting with the incorrect strand and inefficiency were tied as the second most commonly made error (13%). Less common errors were as follows: laying down the suture ends between individual throws (7%), being unable to initiate a hand tie (4%), knot loosening between throws (4%), combining more than one way to perform hand tie (2%), creating granny knots (2%), and changing hands during the one-handed tie (2%).
The overall pass rate for instrument ties was 95.9% (Table 4). No effects on pass rates were observed when students performed an examiner-chosen secure square knot or surgeon’s knot (χ2 = 0.28, degrees of freedom = 2, p = 0.60).
|
| Fail | Pass | |
| Secure square knot | 7 | 234 |
| Surgeon’s knot | 13 | 260 |
| Grand Total | 22 | 510 |
| Percentage | 4.1 | 95.9 |
According to examiner written feedback, the following errors were identified. Half hitch ties (resulting from unequal tension applied to the two suture ends, typically by pulling vertically upwards on one end of the suture) were the most commonly made error associated with instrument ties (38%). Performing the incorrect knot (secure square knot vs. surgeon’s knot) was the second most commonly made error (21%). Less common errors were as follows: improper knot tying (16%), improper instrument handling (6%), granny knots (8%), improper needle handling (5%), examiner’s perception of inefficiency (3%), and improper suture handling (3%).
The overall pass rate for the suturing portion of the assessment was 97.4% (Table 5). None of the students who were asked to perform a cruciate pattern failed this section of the assessment. On post hoc analysis, it was seen that students were significantly more likely to pass the competency exam when asked to perform a cruciate pattern (standardized residual = 2.51, p < 0.05; χ2 = 23.46, degrees of freedom = 9, p < 0.01). Of the students asked to perform a Lembert pattern, 5.9% failed. Of those students who failed the suturing section of the assessment, 46.2% had been asked to perform a Lembert pattern. On post hoc analysis, students were significantly more likely to fail the competency exam when asked to perform a Lembert pattern (standardized residual = −3.01, p < 0.01; χ2 = 23.46, degrees of freedom = 9, p < 0.01).
|
| Fail | Pass | |
| Connell | 1 | 9 |
| Cushing | 1 | 23 |
| Cruciate | 0* | 85* |
| Ford interlocking | 2 | 112 |
| Horizontal mattress | – | 1 |
| Lembert | 6** | 95** |
| Near-far-far-near | 2 | 48 |
| Simple continuous | 1 | 63 |
| Vertical mattress | – | 50 |
| Grand Total | 13 | 486 |
| Percentage | 2.6 | 97.4 |
* standardized residual = 2.51, p < 0.05; χ2 = 23.46, degrees of freedom = 9, p < 0.01
** standardized residual = −3.01, p < 0.01; χ2 = 23.46, degrees of freedom = 9, p < 0.01
According to written examiner feedback, the following errors were identified. Inappropriate instrument handling was the most commonly made error (31%). Improper needle handling was the second most commonly made error (20%). Less common errors were as follows: improper performance of pattern (15%), improper knot tying (14%), examiner’s perception of inefficiency (13%), and performance of a different suture pattern other than the one requested (7%).
To the authors’ knowledge, this is the first veterinary publication to outline the results of a retrospective study evaluating pass/fail rates and common errors associated with a first-year surgical skills laboratory for veterinary students. While the pass rate on first attempt was 88.2%, this study revealed several areas for program improvement. We identified areas to enhance student learning by emphasizing the differences between Mayo and Metzenbaum scissors, the specific indications for using Allis versus Babcock tissue forceps, and the causes of half hitch and granny knots. We also established areas to optimize our student assessment, including pre-selection and randomization of each assessed skill, to eliminate examiner selection bias, and re-evaluation of appropriate competencies for first-year veterinary students. For example, the requirement to perform an easily mastered cruciate pattern may be more appropriate for first-year students, while a more challenging Lembert pattern could be assessed in subsequent years.
This internal review highlights that our first-year basic surgical skills laboratory successfully teaches and assesses knot tying. Similar pass/fail rates were observed with both hand ties and instrument ties. It is important to know that neither left- or right-handedness nor one- or two-handed ties influenced student outcomes. It is the authors’ belief that first-year students should continue to select the hand tie performed, as consistently performing any one hand tie successfully is better than confusing and combining multiple hand tie techniques. It would also be remiss to discourage first-year students from relying on natural tendencies when developing their hand-tying skill set. Students are encouraged to learn the other ties as they progress through their development. With respect to instrument ties, it is also important to know that neither secure square nor surgeon’s knots were associated with different pass/fail rates. It is fair to evaluate first-year students on both secure square and surgeon’s knots, despite perceptions that the latter may be more difficult to learn.
Significant limitations of this study must be acknowledged. The test methodology is a retrospective review of student data and not a prospective intervention. Taking into account a single point in time (the first year), our results do not assess learner perceptions of the laboratory, skill retention, and future professional habits, such as how likely it is that students will practice these skills before live surgery. In addition, the examiners selected the specific instrument tie, suture pattern, and surgical instrument, which introduces selection and examiner bias. Additionally, a more comprehensive scoring rubric would have helped capture examiner feedback. In a few instances, multiple examiner comments were recorded, making it difficult to retrospectively determine which of the recorded errors resulted in failure of a specific section of the assessment. In addition, during 33 assessments, more than one surgical instrument was requested, possibly representing a last-minute change, a transcription error, or the examination of students who were performing well on multiple instruments, as all 33 assessments resulted in a pass. These limitations may limit the applicability of this study’s findings to other institutions.
In conclusion, the web-based learning format, the optional eight weekly laboratories, and the mandatory competency-based assessment provide students with a model that encourages self-directed basic surgical skill development early in their professional careers. The mandatory competency-based assessment provides a valuable individualized, instructional opportunity for redirection and reteaching when performance gaps are identified. Future studies prospectively evaluating skill retention, student perception, examiner effects, year effects, and future habits relating to the adoption of skill practice before live surgery are warranted.
Acknowledgment
The authors would like to thank Dr. Sheryl Mills with the Gwenna Moss Centre for Teaching and Learning Effectiveness for her editorial support. Thank you to the veterinary students, staff, and faculty involved in the surgical education program at the WCVM.
a VSAC, https://wcvm.usask.ca/vsac205/
| 1. | Smeak DD. Teaching surgery to the veterinary novice: the Ohio State University experience. J Vet Med Educ. 2007;34(5):620–7. https://doi.org/10.3138/jvme.34.5.620. Medline:18326773 Link, Google Scholar |
| 2. | Smith FCT, Greenwood SR. Modern ways to enhance surgical teaching skills. Surg (UK). 2012;30(9):471–6. Google Scholar |
| 3. | Carroll HS, Lucia TA, Farnsworth CH, et al. Development of an optional clinical skills laboratory for surgical skills training of veterinary students. J Am Vet Med Assoc. 2016;248(6):624–8. https://doi.org/10.2460/javma.248.6.624. Medline:26953914 Medline, Google Scholar |
| 4. | Zeugschmidt EL, Farnsworth CH, Carroll HS, et al. Effects of an optional clinical skills laboratory on surgical performance of third-year veterinary students. J Am Vet Med Assoc. 2016;248(6):630–5. https://doi.org/10.2460/javma.248.6.630. Medline:26953915 Medline, Google Scholar |
| 5. | Schnabel LV, Maza PS, Williams KM, et al. Use of a formal assessment instrument for evaluation of veterinary student surgical skills. Vet Surg. 2013;42(4):488–96. https://doi.org/10.1111/j.1532-950X.2013.12006.x. Medline:23581861 Medline, Google Scholar |
| 6. | Hill LN, Smeak DD, Lord LK. Frequency of use and proficiency in performance of surgical skills expected of entry-level veterinarians by general practitioners. J Am Vet Med Assoc. 2012;240(11):1345–54. https://doi.org/10.2460/javma.240.11.1345. Medline:22607603 Medline, Google Scholar |
| 7. | Smeak DD, Hill LN, Lord LK, et al. Expected frequency of use and proficiency of core surgical skills in entry-level veterinary practice: 2009 ACVS core surgical skills diplomate survey results. Vet Surg. 2012;41(7):853–61. https://doi.org/10.1111/j.1532-950X.2012.00978.x. Medline:22381004 Medline, Google Scholar |
| 8. | Bowlt KL, Murray JK, Herbert GL, et al. Evaluation of the expectations, learning and competencies of surgical skills by undergraduate veterinary students performing canine ovariohysterectomies. J Small Anim Pract. 2011;52(11):587–94. https://doi.org/10.1111/j.1748-5827.2011.01120.x. Medline:22026742 Medline, Google Scholar |
| 9. | Millard HAT, Millard RP, Constable PD, et al. Relationships among video gaming proficiency and spatial orientation, laparoscopic, and traditional surgical skills of third-year veterinary students. J Am Vet Med Assoc. 2014;244(3):357–62. https://doi.org/10.2460/javma.244.3.357. Medline:24432969 Medline, Google Scholar |
| 10. | Langebæk R, Eika B, Jensen AL, et al. Anxiety in veterinary surgical students: a quantitative study. J Vet Med Educ. 2012;39(4):331–40. https://doi.org/10.3138/jvme.1111-111R1. Medline:23187026 Link, Google Scholar |
| 11. | Langebæk R, Eika B, Tanggaard L, et al. Emotions in veterinary surgical students: a qualitative study. J Vet Med Educ. 2012;39(4):312–21. https://doi.org/10.3138/jvme.0611.068R1. Medline:23187024 Link, Google Scholar |
| 12. | McAnena PF, O’Halloran N, Moloney BM, et al. Undergraduate basic surgical skills education: impact on attitudes to a career in surgery and surgical skills acquisition. Ir J Med Sci. 2018;187(2):479–84. https://doi.org/10.1007/s11845-017-1696-7. Medline:29043542 Medline, Google Scholar |
| 13. | Morris M, Caskey R, Mitchell M, et al. Surgical skills training restructured for the 21st century. J Surg Res. 2012;177(1):33–6. https://doi.org/10.1016/j.jss.2012.03.060. Medline:22534253 Medline, Google Scholar |
| 14. | Singh P, Aggarwal R, Pucher PH, et al. An immersive “simulation week” enhances clinical performance of incoming surgical interns improved performance persists at 6 months follow-up. Surgery. 2015;157(3):432–43. https://doi.org/10.1016/j.surg.2014.09.024. Medline:25633735 Medline, Google Scholar |
| 15. | Shindholimath, VV, Goyal A, Anurag S, et al. Teaching and assessment of surgical skills through simulation in surgical training. Indian J Surg [serial on the Internet]. 2005 [cited 2019 Jan 19];65(6). http://hdl.handle.net/1807/22996. Google Scholar |
| 16. | Herrmann-Werner A, Nikendei C, Keifenheim K, et al. “Best practice” skills lab training vs. a “see one, do one” approach in undergraduate medical education: an RCT on students’ long-term ability to perform procedural clinical skills. PLoS One. 2013;8(9):e76354. https://doi.org/10.1371/journal.pone.0076354. Medline:24086732 Medline, Google Scholar |
| 17. | Mileder L, Wegscheider T, Dimai HP. Teaching first-year medical students in basic clinical and procedural skills—a novel course concept at a medical school in Austria. GMS Z Med Ausbild. 2014;31(1):Doc6. http://doi.org/10.3205/zma000898. Medline:24575157 Medline, Google Scholar |
