Volume 42 Issue 3, Fall 2015, pp. 259-270

ABSTRACT

Many veterinary curricula use seminars, interactive educational group formats in which some 25 students discuss questions and issues relating to course themes. To get indications on how to optimize the seminar learning process for students, we aimed to investigate relationships between factors that seem to be important for the seminar learning process, and to determine how these seminar factors account for differences in students' achievement scores. A 57-item seminar evaluation (USEME) questionnaire was administered to students right after they attended a seminar. In total, 80 seminars distributed over years 1, 2, and 3 of an undergraduate veterinary medicine curriculum were sampled and 988 questionnaires were handed in. Principal factor analysis (PFA) was conducted on 410 questionnaires to examine which items could be grouped together as indicators of the same factor, and to determine correlations between the derived factors. Multilevel regression analysis was performed to explore the effects of these seminar factors and students' prior achievement scores on students' achievement scores. Within the questionnaire, four factors were identified that influence the seminar learning process: teacher performance, seminar content, student preparation, and opportunities for interaction within seminars. Strong correlations were found between teacher performance, seminar content, and group interaction. Prior achievement scores and, to a much lesser extent, the seminar factor group interaction appeared to account for differences in students' achievement scores. The factors resulting from the present study and their relation to the method of assessment should be examined further, for example, in an experimental setup.

Rapid progression in clinical practice and technological development has raised several challenges for the veterinary profession.1 Lifelong learning skills, such as critical scientific thinking; problem solving; finding, organizing, and analyzing information; decision making; and communication skills are needed to perform the complex tasks of the modern veterinary profession.2 Based on the Pew National Veterinary Education Program, which conducted the “Future Directions for Veterinary Medicine” study,3 and the Royal College of Veterinary Surgeons and the American Veterinary Medical Association Day One Competencies,4,5 veterinary schools have focused on implementing appropriate educational strategies to graduate competent veterinarians who meet these requirements. Active instructional methods have been introduced in curricula, allowing students to be more proactively engaged in their own learning through dialogue, discussion, questioning, and collaborating with peers and teachers.6,7

In introducing these methods, the focus of education can be redirected from an excessive emphasis on the accumulation of information toward the acquisition of the above-mentioned skills. Different curricula in the health profession domain have approached this implementation in several ways, allowing diverse manifestations of active learning strategies in education worldwide.8,9 Some have implemented full-fledged small-group curricula (e.g., problem-based learning curricula), while others combine more traditional teacher-centered large-group lectures with different types of small- and medium-sized group learning methods in a hybrid curriculum.911 Worldwide, the terminology and execution of these small- and medium-sized group learning methods differ (e.g., seminars, tutorials, team-based learning sessions, Socratic dialogs, workshops). In this study, we focus on a group learning method that is characterized by interaction between student peers and a teacher, which we refer to as seminars. Dennick and Spencer12 describe seminars as teaching and learning sessions that are usually midway in size between a small group and a lecture group, and during which there is discussion. Jaarsma11 refined this definition: “a seminar is a learning session facilitated by a content expert during which a group of some 25 students discusses questions and issues emerging from assigned readings, with the underlying idea being that students learn through confrontation with problems of practical relevance.”11(p.1180) The idea of seminars is that they are focused on advanced topics and include interactivity, allowing learners to engage in a deep learning approach and the critical evaluation of evidence.12 It is postulated that during seminars students move from acquiring and reproducing knowledge to seeking meaning through the application of knowledge in assignments and discussion with peers.13,14 By engaging in interactive group work, students elaborate and restructure facts, principles, and concepts, thereby building robust cognitive frameworks12,15,16 that are assumed to be helpful to students applying what they have learned in new situations.17 Another aspect of group learning is that the social interaction in groups might help in acquiring collaborative and communication competencies that are needed for a veterinary and medical professionals.2,18,19 In veterinary educational literature, these kinds of sessions are also sometimes defined as “tutorials.”20

Factors that seem to be important for the seminar learning process are teacher performance, group interaction, and the quality of seminar assignments.10 Other potentially important factors, investigated by qualitative studies,21,22 are the amount and quality of the preparatory materials, the extent of student preparation, students' course schedule, coherence and alignment of different educational methods within a course, group size, and student motivation in the group. The literature does not provide studies on the impact of these seminar learning factors on academic achievement.

For problem-based learning (PBL) curricula in health science institutions, the relationship between group learning methods (often referred to as PBL tutorials) and academic achievement has been investigated.2331 In these studies, various variables were tested for their relation with academic achievement, including prior knowledge for tutorial, tutor functioning, group functioning, problem characteristics, time spent on self-study, students' interest in the subject, and depth and breadth of the reporting in the group. Because of the differences between the learning process in PBL tutorials and seminars (i.e., in PBL tutorials, students are responsible for defining their own learning goals and working in smaller groups with the teacher as facilitator; in seminars, students have assigned preparatory materials and learning goals, work together in larger groups, and have a content expert as teacher), we cannot just assume that these variables are also important for the seminar learning process. A more detailed understanding of the relationships between factors that are important for the seminar learning process and students' achievement scores is therefore needed. These insights may help teachers in a veterinary curriculum to optimize the seminar learning process for their veterinary students.

We conducted a questionnaire study to address the following research questions:

  1. What are the relations between factors that seem to be important for the seminar learning process?

  2. How do these seminar factors account for differences in students' achievement scores?

Educational Context

The study was conducted in the first three years of the six-year undergraduate curriculum of the Faculty of Veterinary Medicine, Utrecht University, the Netherlands (FVMU). About 225 new students enroll every academic year. The integrated curriculum of the first three years is organized around organ systems (circulatory system, hepato-biliary system) and focuses on basic science, clinical science, and practical skills. Seminars, lectures, and practicals are the main educational formats. Contact time between students and teachers takes up 30%–40% of total study time and the remaining time is designated for self-study to prepare for sessions and exams. Assessment consists mostly of written end-of-course exams.

Seminars of around 25 students make up 40%–60% of contact time. The system-based courses comprise around eight seminars on different themes (e.g., “Pathophysiology of jaundice and cholestasis”). For each seminar theme, nine seminars are conducted (225 students per year, 25 students per group), facilitated by different teachers. Attendance of seminars is optional. Students prepare for seminars by completing assigned readings with guiding questions. Preparation is compulsory but students are free to work individually or in groups. Except for duration (105 minutes maximum), there is no prescribed facilitating method for seminars. In practice, teachers mainly use two seminar approaches: some teachers ask students to discuss the seminar questions in subgroups of two to six students, followed by plenary discussion. Other teachers ask students to prepare the questions individually before discussing them with the group.19 Most teachers have attended a two-year faculty development program with personal mentoring. During that program, they learn to promote thinking and to stimulate students to actively participate in discussions. Teachers are expected to have good skills with regard to questioning, listening, reinforcing, reacting, and summarizing. The faculty development program has also focused on designing seminars with questions that provoke discussion and are clinically relevant and academically challenging. A seminar is not a teaching format that can be used by the teacher to give a presentation or explain a specific topic. The task of the teacher is to facilitate student understanding of a topic, helping students to apply their knowledge while working on the seminar questions. In this way, students talk at least as much as their teacher. Students are not given an incentive for participation: the intention is that the instructiveness of participation in the seminar is an incentive in itself and prepares them for the end-of-course exam and their work as veterinarians later on. Seminar student groups remain together for the duration of a semester, but teachers vary depending on the seminar theme. Quality assurance is based on student evaluations of a complete course.

Questionnaire Development

We designed a preliminary questionnaire consisting of 58 items, of which 53 related to aspects affecting seminar learning. These were derived from earlier questionnaire and focus group studies on seminar learning.11,21,22 The questionnaire was piloted among 15 veterinary students, who were asked to judge whether the items were clear, unambiguous, and pertained to one single aspect. The pilot led to rewording of two items and exclusion of one item. The resulting Utrecht Seminar Evaluation questionnaire (USEME) was divided into five sections each containing related items to be evaluated on five-point Likert scales (from 1=strongly disagree to 5=strongly agree). The five sections were “general characteristics of student and seminar” (5 items), “preparation” (17 items), “content of seminar” (23 items), “teacher performance” (8 items), and “overall scores on aspects of the seminar” (4 items). The full questionnaire can be found in Appendix 1.

Data Collection

During the period from April–June 2012, data were collected from a sample of all seminars in the second semester of years 1–3 of the undergraduate curriculum of the FVMU. Although sampling was guided by logistical considerations, we aimed to collect data from at least two out of nine seminar sessions devoted to the same theme. All selected seminars were designed with clear learning goals, preparation assignments, integrated seminar questions, and clear teacher guidelines, with the goal of encouraging students to be active during the session. Different teachers guided them and mainly two teaching approaches were used: plenary discussion of the seminar questions following either individual reflection or group discussion. This resulted in a sample of 80 seminars dealing with 34 different themes relating to six different courses. The seminars were conducted by 36 different teachers. All students attending these seminars were invited to participate in the study. Students were asked to complete the questionnaire immediately after the given seminar to ensure a proper match between answers and seminar. After the teacher had left the classroom, a student assistant handed out the questionnaire with a letter explaining the aims of the study and informing students that participation was voluntary and would take about 10 minutes of their time. An additional form requested that students consent to the use of their course results for research purposes. Students were assured that the data would be treated confidentially. The student assistant collected the completed questionnaires and consent forms. Questionnaires were only analyzed if the consent form was signed. Due to the procedure that was used, it was possible for students to complete more than one questionnaire, albeit on different themes.

Course exam results (i.e., achievement scores) were obtained at the end of courses in which students had filled out seminar questionnaires. Summative course exams contain multiple-choice questions that corresponded with the content of the course. In the Netherlands, grades range from 1 (lowest) to 10 (highest). To be able to control for prior achievement, students' prior achievement scores were calculated by averaging individual scores on all obtained previous course exams weighted for the amount of credits per course. To pass an exam, a score of at least 5.5 is required for each course; therefore, the prior achievement scores of students can range between 5.5 and 10.

Research Question 1: What are the Relations between Factors That Seem to be Important for the Seminar Learning Process?

We performed principal factor analysis (PFA) with promax (oblique) rotation to check whether the items grouped together in the questionnaire were congruent with factors that were found to affect seminar learning in earlier qualitative studies11,21,22 and to take into account correlations between factors.32

The number of extracted factors was based on Kaiser's criterion (retaining factors with associated eigenvalues of 1 or larger) and inspection of the scree plot.33 Although the factor analysis was based on pair-wise deletion of missing values, list-wise deletion and mean imputation of missing values resulted in highly similar factor loadings and correlations between factors. Because PFA cannot handle repeated-measurement structures, only the first questionnaire that a student handed in was used for PFA. Including multiple responses from one student would violate the assumption of independent observations and could easily lead to artificial factors.34

Items were retained when factor loadings were at least 0.4 and no larger cross loadings were found. While higher factor loadings are generally better, the chosen strategy is defendable from a statistical point of view,33 especially given the relatively large sample size. Analysis with remaining items was repeated. SPSS 21 software was used for the analyses. The factors were analyzed for their interpretability as a meaningful scale. Internal consistency was determined by calculating Cronbach's alpha for the matching cluster of items, and values of 0.70 and larger were considered acceptable. Average scores were computed for the items grouped together as factors. Correlations between these derived factors were calculated.

Research Question 2: How Do These Seminar Factors Account for Differences in Students' Achievement Scores?

Because individual students are part of the seminar groups, multilevel regression was used for the analysis of the questionnaire responses. Multilevel analysis takes interdependence of ratings into account. In the present study, for example, students are nested within seminar groups. Multilevel modeling disentangles these dependencies by quantifying the degree to which variance in student ratings is due to differences between individual students or to the higher level construct that is being evaluated (seminars) and within which ratings are nested. The multilevel method also allows the inclusion of explanatory variables (e.g., seminar group characteristics like group size) that may explain differences between students or between seminar groups. Another attractive feature of multilevel analysis is that the effects of explanatory variables can be estimated not only as a fixed effect (as in multiple regression analysis) but also as a random effect. This means that it is possible to estimate not only the average effect of, for example, student's prior achievement scores on students' academic achievement scores but also the degree to which this effect differs across seminars (random slopes). By choosing multilevel regression instead of multiple regression, we are able to deal with the violation of the assumption of independence of observations (students nested in groups, multiple responses of a part of the students) and with deviations from the classical assumption of homogeneity of variances (by specifying random effects). We used students' academic achievement scores as the dependent variable and the factors revealed by factor analysis together with prior achievement scores and group size as explanatory (i.e., predictive) variables. The predictive variables will form the fixed effects of the model. To account for intra-student and intra-seminar correlation in the data (i.e., multiple responses from the same students who participated in different seminars), we performed multilevel regression analysis using seminar-level and student-level intercepts and seminar-level random slope as random effects of our model. P values of <.05 were considered significant. SPSS 21 software was used for the analyses.

Written informed consent was obtained from all participants and the study was approved by the Ethical Review Board of the Dutch Society for Medical Education (NVMO-ERB; dossier number 46).

Data Collection

Of the possible 1,582 questionnaires that could have been handed in after the seminars, 988 (62.5%) completed questionnaires with signed consent forms were returned. It turned out that the 988 questionnaires were completed by 410 different students (median number of questionnaires per student=2, range 1–6). Of the students who completed questionnaires, 79.4% were female, which adequately reflects the gender distribution in the student population. Academic achievement of the final sample was largely comparable to the full groups of students taking each of the exams. Prior achievement scores ranged between 6.17 and 8.96, M=6.97, SD=0.53.

Research Question 1: What Are the Relations between Factors That Seem to Be Important for the Seminar Learning Process?

PFA was conducted on the data from 410 questionnaires. The Kaiser–Meyer–Olkin measure verified the sampling adequacy for the analysis (KMO 0.890 is “meritorious” according to Hutcheson and Sofroniou35). Bartlett's test of sphericity, χ2 (171)=3351.159, p<.001, indicated that the inter-item correlations were sufficiently large for PFA. Four factors had eigenvalues above Kaiser's criterion of 1. The scree plot indicated an inflection point (elbow) at the fifth factor, indicating that extracting four factors was acceptable. Table 1 shows factors, items, and factor loadings after rotation. PFA revealed four factors: teacher performance (factor 1), seminar content (factor 2), extent of a student's preparation for the seminar (factor 3), and opportunities for interaction in the seminar (factor 4). Cronbach's alpha was larger than .70 for all four factors, which is considered satisfactory. Because of unacceptable loadings or cross loadings, 33 items were deleted. Item 47 (“The teacher stimulated group interaction”) had a cross loading of 0.47 on the factor “teacher performance” and 0.48 on the factor “group interaction.” We included this item in the factor “group interaction” based on item content and because Cronbach's alpha of the scales supported this decision.

Table

Table 1: Principal factor analysis structure

Table 1: Principal factor analysis structure

Rotated factor loadings
Eigen
values
% of
variance
Cronbach's
α
Factors Items F1 F2 F3 F4
F1) Teacher The teacher explained the topics clearly .81 .13 .01 −.10 6.76 35.6 .90
The teacher had sufficient expertise on the seminar topic to facilitate the seminar group .69 .06 −.06 −.11
The teacher gave useful feedback on my activities .66 −.06 .05 .14
The teacher guided us through the questions in an adequate manner .82 .06 .00 −.10
The teacher adapted the seminar to students' knowledge levels .69 −.02 .00 .11
The teacher stimulated us to use our prior knowledge to answer the questions .68 .04 −.02 .05
The teacher created a safe learning environment during the seminar .66 −.11 .01 .02

My overall rating of teacher performance is
.80
.00
−.00
−.04



F2) Seminar
content
The relevance of the questions for the field of veterinary medicine is sufficiently clear to me −.16 .73 −.08 .06 2.42 12.8 .80
This seminar was well structured .17 .44 .02 .00
This seminar stimulated my interest in the subject matter .05 .66 .01 .05
This seminar has increased my understanding of the subject .14 .64 .03 −.05

In this seminar it has become clear to me that the content of this seminar is important within the field of veterinary medicine
−.01
.74
.04
.04



F3) Student
preparation
To what extent did you prepare for this seminar −.01 −.04 .75 −.04 1.68 8.9 .84
Before the seminar, I thought I was prepared sufficiently to be able to participate actively .01 −.01 .95 .02

After the seminar, I thought I had prepared sufficiently to be able to participate actively
−.02
.04
.72
.02



F4) Group
interaction
The questions stimulated group discussion sufficiently −.07 .20 −.00 .69 1.28 6.7 .79
The structure of the seminar encouraged interaction between students −.01 −.01 .01 .82
The teacher stimulated group discussion .47 −.14 −.04 .48

Table 2 presents the correlations between factors.

Table

Table 2: Correlations between factors

Table 2: Correlations between factors

Factor (p value) F1 F2 F3
F1: Teacher performance
F2: Seminar content 0.61
F3: Extent of preparation −0.02 0.15*
F4: Group interaction 0.57 0.58 0.07

*p<.005

p<.001

The results demonstrate strong and significant correlations between teacher performance, seminar content, and group interaction. No substantial correlations were found between extent of preparation and the other factors.

A zero-order correlations table with all variables and an explanation of how to interpret these correlations can be found in Appendix 2.

Research Question 2: How Do These Seminar Factors Account for Differences in Students' Achievement Scores?

Table 3 presents the full main effects model with the factors that relate to the seminar learning process, group size, and prior achievement scores as predictor variables.

Table

Table 3: Full main effects model

Table 3: Full main effects model

Parameter B (SE) df t p
Fixed effects
 Intercept 0.003 (0.058) 87.848 −0.0482 .962
 Teacher performance 0.000 (0.036) 399.063 0.000 1.000
 Group interaction −0.045 (0.032) 385.929 −1.387 .166
 Seminar content −0.012 (0.031) 408.174 −0.384 .701
 Student preparation 0.038 (0.023) 369.617 1.626 .105
 Group size −0.002 (0.056) 49.921 −0.044 .965
 Prior achievement scores 0.639 (0.035) 364.142 18.236 <.001
VAR (SE) df Wald Z p
Random effects
 Seminar group intercept 0.158 (0.039) 1 4.048 <.001
 Student intercept 0.394 (0.036) 1 11.042 <.001
 Residual 0.084 (0.009) 1 9.264 <.001

The table shows regression coefficients (B) for fixed effects and variance estimates (VAR) with their associated standard errors (SE) along with t-tests (N=620)

The contribution of significant predictors can only be properly estimated if we delete the non-significant predicators step by step. Multilevel regression analysis with a backward procedure that deleted the least significant variables one by one resulted in the end model presented in Table 4. For interpretation purposes, all predictors were standardized (i.e., yielding variables with an average score of 0 and standard deviation of 1).

Table

Table 4: Multilevel linear model for the prediction of academic achievement

Table 4: Multilevel linear model for the prediction of academic achievement

Parameter B (SE) df t p Lower Upper
Fixed effects
 Intercept 0.041 (0.056) 107.799 0.732 .466 −0.070 0.151
 Group interaction* −0.041 (0.020) 610.879 −2.077 .038 −0.081 −0.002
 Prior achievement scores* 0.587 (0.034) 217.130 17.339 <.001 0.520 0.653
VAR (SE) df Wald Z p Lower Upper
Random effects
 Seminar group intercept 0.166 (0.033) 1 5.012 <.001 0.113 0.246
 Slope seminar group: PAS 0.012 (0.006) 1 2.012 .044 0.004 0.030
 Student intercept 0.378 (0.030) 1 12.493 <.001 0.323 0.442
 Residual 0.099 (0.008) 1 12.546 <.001 0.085 0.116

PAS=prior achievement scores

*Academic achievement, group interaction, and prior achievement scores were standardized (i.e., mean 0 and standard deviation 1) for interpretation purposes

The table shows regression coefficients (B) for fixed effects and variance estimates (VAR) with their associated standard errors (SE) along with t-tests and 95% CIs (lower bound and upper bound) (N=925)

Following Lipsey and Wilson's36 definition of values of 0.10, 0.25, and 0.40 as indicating small, medium, and large effects, respectively, Table 4 indicates that prior achievement scores appeared to have a large effect on students' achievement on course tests. Group interaction appeared to have a very small negative effect.

We conducted a questionnaire study to investigate relationships between factors that seem to be important for the seminar learning process and we wanted to determine how these seminar factors account for the differences in students' academic achievement scores.

The analysis revealed four factors that are important for the seminar learning process according to students: teacher performance, seminar content, extent of student preparation, and opportunities for group interaction. The strong correlation found between teacher performance and seminar content replicated the relation that was found in an earlier study on seminar learning.11 The strong correlation between teacher performance and group interaction may suggest that the seminars were less teacher centered than the seminars in the study of Jaarsma et al.,11 where this relationship was non-significant and weak. This may be explained by the fact that most of the teachers in the present study had completed the faculty development program focused on interactive teaching, and the fact that students and teachers were aware of the educational philosophy of the curriculum, that is, to promote active participation and group interaction. The strong positive correlation between seminar content and group interaction suggests that group interaction might be enhanced by improving seminar content. However, a questionnaire study such as the present one does not allow for inferences of causal relationships between factors, and it is equally possible that improving group interaction might raise students' perceptions of seminar content.

In comparison with Jaarsma's study on seminars, the factor analysis of this study revealed a new factor (extent of preparation) that was hypothesized based on earlier qualitative studies on seminar learning.21,22 Surprisingly, however, the new factor showed no statistically significant important relationship with the other factors, which implies that the results of the present study do not support the importance of good preparation for group interaction, as was assumed by teachers in an earlier study.22 This study suggest that while some teachers may think that preparation is a prerequisite for group interaction, students do not consider or experience that doing the preparatory assignments has a positive effect on group interaction. In this study, we have relied on students' general report of their own preparation. In a follow-up study, we suggest adding different measurements of “the extent of student preparation,” for example, by asking students to keep a log of time spent on preparing, or by measuring the extent of preparation with a pre-seminar test like those used in team-based learning curricula (readiness assurance process test37). In addition, an observational study could provide us more insight in the way students use the knowledge that they have acquired through preparing the materials in their group discussions. This may offer useful insights to help improve the seminar learning process.

A look at the results of the second research question shows that prior achievement scores appear to account most for the difference between students' achievement scores. We were quite surprised by the small effect sizes of the association between teacher performance, group interaction, and seminar content, on the one hand, and achievement scores, on the other hand. However, these findings are supported by Hattie's statement38 that in explaining academic achievement, variables of the learning environment are less important than variables at the student level. Also, compared with other quantitative studies on academic achievement in other contexts, the relatively strong influence of prior achievement scores should not come as a surprise.39,40 However, given the findings of the qualitative studies on seminar learning,21,22 a greater impact of the factor “extent of preparation” on academic achievement was expected. In the general educational literature, the impact of the extent of preparation on academic achievement is a topic of debate. In a review of homework in secondary education,41 homework was assumed to be associated with improved academic achievement while research in higher education reports equivocal results for this relationship.42 In line with the results of the present study, research in medical education in PBL curricula on “time spent on self-study” showed no effect on unit test scores,31 while “time spent on individual study” also correlated poorly with scores on tests measuring short- and long-term knowledge retention.24 In contrast to the study by Webb et al.43 where group interaction was identified as a strong predictor of student achievement, group interaction in our study seem to be a very small negative predictor for achievement scores. A possible reason for not finding a strong relationship between these two variables might be a mismatch between the objective of seminar learning (active learning and promoting a deep learning approach through group interaction) and the type of assessment or exam questions. To be able to answer the course exam questions correctly, group interaction and a deep learning approach during seminars may not be necessary.44 Besides this argument, the items we used in the factor group interaction give no indication of the breadth and depth of group interaction. So it might be that group interactions encouraged during a seminar are not the kinds of interactions needed to get high scores on end-of-course exams.

A strength of this study is that the questionnaires were collected in the authentic context of many different seminars. Even though we knew that the trade-off was a loss of richness in specificity, we felt that measuring what happens in many actual real-life seminars would provide us rich insights into this educational method in general. We must be aware that it is possible that another effect could have been lost due to the variability among the seminar approaches. The insights we gained can be used for future, more systematic studies on seminar learning, for example, by conceptualizing interventions that can be implemented iteratively in seminars. Even though this study has provided rich information about the relations between factors influencing seminar learning and their association with student's academic achievement, there are some limitations. The study was restricted to the perceptions of students at one university. However, since our main focus was on seminar groups, we sampled a broad variety of seminars relating to different disciplines and themes, and facilitated by different teachers. Consequently, we believe that the results may be of value to other institutions that use seminars in their curriculum. Another limitation is that the number of times students responded to questionnaires varied. A stronger sampling design could result in a larger proportion of students responding to the questionnaire in multiple seminars. This could enable researchers to further study factors within and between students that contribute to seminar perception and academic achievement. After factor analysis, we lost 33 questionnaire items, which could not be included in the results. This does not mean, however, that the answers to those items are not useful. They can provide rich information to course coordinators and teachers, for example, for use in quality assurance.

One practical implication of this study is that attention should be paid to the connection between seminars and the type of assessment. Another practical implication of this study is that the USEME questionnaire can be used for identifying the significance of teacher performance, seminar content, preparation by students, and interaction within seminars.

The suggestion for future research is that the four factors resulting from the present study be examined further, for example, in an experimental setup, using different instructional manipulations in different seminars dealing with the same content in the same curricular year. We also recommend further studies on the connection between the aim of the seminar and the method of assessment of seminar learning.

ACKNOWLEDGMENTS

The authors would like to thank the students and teachers who participated in this study and Mereke Gorsira for editing the manuscript.

CONFLICT of interest

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

1. Elsheikha HM, Kendall NR. Linking theory to practice in an undergraduate veterinary curriculum: students' perspectives. J Vet Med Educ. 2009;36(3):2916. http://dx.doi.org/10.3138/jvme.36.3.291. Medline:19861717 LinkGoogle Scholar
2. Bok HG, Jaarsma DA, Teunissen PW, et al. Development and validation of a competency framework for veterinarians. J Vet Med Educ. 2011;38(3):2629. http://dx.doi.org/10.3138/jvme.38.3.262. Medline:22023978 LinkGoogle Scholar
3. Pritchard WR, editor. Future directions for veterinary medicine: report of the Pew National Veterinary Education Program. Durham, NC: Duke University; 1988 Google Scholar
4. Welsh PJ, Jones LM, May SA, et al. Approaches to defining day-one competency: a framework for learning veterinary skills. Rev Sci Tech. 2009;28(2):7717. Medline:20128489 MedlineGoogle Scholar
5. Larkin M. Starting from day one. OIE seeking to define minimum veterinary competencies. J Am Vet Med Assoc. 2011;239(3):2725. Medline:21916047 MedlineGoogle Scholar
6. Thornburg LP. Four essential components of veterinary education for the 21st century. J Am Vet Med Assoc. 1992;201(8):11803. Medline:1429154 MedlineGoogle Scholar
7. Turnwald GH, Sponenberg DP, Meldrum JB. Part I: twenty-year literature overview of veterinary and allopathic medicine. J Vet Med Educ. 2008;35(1):6673. http://dx.doi.org/10.3138/jvme.35.1.066. Medline:18339961 LinkGoogle Scholar
8. Kelly PA, Haidet P, Schneider V, et al. A comparison of in-class learner engagement across lecture, problem-based learning, and team learning using the STROBE classroom observation tool. Teach Learn Med. 2005;17(2):1128. http://dx.doi.org/10.1207/s15328015tlm1702_4. Medline:15833720 MedlineGoogle Scholar
9. Thurman J, Volet SE, Bolton JR. Collaborative, case-based learning: how do students actually learn from each other? J Vet Med Educ. 2009;36(3):297304. http://dx.doi.org/10.3138/jvme.36.3.297. Medline:19861718 LinkGoogle Scholar
10. Shatzer JH. Instructional methods. Acad Med. 1998;73(9 Suppl):S3845. http://dx.doi.org/10.1097/00001888-199809000-00034. Medline:9759117 MedlineGoogle Scholar
11. Jaarsma ADC, de Grave WS, Muijtjens AMM, et al. Perceptions of learning as a function of seminar group factors. Med Educ. 2008;42(12):117884. http://dx.doi.org/10.1111/j.1365-2923.2008.03170.x. Medline:19120948 MedlineGoogle Scholar
12. Dennick RG, Spencer J. Teaching and learning in small groups. In: Dornan T, Mann K, Scherpbier A, et al, editors. Medical education: theory and practice. Edinburgh: Elsevier; 2011. p. 13156 Google Scholar
13. Marton F, Säljö R. Approaches to learning. In: Marton F, Hounsell D, Entwistle NJ, editors. The experience of learning: implications for teaching and studying in higher education. 3rd (Internet) ed. Edinburgh: University of Edinburgh, Centre for Teaching, Learning and Assessment; 2005. p. 3958 Google Scholar
14. Entwistle NJ. Teaching for understanding at university: deep approaches and distinctive ways of thinking. Basingstoke, UK: Palgrave Macmillan; 2009 Google Scholar
15. Novak JD. Meaningful learning: the essential factor for conceptual change in limited or inappropriate propositional hierarchies leading to empowerment of learners. Sci Educ. 2002;86(4):54871. http://dx.doi.org/10.1002/sce.10032 Google Scholar
16. Kalyuga S. Knowledge elaboration: a cognitive load perspective. Learn Instr. 2009;19(5):40210. http://dx.doi.org/10.1016/j.learninstruc.2009.02.003 Google Scholar
17. Crosby JR, Hesketh EA. Developing the teaching instinct. 11: small group learning. Med Teach. 2004;26(1):169. http://dx.doi.org/10.1080/01421590310001648122. Medline:14744688 MedlineGoogle Scholar
18. Khosa DK, Volet SE, Bolton JR. An instructional intervention to encourage effective deep collaborative learning in undergraduate veterinary students. J Vet Med Educ. 2010;37(4):36976. http://dx.doi.org/10.3138/jvme.37.4.369. Medline:21135404 LinkGoogle Scholar
19. Kurtz SM, Silverman DJ, Draper J, et al. Teaching and learning communication skills in medicine. 2nd ed. Abingdon, UK: Radcliff Publishing; 2005 Google Scholar
20. Canfield PJ. An interactive, student-centered approach to teaching large-group sessions in veterinary clinical pathology. J Vet Med Educ. 2002;29(2):10510. http://dx.doi.org/10.3138/jvme.29.2.105. Medline:12143029 LinkGoogle Scholar
21. Spruijt A, Jaarsma ADC, Wolfhagen HA, et al. Students' perceptions of aspects affecting seminar learning. Med Teach. 2012;34(2):e12935. http://dx.doi.org/10.3109/0142159X.2012.644829. Medline:22289011 MedlineGoogle Scholar
22. Spruijt A, Wolfhagen I, Bok H, et al. Teachers' perceptions of aspects affecting seminar learning: a qualitative study. BMC Med Educ. 2013;13(1):22. http://dx.doi.org/10.1186/1472-6920-13-22. Medline:23399475 MedlineGoogle Scholar
23. Gijselaers WH, Schmidt HG. Development and evaluation of a causal model of problem-based learning. In: Nooman ZM, Schmidt HG, Ezzat ES, editors. Innovation in medical education: an evaluation of its present status. New York: Springer; 1990. p. 95113 Google Scholar
24. van den Hurk MM, Wolfhagen HA, Dolmans DHJM, et al. The relation between time spent on individual study and academic achievement in a problem-based curriculum. Adv Health Sci Educ Theory Pract. 1998;3(1):439. http://dx.doi.org/10.1023/A:1009732511707. Medline:12386394 MedlineGoogle Scholar
25. Van Berkel HJM, Schmidt HG. Motivation to commit oneself as a determinant of achievement in problem-based learning. High Educ. 2000;40(2):23142. http://dx.doi.org/10.1023/A:1004022116365 Google Scholar
26. van den Hurk MM, Dolmans DHJM, Wolfhagen IH, et al. Testing a causal model for learning in a problem-based curriculum. Adv Health Sci Educ Theory Pract. 2001;6(2):1419. http://dx.doi.org/10.1023/A:1011402507476. Medline:11435764 MedlineGoogle Scholar
27. Nieminen J, Sauri P, Lonka K. On the relationship between group functioning and study success in problem-based learning. Med Educ. 2006;40(1):6471. http://dx.doi.org/10.1111/j.1365-2929.2005.02344.x. Medline:16441325 MedlineGoogle Scholar
28. te Winkel W, Rikers R, Loyens S, et al. Influence of learning resources on study time and achievement scores in a problem-based curriculum. Adv Health Sci Educ Theory Pract. 2006;11(4):3819. http://dx.doi.org/10.1007/s10459-006-9016-4. Medline:16897559 MedlineGoogle Scholar
29. Van Berkel HJ, Dolmans DHJM. The influence of tutoring competencies on problems, group functioning and student achievement in problem-based learning. Med Educ. 2006;40(8):7306. http://dx.doi.org/10.1111/j.1365-2929.2006.02530.x. Medline:16869917 MedlineGoogle Scholar
30. Sockalingam N, Rotgans JI, Schmidt HG. The relationships between problem characteristics, achievement-related behaviors, and academic achievement in problem-based learning. Adv Health Sci Educ Theory Pract. 2011;16(4):48190. http://dx.doi.org/10.1007/s10459-010-9270-3. Medline:21293923 MedlineGoogle Scholar
31. Kamp RJ, Dolmans DD, van Berkel HJ, et al. The relationship between students' small group activities, time spent on self-study, and achievement. High Educ. 2012;64(3):38597. http://dx.doi.org/10.1007/s10734-011-9500-5 Google Scholar
32. Wetzel AP. Factor analysis methods and validity evidence: a review of instrument development across the medical education continuum. Acad Med. 2012;87(8):10609. http://dx.doi.org/10.1097/ACM.0b013e31825d305d. Medline:22722361 MedlineGoogle Scholar
33. Field A. Discovering statistics using IBM SPSS Statistics. London: Sage; 2013 Google Scholar
34. Tacq J. Multivariate analysis techniques in social science research: from problem to analysis. London: Sage; 1997 Google Scholar
35. Hutcheson G, Sonfroniou N. The multivariate social scientist. London: Sage; 1999 Google Scholar
36. Lipsey MW, Wilson DB. Practical meta-analysis. London: Sage; 2001 Google Scholar
37. Michaelsen LK, Parmelee DX, McMahon KK. Team-based learning for health professions education: a guide to using small groups for improving learning. Sterling, VA: Sterling Stylus Publishing, LLC; 2008 Google Scholar
38. Hattie J. Visible learning: a synthesis of over 800 meta-analyses relating to achievement. London: Routledge; 2009 Google Scholar
39. Jansen EP, Bruinsma M. Explaining achievement in higher education. Educ Res Eval. 2005;11(3):23552. http://dx.doi.org/10.1080/13803610500101173 Google Scholar
40. de Koning BB, Loyens SM, Rikers RM, et al. Generation psy: student characteristics and academic achievement in a three-year problem-based learning bachelor program. Learn Individ Differ. 2012;22(3):31323. http://dx.doi.org/10.1016/j.lindif.2012.01.003 Google Scholar
41. Cooper H, Robinson JC, Patall EA. Does homework improve academic achievement? A synthesis of research, 1987–2003. Rev Educ Res. 2006;76(1):162. http://dx.doi.org/10.3102/00346543076001001 Google Scholar
42. Masui C, Broeckmans J, Doumen S, Groenen A, Molenberghs G. Do diligent students perform better? Complex relations between student and course characteristics, study time, and academic performance in higher education. High Educ. 2014;39(4):62143. http://dx.doi.org/10.1080/03075079.2012.721350 Google Scholar
43. Webb NM, Troper JD, Fall R. Constructive activity and learning in collaborative small groups. J Educ Psychol. 1995;87(3):40623. http://dx.doi.org/10.1037/0022-0663.87.3.406 Google Scholar
44. Scouller K. The influence of assessment method on students' learning approaches: multiple choice question examination versus assignment essay. High Educ. 1998;35(4):45372. http://dx.doi.org/10.1023/A:1003196224280 Google Scholar
45. Snijders TAB, Bosker RJ. Multilevel analysis: an introduction to basic and advanced multilevel modeling. London: Sage; 2012 Google Scholar
Teacher
performance
Group
interaction
Seminar
content
Student
preparation
Group
size
Prior
achievement
scores
Academic
achievement
scores
Teacher performance
 Group interaction 0.66
 Seminar content 0.55 0.55
 Student preparation 0.02 0.12 0.15
 Group size −0.34 −0.25 −0.25 −0.09*
 Prior achievement scores −0.03 −0.01 0.01 0.18 0.01
 Academic achievement scores −0.09* −0.06 −0.08* 0.20 0.02 0.64
 Gender 0.02 0.02 0.06 0.03 −0.04 0.08* 0.06

*p<.05

p<.01

Zero-order correlations are bivariate correlations. That is, the correlation between, say, academic achievement scores and group interaction is not corrected for overlap that these two variables have with other variables in the zero-order correlations table. When running a regression with academic achievement scores as a dependent variable and group interaction as an independent variable, the standardized regression coefficient (beta) equals that zero-order correlation coefficient and the latter equals the semi-partial and partial correlation coefficient. However, once we include other variables from the zero-order table in the aforementioned regression model—variables that have non-zero correlations with academic achievement scores and with group interaction—semi-partial and partial correlation coefficients will be different. Many researchers believe that semi-partial and partial correlation coefficients always get smaller once you include other independent variables, and hence they reason that if a zero-order correlation coefficient is close to zero and not statistically significant, the semi-partial or partial coefficient of that variable in a multiple regression model will be even closer to zero and even less significant. However, Tacq34 provides some very good examples of how semi-partial and partial correlation coefficients may get smaller as well as larger when including other independent variables. Further, zero-order correlations do not correct for hierarchical levels in the data; like classical simple and multiple regression analysis, they are built on the assumption that all observations in a study are statistically independent. In our study, we have multiple responses from the same students—which are by definition correlated and thus not statistically independent—who are nested within seminar groups, tending to create an intra-seminar-group correlation. Snijders and Bosker45 among others provide examples of how in some cases not accounting for such multilevel structures zero-order correlations can go negative while coefficients in appropriate multilevel models are positive and vice versa. We have therefore chosen to present and interpret the multilevel end model presented in Table 4.