Should TBL Readiness Assurance Tests Be Graded?

by Kaylee Hall, PGY1 Pharmacy Practice Resident, University of Mississippi Medical Center

Team-based learning (TBL) is a learning strategy that requires students to prepare for topics prior to class and be ready to apply that knowledge to solve problems in a group setting. In professional schools, this is typically done by working through patient cases and clinical questions as a team. TBL is structured so the class starts with an Individual Readiness Assurance Test (iRAT), typically consisting of multiple-choice questions designed to test if the student is prepared to work through the cases/questions during that class. Following the iRAT, students then complete the group Readiness Assurance Test (gRAT), which consists of the same questions from the iRAT; however students work together in groups to answer the questions, facilitating group discussion and problem-solving. Students typically receive immediate feedback on the gRATs to promote conversations within the group and a deeper understanding of the subject matter. While the RATs are an important part of the learning process, controversy exists on whether the RATs scores should be counted towards students’ grades or counted as participation (i.e., the student is awarded full credit for being present and completing both the iRAT and gRAT).¹

A recent study conducted at the University of Tennessee College of Pharmacy assessed student performance when RATs were graded versus ungraded in a TBL course. They compared results from an elective course offered in the spring of 2020 and 2021. RAT scores were graded in 2020 and not graded in 2021. After the 2021 course, students were asked to take a survey to assess class preparation and perceived team accountability. The investigators found no significant difference in student exam performance when comparing the ungraded versus graded cohorts and concluded that ungraded RATs did not adversely impact students' examination performance.²

Pros to counting RAT scores

Assessment drives learning. Having RAT scores count towards the students' grades may provide them with extrinsic motivation to complete the pre-class materials which are essential for TBL. Previous investigations have looked at how grading iRATs affected class preparation and performance. They found that when iRATs were graded, students were more likely to download pre-class materials and performed significantly better on iRATs. When they compared the download frequency of preclass materials, it dropped by about 30% for Year 1 and by nearly 50% in Year 2 courses when iRATs were ungraded compared to graded.³

Cons to counting RAT scores

An assessment focus rather than a mastery focus. Assessments may promote cramming of material and superficial learning of the subject in order to pass the quiz or examination without truly understanding the concepts.⁴ Grading RATs may also encourage students to use a performance or performance-avoidance approach to learning, where students focus on the assessment with the goal to perform well or outperform their peers. This orientation toward learning promotes superficial learning of the material without encouraging students to master the subject.⁵

Negative effects on group cohesion. Grading RATs may push students to focus on individual efforts instead of working as a team. Poor group cohesion may promote social loafing where students give less effort because they can rely on other members of the team to do the work.⁶ In contrast, not grading RATs may encourage students to work together to achieve mastery of the subject. Without the need to achieve a grade, group assessments encourage students to prepare for the materials based on their intrinsic motivation and the desire to contribute their ideas. When students are intrinsically motivated, they tend to be more engaged and stay engaged longer than students who are extrinsically motivated.

Indeed, student surveys have found that ungraded cohorts feel more responsible for the team and have a greater desire to contribute to the group’s work. Students also report that they felt their contributions were important, indicating good group involvement and cohesion. Additionally, the majority of students reported that class preparation is necessary to perform well in the course.²

Increased pressure to perform. The pressure to perform well on assessments may encourage academic dishonesty and promote unneeded stress. Without the pressure to perform, students are able to focus on a deeper understanding of the material and are less likely to engage in superficial learning simply to perform well on examinations.

Conclusion

Few studies have looked at student performance when RATs are graded versus ungraded in the Team-Based Learning model.  Professional opinions differ on which approach is best. Historically, we know that assessment drives learning, but graded assessments may not be the most appropriate approach to the team-based learning strategy and may have unintended consequences. Grading readiness assurance tests may promote superficial learning of the material, lead to poor group cohesion and inflict unnecessary stress on students.  At least one study suggests that ungraded RATS does not harm student grades. Removing grades may diminish extrinsic motivation for students to prepare for readiness assessments but allows students to foster their intrinsic motivation — to be motivated more by the desire to contribute to the group and to master the material. More evidence is needed to truly assess the pros and cons of graded versus ungraded readiness assurance tests in professional schools.

Resources:

1. Hrynchak P, Batty H. The educational theory basis of team-based learning. Med Teach. 2012;34(10):796-801.
2. Eudaley ST, Farland MZ, Melton T, et al. Student Performance on Graded Versus Ungraded Readiness Assurance Tests in a Team-Based Learning Elective. Am J Pharm Educ. 2022;86(9): 8851.
3. Koh YYJ, Rotgans JI, Rajalingam P, et al. Effects of graded versus ungraded individual readiness assurance scores in team-based learning: a quasi-experimental study. Adv Health Sci Educ Theory Pract. 2019;24(3):477-488.
4. Epstein RM. Assessment in medical education. N Engl J Med. 2007;356(4):387-96.
5. Meece JL, Anderman EM, Anderman LH. Classroom goal structure, student motivation, and academic achievement. Annu Rev Psychol. 2006;57:487-503.
6. Karau SJ, Williams KD. Social loafing: A meta-analytic review and theoretical integration. Journal of Personality and Social Psychology. 1993;65(4):681–706.

Leading Future Leaders? Developing a Curriculum to Enhance Leadership Skills

by Blake Mangum, PharmD, PGY1 Pharmacy Practice Resident, University of Mississippi Medical Center

When enrolling in a professional health program such as medicine, pharmacy, or nursing, many students must adjust to the demands of their program. The material is complex, the workload is demanding, and there are often co-curricular organizations and responsibilities to make time management even more complicated. Didactic instruction is critical in career development to understand the core material needed, but it's equally important to develop the professional and leadership skills required to excel. We often focus on helping students develop as future clinicians and instill the requisite foundational knowledge for their chosen professional field. But it is easy to overlook the importance of helping students develop as future leaders. Organizations that accredit professional programs, such as the Accreditation Council for Pharmacy Education (ACPE), require the curricula to address professional development and leadership.  However, the method and material to be covered are left to the individual program. This introduces some variability between programs as institutions have various philosophies on the optimal method to deliver this material and may even define leadership differently. Do you structure the leadership curriculum by delivering a series of lectures, a couple of exams, and a final project?  Or do you divide students into teams and have small-group discussions?  Or do you send students out on their clinical rotations to develop their leadership skills in the "real world"? This essay will reflect on methods currently used in leadership development education and the outcomes reported by those using these methods.

Optimal Structure of a Leadership Curriculum

During my time as a pharmacy student, I took several professional and leadership development courses that were required in my program. I had assumed the structure and methods used would be similar at every health professions program. My classes started by taking a strengths finder analysis and discussing the impacts that various skills could have on team dynamics and performance. While researching what goes into making a leadership development course, I found that some programs take a similar approach, but others are very different. A recent scoping review examined fourteen published reports on leadership education in PharmD programs.¹ Of those, only six employed traditional didactic lectures in the required curriculum. Others relied on co-curricular activities, elective courses, and leadership retreats. Even didactic lectures had variable content and schedules. Another systematic review provides insight into what components give us the most return on investment.² The review used Kirkpatrick levels of evaluation to determine the effectiveness of each intervention or approach. They found that leadership interventions that combined didactic lectures with projects, lab activities, and small groups were more effective than didactic lectures alone.² Both papers report that students prefer a multimodal approach. The course structure is not the only variable to consider.

Optimal Duration of a Leadership Curriculum

We have all heard that practice makes perfect. It is important to consider how long it takes to conduct a program aimed at leadership development and how often the participants should meet. The literature is quite variable in this regard.  Some programs met once monthly for a semester or entire academic year, while others consisted of a 5-module series over multiple years, and others were a single-day leadership retreat.^1,2 The exact length of a program and how often to meet will depend on several institution-specific factors, such as the availability of professors to deliver lectures, the schedules of students' other classes, physical facilities, and more. Based on a review of the literature, longitudinal courses have greater efficacy than shorter program durations; however, programs that lasted half a day were compared to programs that lasted the entire four-year curriculum.² The answer to how long to teach and how often to meet is debatable, but a good starting point would be a standard semester-long course.

Optimal Class Size

The literature reviews report cohorts ranging from 6 to over 200 students.^1,2 Smaller cohorts scored higher on efficacy rankings than large cohorts. This area is a little harder to navigate as not much can be done about a class size if you require the entire student body to complete the program. One possible method of ensuring smaller cohorts would be to divide classes into sections, with some sitting in a lecture for one week while the other section is involved in a skills lab, and perhaps dividing up the cohort into even smaller groups for discussion sessions. This method would provide students with a multimodal learning experience and more direct learning opportunities.

Optimal Course Development

The University of Massachusetts Medical School recently created a course to enhance medical students' leadership and teamwork skills.³ The course objectives included leadership style, communication within the health care team, giving and receiving feedback, delegating responsibilities, and setting direction for a team. These leadership skills were identified as core competencies in a previous needs assessment survey. The course involved six 90-minute sessions.  A portion of each session was a didactic lecture, plus there were interactive activities to reinforce the material, post-session questionnaires, and individual reflections. Similar to previous studies, the authors used pre- and post-course surveys to gauge course impact. Students completed a survey eight months after the course conclusion to assess the long-term skill use and effects of the course. The authors documented increased student confidence related to each course objective and the key skills learned during the course were still being used in everyday life eight months later. This model indicates that surveying students about impact is a best practice to assess the effectiveness of a course. Of note, the creators of this course were students themselves! This underscores the fact that students actively seek to strengthen their leadership skills and, in this case, help others develop them as well. Healthcare is a team sport, and having the ability to lead the team in your area of expertise is an essential skill.

Leadership skill development is not routinely considered when entering a clinical profession. However, these skills are critically important and need to be considered when developing a curriculum. The optimal methods will vary between institutions, but using a multimodal approach with lectures, longitudinal projects, and having smaller cohorts in classes are factors that can improve the effectiveness of a leadership course.

References

1. DeVolld T, DiPietro Mager N, et al. Management, entrepreneurship, continuing professional development, and leadership education in United States doctor of pharmacy curricula: A scoping review. Curr Pharm Teach Learn 2022;14(6):798–808
2. Evans MA, James EJ, Misa Mi. Leadership training in undergraduate medical education: A systematic review. Int J Med Students 2023;11(1):58–66.
3. Richard K, Noujaim M, Thorndyke L, Fischer MA. Preparing medical students to be physician leaders: A leadership training program for students designed and led by students.. Med Ed Portal. 2019; 15: 10863