Assessing Students on Advanced Pharmacy Practice Experiences

by Taylor Loper, Pharm.D., PGY1 Pharmacy Practice Resident, Baptist Memorial Hospital-North Mississippi

One thousand, four hundred and forty. That’s the minimum number of Advanced Pharmacy Practice Experience (APPE) hours the Accreditation Council for Pharmacy Education (ACPE) mandates a pharmacy student receive before they graduate.¹  Over the course of at least thirty-six weeks (and sometimes more), these students are exposed to a variety of practice settings, institutions, and situations designed to support the development of their knowledge, skills, and abilities.  Essentially, their competence to practice as a pharmacist following graduation. The ACPE Standards state that the APPEs should be designed to “hone practice skills, professional judgment, behaviors, attitudes and values, confidence, and sense of personal and professional responsibility.” Schools are also required to have a formal assessment of the achievement of APPE competencies using validated assessments. Student performance must be documented at key time points throughout these experiences.

https://pharmacy.uiowa.edu/students/academic-programs/doctor-pharmacy/pep/appe

The CAPE educational outcomes developed by the American Association of Colleges of Pharmacy (AACP) are a set of goals that all pharmacy curriculums should be focused on achieving. These educational outcomes are linked to Entrustable Professional Activities (EPAs) that all pharmacy graduates should be able to perform.² (For more on EntrustableProfessional Activities, see the post by Andrew Mays) Students should have ample opportunity to practice these activities become proficient, and demonstrate mastery before becoming licensed pharmacists. This can be problematic in that the heavily didactic nature of the first three years of the pharmacy curriculum results in few opportunities to practice these EPAs.  Too often students are often being assessed before they have an opportunity to master the EPAs during APPE rotations.

Reflecting back on my experiences as a pharmacy student, the assessment of learning on APPE rotations involved a series of assignments that had to be completed by the end of the rotation.  An example of this would be a set of questions asking you to reflect on interprofessional teams and the benefits of working with different professions at your current practice site. While most assignments were site-specific, several (like the one above) were repeated for multiple rotations. Additionally, specific objectives were set forth and students were asked to provide evidence of assignments or activities they completed that enabled them to meet those objectives. An example of this would be to “evaluate and interpret patient data.” A student could then provide details of working patients up, reviewing medical records, or conducting patient interviews. This gave students an opportunity for reflection while providing concrete examples of progress that the APPE preceptors could then base their end-of-rotation evaluations on. However, completing these assignments and documenting these examples was often time-consuming. By the final APPE, they felt cumbersome, especially the assignments that we had to repeatedly do on multiple rotations.

This process of assessing student performance raises several questions. First, how do we ensure each student meets the required competencies for each rotation? With practices settings and sites varying significantly, assessing each student on basic competencies can be difficult. Moreover, different preceptors have different expectations. All this variability makes it very difficult to create a consistent assessment process that is not dependent on the student’s learning experiences. Second, how do we assure an assessment tool can be applied in a variety of APPE rotations without omissions or redundancy? Requiring the same assignments and reassessing the same set of skills for a student who is taking two community rotations puts a strain on the student and preceptor. But we must find a way to ensure the student is developing on each rotation. Finally, how do we measure competency, such as the EPAs?  Should be rated “acceptable”?  Or “completed”? Should a student be required to “complete” them by the end of each APPE or by the end of all APPE experiences?

Several institutions have tried to address these questions. The System of Universal Clinical Competency Evaluation in the Sunshine State (SUCCESS), is an internet-based APPE assessment tool created by the colleges/schools of pharmacy in Florida.³ Under this system, preceptors rate students as “excellent”, “competent”, or “deficient” for each competency at the end of each APPE. They are also allowed to select “no opportunity” if not observed. These ratings were then converted by the school to determine the student’s grade.  There was a correction factor for students that were earlier along during their APPE schedule. It also allowed preceptors to weigh each competency based on importance and frequency in the practice setting / site. This weight provides preceptors the ability to focus on the learning goals that are most relevant. Another such tool was created by faculty at the University of Colorado Skaggs School of Pharmacy after the addition of 14 ability-based outcomes to their curriculum.⁴ By polling current preceptors, they were able to determine which competencies and outcomes were frequently observed and how important they are to the success of students on each APPE. These responses were used to create APPE-specific tools to ensure students met rotation goals that aligned with the ability-based outcomes of the curriculum.

It’s clear that assessing the performance of APPE students is a crucial, yet complex, task. Based on the two methods documented above, implementing an effective evaluation method requires the active participation of preceptors in developing a tool that is specific to each APPE experience. Preceptor evaluations of students need to be specific to the setting and site but must also relate to the overarching ACPE standards and ACCP outcomes.

I believe that monthly preceptor evaluations of students and their progress toward or achievement of learning objectives are necessary to ensure each APPE experience is helping to develop the student’s competence. However, rather than completing a series of monthly (and sometimes redundant) assignments, a series of unique assignments completed over the ENTIRE year coupled with specific ability-based assessments might be a better strategy.  This can reduce assignment fatigue and still provide appropriate documentation that each student can competently perform the EPAs and other educational outcomes before they graduate. It would great to see some research to determine the validity of this approach.

References

1. Accreditation Council for Pharmacy Education. Accreditation Standards and Key Elements for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree (Standards 2016). 2015.
2. Haines ST, Pittenger AL, Stolte SK, et al. Core Entrustable Professional Activities for New Pharmacy Graduates. Am J Pharm Educ. 2017; 81(1): Article S2.
3. Reid LD, Nemire R, Doty R, et al. An Automated Competency-Based Student Performance Assessment Program for Advanced Pharmacy Practice Experiential Programs. Am J Pharm Educ. 2007; 71(6): Article 128.
4. Gilliam EH, Brunner JM, Nuffer W, et al. Design and Content Validation of Setting-Specific Assessment Tools for Advanced Pharmacy Practice Experience Rotations. [published online ahead of print March 6, 2019] Am J Pharm Educ. Article 7067.

Social-Class Achievement Gaps in Higher Education: Can Values Affirmation Interventions Help?

by Lily Van Cheng, PharmD, PGY1 Community Pharmacy Practice Resident, University of Mississippi School of Pharmacy

As an underrepresented minority (URM) and first-generation (FG) college student, the psychosocial factors that influence one’s success at the collegiate level of education is both fascinating and frightening. FG college students comprise roughly 15-20% of students in American universities today.¹ FG students are more likely to come from working-class backgrounds and face significant economic and psychosocial barriers that create performance discrepancies called the “social-class achievement gap.”² The performance gap might be the result of poverty, the rigor of high school preparation, parenting practices, and/or cultural mismatches. None-the-less, it is arguable that the gap between FG students and continuing-generation (CG) students are merely the results of differences in baseline academic preparation or readiness.

Martin Leon Barreto for The Chronicle Review

A tool that some educators have used to address these challenges has been the values affirmation (VA) intervention.^3,4 VA interventions are designed to address the students’ perceived “stereotype threat.”  FG students are more likely to be confronted with stereotypes that threaten their identity and self-esteem which affect their academic performance. The VA intervention technique addresses stereotype threat by asking students to reflect and write about their most important values. It is hypothesized that this practice enhances the student’s ability to cope with internal identity threat and reaffirms their core values to reestablish their personal integrity and worth. In one study conducted with middle school students, a VA intervention significantly improved the grades of Latino students. The grades of white students were not impacted. The VA intervention thereby partially closed the achievement gap for URM students.⁵

In a more recent study conducted at the University of Wisconsin – Madison, researchers evaluated the role of a VA intervention comparing the performance of FG versus CG college students in a double-blinded randomized experiment in an introductory biology course.⁶ Outcome measures included confidence in their innate academic abilities and perceived concern about their generational background on academic success.  In addition, the researchers compared final course grades, overall GPA in other courses (excluding the biology course), and rate of continuation in the second-semester biology sequence. Students were randomized in blocks based on a variety of characteristics, including generational and URM status. In both the VA intervention and control groups, there were FG and CG students.  All students in the VA intervention were instructed to identify and write about values that were most important to them.  Students in the control group were instructed to identify values least important to them and write about why these values would be important to someone else.

The results?  The researchers found a significant generational status effect. While FG students obtained lower grades than their CG counterparts in the same biology class (p < 0.01), the VA intervention led to significant improvements in the FG students grades (p < 0.05), resulting in a 50% reduction in the social class achievement gap. In terms of progression into the second-semester biology course, in the control group, CG students (77.7%) were significantly more likely to enroll in the second course in comparison to FG students (66.2%).  Conversely, in the VA intervention group, FG students (85.7%) were more likely to enroll than CG students (74.8%). This represents a 20% increase in enrollment for FG students (p < 0.01) who participated in the VA intervention.  In contrast, CG students were no more likely to enroll regardless of whether they were in the intervention or control group (p = 0.41). The results suggest that a VA intervention can indeed narrow the social class achievement gap, improve the success for FG students in an introductory biology course and other college classes, and help keep them on track to progress in the science sequence.

Factors that threaten a student’s motivation or ability to learn vary from classroom to classroom, but it is vital that educators identify the variables that might influence a student’s success. In addition to the generational differences, other variables such as ethnicity, sex/gender, stress, and cultural mismatch may influence a student’s ability to academically succeed.^7,8 Learners come from different backgrounds and have individual struggles. Some are pretty obvious such as ethnicity and language. But others, like generational differences in educational attainment, are harder to identify and trickier to address. Supporting our learners so they can succeed to the best of their ability starts with acknowledging that barriers exist and doing our best to address those barriers. Whether an achievement gap is the result of stereotype threat or a cultural mismatch, VA interventions can play a positive role in influencing our learners’ success.

As healthcare providers, we strive for ways to bridge the health disparities that exist between people of different social classes. As health professional educators, shouldn’t we be striving for ways to bridge the academic disparities that exist? Taking a 10-minute check-in with our students using a VA intervention could be the difference that a student needs to succeed. I challenge every educator to try this in their classroom. Take 10 minutes at the beginning of class every month to have your students identify and write about what positive traits they value. Is it empathy? Compassion? Athleticism? It doesn’t matter if it’s for a grade or not. But portray it in a way that the students realize it is important to really give it honest thought. We spend so much time teaching what they lack or don’t know. It’s time we start reminding and reaffirming our students that what they currently know or possess is just as important. When we help our students reaffirm interdependent values they perceive as integral to their self-worth, we will see positive improvements in and out of our grade books.

References

1. Saenz, VB.; Hurtado, S.; Barrera, D.; Wolf, D.; Yeung, F. First in my family: A profile of first-generation college students at four-year institutions since 1971. Los Angeles, CA: Higher Education Research Institute; 2007. http://www.heri.ucla.edu/PDFs/pubs/TFS/Special/ Monographs/FirstInMyFamily.pdf
2. Snibbe AC, Markus HR. You can’t always get what you want: Educational attainment, agency, and choice. Journal of Personality and Social Psychology 2005; 88:703–720.
3. Cohen GL, Garcia J, Apfel N, Master A. Reducing the racial achievement gap: A social-psychological intervention. Science 2006; 313:1307–1310.
4. Sherman, DK.; Cohen, GL. The psychology of self-defense: Self-affirmation theory. In: Zanna, MP., editor. Advances in experimental social psychology. Vol. 38. San Diego, CA: Academic Press; 2006. p. 183-242.
5. Sherman DK, Hartson KA, Binning K, Purdie-Vaughns V, Garcia J, Taborsky-Barba S, Tomassetti S, Nussbaum AD, Cohen G. Deflecting the trajectory and changing the narrative: How self- affirmation affects academic performance and motivation under identity threat. Journal of Personality and Social Psychology 2013; 104:591–618.
6. Harachiewicz JM, Canning EA, Tibbetts Y, Giffe CJ, Blair SS, Rouse DI, Hyde JS. Closing the Social Class Achievement Gap for First-Generation Students in Undergraduate Biology. Journal of Educational Psychology 2014; 106(2): 375-389.
7. Smart-Richman L, Leary MR. Reactions to discrimination, stigmatization, ostracism, and other forms of interpersonal rejection: A multimotive model. Psychological Review  2009; 116:365–383.
8. Steele CM, Aronson J. Stereotype threat and the intellectual test performance of African Americans. Journal of Personality And Social Psychology 1995; 69:797–811.

-->