Situational Judgment Tests (SJTs) to Measure Beyond Recall

by Karmen McMinn, PGY1 Pharmacy Community Practice Resident, Mississippi State Department of Health

In order to become a pharmacist, a student must receive a Doctor of Pharmacy degree. This means that they receive several years of education, but does all of that knowledge mean they will be successful? In addition to having a wide range of factual knowledge, pharmacists must also be able to display empathy towards patients and caregivers, work well in teams, and many other qualities that do not rely on the recall of facts.^1,2 These qualities (empathy, team player, etc) are sometimes be referred to as “soft” skills while being able to recall factual knowledge is often referred to as academic or cognitive skills.³ Some have argued that strong academic skills are inversely related to soft skills.^1,2 Thus, someone who earned straight A’s in those early science classes may struggle during advanced practice experiences.

One common requirement for pharmacy school admission is the Pharmacy College Admission Test (PCAT). The PCAT was designed to measure general academic ability and scientific knowledge. This background knowledge is something all students must have in order to be successful in pharmacy school. The problem with a test like this is that it only looks at a student's general academic knowledge while neglecting to assess other important skills a successful pharmacist must possess. For example, the PCAT does not assess a person’s ethical decision-making ability. This is where situational judgment tests (SJTs) might be useful. They can be an effective tool for assessing soft skills.^2,4

Icons made by Smashicons from www.flaticon.com

SJTs are a type of psychometric test in which learners are presented with a realistic scenario or situation. Examinees are then presented with 4 or more actions they could take and instructed to pick the option they believe is the most appropriate – an action they would take in real life. The test taker is often asked to pick not only the most appropriate response but also the least appropriate response. In some SJTs, they are asked to rank the relative appropriateness of the responses from most appropriate to least appropriate.⁵ SJTs can measure traits not related to a person’s ability to recall factual knowledge.⁶ This can be done by making sure questions ask what the person “would do” instead of what someone “should do.”⁷ Skills that can be measured on an SJT include personality traits like conflict management, interpersonal skills, teamwork, and cultural awareness.^3,5 All of these skills can help identify people who would make amazing pharmacists that have the ability to interact with a wide range of people and work effectively as a part of a healthcare team.¹

In one study, investigators at Monash University in Australia developed an SJT. They used experts to evaluate the tool’s validity, reliability, fairness, and to determine the appropriateness of using an SJT as a formative assessment. This study appears to be the first to report on the development, implementation, and evaluation of SJT as a formative assessment for pharmacy students. They developed the test to help identify students that might need more training to develop the soft skills integral to becoming a successful pharmacist.³

Here is an example SJT scenario and directions³

Scenario
Nikhil, a pharmacy student, is working in a community pharmacy. A customer explains to Nikhil that she came to the pharmacy yesterday to collect some blood pressure tablets. However, when she arrived home, she realized that she had been given double the strength of the tablet that she required and has not taken any of the new medications. Nikhil arranges for the pharmacist to correct the medication and apologizes to the customer for the error. However, the customer looks angry and says, “sorry is not good enough.”

Response Instructions
How appropriate are each of the following responses by Nikhil in this situation?

Options: 
1 = a very appropriate thing to do; 2 = appropriate, but not ideal; 3 = inappropriate, but not awful; 4 = a very inappropriate thing to do

Responses

Inform the customer that he has already apologized to her and that there is nothing more that he can do

1. Tell the customer that he was not working yesterday
2. Tell the customer that she needs to calm down
3. Ask the customer whether she would like compensation
4. Ask the pharmacist to come and speak to the customer
5. Provide the customer with information on the pharmacy’s formal complaints procedure

The potential advantages of using SJTs in health professional curricula include building a student’s understanding of the concept “best” and “better” ways of performance and increases self-assessment skills. Self-assessment skills are an important part of continuing professional development. Providing students with feedback and the opportunity for reflection can help motivate further development of these soft skills. It can also be helpful to students by administering multiple SJTs so that they can see their improvement over time.^3,6

There are a few issues that educators should consider before implementing SJTs. First, it is important to make sure the scenario or situation is well described. There must be enough information for a student to be able to fully visualize the scenario. If a student cannot envision the scenario, it will be difficult for them to pick the “best” answer. Secondly, it is best to develop a scenario that does not force a student to choose an action that would go violate their personal beliefs and values.^3,7

Educators can use SJTs to help develop skills and traits, such as interprofessional skills and cultural sensitivity, that help students become better pharmacists. These tests can be used as a tool to assist with admission decisions but also deployed repeatedly throughout the curriculum in order to document change over time. By using SJTs for formative purposes, an institution can personalize the development of soft skills, focusing the student’s attention on weaknesses as well as uncovering strengths. In the end, every school wants to graduate well-rounded and well-educated pharmacists.^1,3


References:

1. Gilchrist A. Top 5 Pharmacist Personality Traits. Pharmacy Times. 2015 July 23.
2. Jones J, Krass I, Holder GM, Robinson RA. Selecting pharmacy students with appropriate communication skills. [Internet]. Am J Pharm Educ 2000; 64(1): 68-73.
3. Patterson F, Galbraith K, Flaxman C, Kirkpatrick CMJ. Evaluation of A Situational Judgement Test to Develop Non-Academic Skills in Pharmacy Students. Am J Pharm Educ 2019 [Ahead of Print]
4. About the PCAT. [Internet]. Pearson. 2019. Cited 2019 Nov 12.
5. Situational Judgement Test [Internet]. Psychometric Tests. 2019 Jan 9. [cited 2019 Nov 11].
6. Austin Z, Gregory PAM. Evaluating the accuracy of pharmacy students’ self-assessment skills. Am J Pharm Educ. 2007;71(5): Article 89.
7. Assessment & Selection. Other Assessment Methods. Situational Judgment Tests. United States Office of Personnel Management. 2019. Cited 2019 Nov 21.

Creating Meaningful International Pharmacy Experiences

by Katherine Baker, Pharm.D., PGY1 Pharmacy Practice Resident, North Mississippi Medical Center

International pharmacy experiences are increasingly popular. Today, students are eager to see the world and compare their learning environments to those of other students around the globe. In 2010, a survey of 114 pharmacy schools in the United States revealed that 40 had an active global/international program.¹ Having completed a global pharmacy experience during pharmacy school, I can speak to their value. I was fortunate to spend 5 weeks in Phitsanulok, Thailand at Naresuan University teaching hospital. What seemed like a distant dream soon became a reality when I found myself in rural Thailand with one of my best pharmacy school friends relying on the Google translator application to help us communicate. While international experiences have important benefits simply from “being there,” it is equally important to know how to design these experiences to maximize learning.

Icons made by turkkub from www.flaticon.com

First, it important to ensure that international experiences foster the development of the competencies required for graduation. Second, that there should be policies in place to ensure that the international site and faculty are qualified.² To address these requirements and the increasing demand for global health experiences, several initiatives have sought to improve the quality of international pharmacy rotations. Indeed, the American Association of Colleges of Pharmacy developed a special interest group that includes a dedicated team of individuals who discuss topics and ideas related to global healthcare.¹  

Prior to establishing an international experience and sending student pharmacists across the globe, great attention should be paid to ensuring that both the student and the host site are prepared for the visit. The planning period for my international rotation was stressful due to unknowns that come with living and learning in a foreign country. These stresses were alleviated when the host site sent me documents with helpful information about our living arrangements and coordinated everything before arrival. A number of articles have explored important considerations that must be taken into account before developing a Global/International Advanced Pharmacy Practice Experience (G/I APPE). These considerations include host country, home institution, host institution, faculty/preceptors, and student issues.² 

The host country is an important factor to consider before establishing an international rotation. While having a well-thought-out curriculum is necessary, ensuring a safe and functional learning environment in a foreign country is even more important. Pharmacy schools need to ensure that students are well informed about the country they will be traveling to.  Students should try to learn about the local culture(s) and basic parts of the native language. This will help them while navigating abroad as well as improving patient interactions during the experience. While in Thailand, I worked in an anticoagulation clinic and was asked to interview and counsel patients. I struggled to communicate and had to rely on my preceptor. This particular experience would have been much more meaningful had I learned the language ahead of time. Logistics of the rotation such as visas, living arrangements, transportation, and financial considerations should also be addressed well in advance. 

Equally important are the host institution and the curriculum. To justify an international experience, the host institution should be vetted and deemed an appropriate pharmacy experience. It is recommended that the host and home institutions have a designate representative. These designated faculty members will be responsible for communicating student educational needs, learning outcomes, and detecting any red flags.¹  In addition, a thorough and thoughtful syllabus must be drafted and approved. Just like any other course or experience in pharmacy school, the syllabus should lay out expectations, goals, objectives, and grading criteria.¹    

Student interaction while on rotation is another important consideration to create a meaningful experience abroad. For students to take full advantage of their international pharmacy experience, they should be meaningfully engaged and aware of what they are learning. One descriptive study discusses the importance of reflective writing while on rotation. Based on student reflections, the authors noted a pattern to student learning while on a G/I APPE. Students use “communication, problem-solving, and adaptability”³ skills while on rotation and then reflect on the major differences between their home and host country. There a couple of ways to get students to reflect about their experiences.  One method is to ask students to keep a personal journal and regularly writing entries.  Another way to encourage reflection is to require students to give a presentation to peers after completing the experience. My pharmacy school required each student to create a weekly travel blog reflecting on their experiences (check out our blog: Thailand Adventures - Two Pharmacy Students Finding Their Way Through Thailand). This exercise helped us to reflect and translate what we learned abroad and apply it to our lives in the United States. Our blog essays were also useful to future participants. Indeed, many of our weekend excursions were based on blog posts from previous students!

Global pharmacy experiences are not only an adventurous, once-in-a-lifetime experience, they also enhance a pharmacy student’s professional advancement. In a world that is becoming smaller and more connected through technology, a G/I APPE will give students an opportunity to grow their perspective and sensitivity towards people from other cultures.¹ Students who have participated in a global pharmacy rotation are more likely to be culturally aware and “demonstrate increased interest in volunteerism, humanitarianism, and public health.”² Moreover, international experiences often help students improve their communication skills, problem-solving skills, and adaptability.³  

Having participated in an international rotation myself, I can attest to many of these benefits. If I could give any piece of advice, I would encourage students heading into an international experience to learn some of the host country’s language and keep a travel journal or blog. It’s also important to give both the host and home sites feedback. The home institution often go to great lengths to ensure that the rotation will be maximally beneficial but the best information is feedback based on personal experience. The host institution that I completed my rotation at required students to complete an in-depth exit survey. The institution wanted an honest review of the experience and it was important to openly discuss any concerns to better serve students for the future.

International pharmacy experiences are a once-in-a-lifetime opportunity that I would recommend to all pharmacy students. If the experience is well planned and executed by both the host and home institutions and if the student approaches the experience with enthusiasm and an open mind, it will be a personal and professional growth experience the student will never forget.

References:

1. Steeb DR, Overman RA, Sleath BL, Joyner PU. Global Experiential and Didactic Education Opportunities at US Colleges and Schools of Pharmacy. Am J Pharm Educ. 2016;80(1): Article 7. doi:10.5688/ajpe8017
2. Alsharif NZ, Dakkuri A, Abrons JP, et al. Current Practices in Global/International Advanced Pharmacy Practice Experiences: Home/Host Country or Site/Institution Considerations. Am J Pharm Educ. 2016;80(3): Article 38. doi:10.5688/ajpe80338.
3. Steeb DR, Miller ML, Schellhase EM, et al. Assessment of Global Health Learning Outcomes on International Experiences. Am J Pharm Educ. 2019. doi: 10.5688/apje7586