Mastery- vs Performance-Oriented Goals and Their influence on Motivation and Success

by Michelle Ha, PharmD, PGY1 Pharmacy Practice Resident, Baptist Memorial Hospital-North Mississippi

Many of us are familiar with setting goals: short-term, long-term, professional, and personal. But do ever think about your goal orientation? The concept of goal orientation was developed by psychologists in the 1980s and explains the mindset that an individual has when developing and achieving goals. There are two primary ways one can approach goals: mastery or performance.

Learners who have a mastery-orientation focus on learning to perform better in the real world – for example, learning in order to become more skillful at taking care of patients. Learners who have a performance orientation focus on demonstrating competence relative to others – for example, performing well on exams, getting high marks on performance evaluations, and (in general) looking smart in front of others. In other words, those who set mastery-oriented goals tend to compete with themselves, and satisfaction is based on internal factors. Those who have performance-oriented goals are primarily motivated by external feedback and validation.^1,2

Summary of Mastery vs Performance-Oriented Goals.³

Mastery-Oriented	Performance-Oriented
More likely to be intrinsically motivated	More likely to be extrinsically motivated
Seek feedback that accurately describes their abilities and helps them improve	Seek feedback that flatters them
Choose tasks that maximize opportunities for learning and seek out challenges	Choose tasks that maximize opportunities for demonstrating competence and avoid tasks that make them look incompetent
Views errors as a normal part of learning and uses errors to improve	Views errors as a sign of failure and incompetence
Satisfied with performance as long as they make progress	Satisfied only if they succeed
Views teacher as a resource	Views teacher as a judge, rewarder, or punisher

Goal orientation is important because it influences one's motivation and selection of learning strategies. Goal setting starts at a young age - regardless if you were aware of it or not. Thus, many people are unaware of the impact that their goal orientation has on their performance in school and life.

Imagine you have two students in your class: Marcus and Marilyn.

Marcus wants to perform well in school. He studies hard and would like to earn an “A” in pharmacology. He pushes himself hard to make his parents proud. However, he sometimes worries about whether he'll get an "A", failing to live up to his parents' expectations, and looking incompetent in front of the teacher/preceptor. He knows what learning methods work best for him and does not want to try other studying strategies. When he performs poorly on an exam, he submits challenges to the instructor in order to “get” points, even if he really didn’t understand the concepts that well. Although Marcus nearly always meets his goals, he beats himself up when he falls a bit short.

On the other hand, Marilyn enjoys the process of learning. She goes beyond the expectations set by the instructor in her pharmacology course and seeks out outside opportunities to learn more about the topic. While getting an “A” in the course would be great, she’s not too worried about the grade, so long as she’s learning new things she feels are important to her future career. She is most happy when she meets her personal goals and continues to strive to do better each day. When she struggles to learn about a concept, she seeks help and tries new strategies, and uses new resources.  When she performs poorly on an exam, she seeks help from the instructor and signs up for tutoring help. Although Marilyn nearly always meets her goals, she knows that stretching herself and falling short is part of the process.

A person can have both mastery and performance-orientations.  Indeed, most people don’t fall exclusively in one camp or the other and their orientation can be different in different circumstances and courses. However, researchers have found that mastery-oriented goals are more effective in terms of student motivation. Satisfaction is not related to external factors. Performance goals are often helpful in the short-term; however, they may stifle a student growing to their full potential.

Back to our example, Marcus' goal was to make an A in pharmacology. Once he believes he’s achieved this goal in the class, he may be less motivated and prefer to “coast” through the rest of the course. "I've made As on the last three exams and as long as I get at least 67 points on the final, I’ll still get an A." In his mind, his success is determined by his grades. If Marcus ends up with a B in the course, he’ll probably avoid taking the pharmacology elective for fear he won’t do well and his GPA will be negatively impacted.  In contrast, for Marilyn, if she gets an A on the first 3 exams in pharmacology, she will continue to study hard and may even continue to seek out opportunities to learn more after the course has ended. If she falls short of a good grade in her pharmacology course, Marilyn might sign up for the pharmacology elective in order “to get better” at something she feels is critical to her success. The factors that move her forward are internal and within her control. Thus, it is easier for Marilyn to stay resilient and adapt during times of struggle. Marilyn experiences less anxiety and stress when she falls short.

To examine the influence of goal orientation on motivation, one study evaluated how students responded to negative feedback. The investigators used a simulation game. Prior to participating in the simulation, each student’s orientation was measured using a multi-item scale adapted from Ames and Archer (1998). Students then completed a Marketing Management Experience, where they manage a simulated company and competed against other groups. Learning implies a change in behavior which was measured by comparing the survey results over time. While both mastery and performance-oriented students performed well during this simulation, those in the mastery-oriented goal group tended to take negative feedback better. In the performance-oriented goal group, some of the students develop a learned behavior known as "learned helplessness". This is a term that describes the reaction to failure that reduces the desire to place oneself in that circumstance again. 

In another study, researchers surveyed medical, pharmacy, and veterinary students. The students completed a series of surveys at the beginning of 5 consecutive semesters that measured their mastery orientation, performance orientation, and self-efficacy. While most students were stable in terms of their goal orientation, there were some differences between students based on their gender, grades, and self-efficacy. Self-efficacy was the biggest predictor of those who adopted mastery-oriented goals. Self-efficacy is the strength of a student's belief in their ability to complete a task.⁵

Self-efficacy in itself is another important concept!  In brief, it is a good predictor of motivation and learning, especially in health professional students and practitioners. Someone with a high level of self-efficacy can visualize a positive outcome and is more likely to perform well in their daily duties such as delivering patient care. The ability to adapt and remain resilient in times of unexpected setbacks is also more easily managed among those with a high level of self-efficacy.⁶

Marilyn, our student who is focused on learning and improvement, is the ideal student. However, it is not an innate trait. Mastery-orientation to learning can be taught and cultivated! As educators, we must encourage students to approach their learning with mastery-oriented goals in mind. Because self-efficacy and mastery-oriented goals go hand in hand, it is important to give students a choice when assigning learning activities.  Asking students to think about how their learning activities link to their careers will increase the likelihood of students developing mastery-oriented goals.^1,7 Allowing students to make choices and linking those choices to career aspirations will help students feel autonomous and motivated to learn.⁷ Below is a list of other things that you may wish to try in the classroom to foster a mastery-oriented mindset.⁷ 

1. Be a role model for students. Show them that you have made mistakes but have learned from them instead of hiding them or avoiding them.
2. Give positive, constructive feedback that focuses on personal improvement. Focus less on grades and more on mastering the skill or concept.
3. Don’t compare the student’s performance to peers. Emphasize growth.  Compare the student's performance to previous performance.
4. Foster a community of trust within the classroom so that students are more likely to seek help from peers and you.

While students with performance-oriented goals are no less likely to get good grades, they may be less likely to develop life-long learning habits. Mastery-oriented goal-setters strive for improvement daily and want to become better even if that means venturing through unknown challenges. Students who approach their goals as an opportunity to master something will be the ones who love what they do and are motivated to learn more. Instilling a mastery-oriented mindset in your students will groom them for a future of success.

References:

1. Donald B. Stanford psychologist: Achievement goals can be shaped by environment [Internet]. Stanford University. 2012 [cited 2021 Jan 11].
2. Bråten I, Strømsø HI. Epistemological beliefs and implicit theories of intelligence as predictors of achievement goals. Contemporary Educational Psychology. 2004 Oct;29(4):371–88.

3. Mastery Vs Performance Goals. [Internet] Western Oregon University. [cited 11 January 2021].

4. Gentry JW, Dickinson JR, Burns AC, Mcginnis L, Park JY. The role of learning versus performance orientations when reacting to negative outcomes in simulation games. Association for Business Simulation and Experiential Learning. 2006;33.

5. Kool A, Mainhard T, Brekelmans M, van Beukelen P, Jaarsma D. Goal orientations of health profession students throughout the undergraduate program: a multilevel study. BMC Med Educ. 2016 Dec;16(1):100.

6. Zamani-Alavijeh F, Araban M, Harandy TF, Bastami F, Almasian M. Sources of health care providers’ Self-efficacy to deliver Health Education: a qualitative study. BMC Med Educ. Jan 2019;19(1):16.
7. Svinicki M. Fostering a Mastery Goal Orientation in the Classroom [Internet]. Austin; 2010 [cited 2021 Jan 23]. p. 25-28.

Team-Based Learning Promotes Self-Reflection and Creates Lifelong Learners

by Austin Simmons, PharmD, PGY1 Pharmacy Practice Resident, Magnolia Regional Health Center

During the first two years of any healthcare provider’s schooling, students often find themselves navigating their curriculum and trying to remember all of the little details that are thrown their way. Most students don’t engage in much self-reflection during this period due to workload demands. Then comes the third and fourth years of school. This is when students try to piece it all together and decipher what they know and what they will need to work on as they transition from student to independent practitioner. I believe team-based learning prepares students to transition from dependent learners to lifelong learners and promotes self-reflection.

Team-based learning is built on the constructivist theory which states that learners process new material and integrate it with existing understandings in order to form a new cognitive structure that is unique to them.¹ Hrynchak and Batty wrote about team-based learning and provide an analysis of how constructivist theory plays a role in student development. Essentially, the professor is a facilitator for learning.  The students encounter inconsistencies between their preconceptions and new experiences.  In team-based learning, the focus is on relevant problems and accompanied by group interactions, and this often leads to reflection.² They go on to explain that team-based learning can be used in large classes that are divided into smaller groups.  The goal should be to maximize the diversity within the teams.² Let’s take a look at the framework team-based learning uses to promote self-reflection and build lifelong learners.

Classically, the design of team-based learning is a three-step process that involves student preparation, readiness testing, and application-focused exercises.³ Now, how does this framework promote learning and increase student self-awareness? Let me draw from my own experience.  At my pharmacy school, we had a class called case studies. The intent of this class was for the students to prepare before the class session and use prior knowledge.  We would then engage in collaborative work discussing a patient case in our assigned small group. Then after our small group discussion, the classroom as a whole would come together and the professor would facilitate a conversation by asking each small group questions related to the patient case. The instructor would also encourage the entire class to openly respond to these questions. It was during these interactions, in our teams and the entire class, that we’d encounter inconsistencies between our preconceptions and the perspectives of our instructor as well as other students.² Doing so, in theory, prompts each student to reflect on his/her own understanding of the material. But what are the individual processes or parts that make team-based learning work and what are the important takeaways for a student and instructor?

From my own experience, I found that the immediate feedback from my classmates and the instructor allowed me a way to rapidly assess how well I understood the material. Our class was a 3-hour session which included the time for our small group discussion. If we discussing a case about a patient with diabetes, I might ask myself: what do the blood glucose data mean?  What are the blood glucose goals for the patient? I would rapidly assess and begin self-reflection by asking myself if I needed to review more about the treatment of diabetes. The immediate feedback is a big part of what makes team-based learning work and vital to increasing self-reflection.⁴

I believe it is important to keep in mind that all aspects of the team-based learning framework must be implemented and the intentional guidance provided by an instructor is essential.⁵ Martirosov and Moser found that a student’s understanding and performance were significantly reduced in the absence of appropriate guidance.⁵  To maximize learning, the instructor must ask probing questions. For example, a patient case about diabetes helped promote self-reflection by getting students to think through the data and recommend starting a medication, perhaps an angiotensin receptor blocker (ARB). Then the instructor would ask questions about why they think the patient should receive an ARB instead of an ACE inhibitor. By prodding the students to explain their choices, it forces them to reflect on that choice and critically examine the thought process. An instructor is the glue that prompts high-level cognitive processing and pulls forth the student’s previous knowledge.  In this way, team-based learning helps students put the pieces together.

Team-based learning is an excellent instructional strategy that many curriculums have used. Team-based learning requires students to engage in reflection because it frequently challenges their preconceived understanding of the material and, in turn, promotes life-long learning.  With guidance from the instructor, students must defend their choices, and this helps them “put it all together.” I firmly believe team-based learning helps students develop lifelong learning skills and helps them become excellent healthcare practitioners.

References:

1. Moon J. A Handbook of Reflective and Experiential Learning. 1st ed. Hoboken: Taylor and Francis; 2004.
2. Hrynchak P, Batty H. The educational theory basis of team-based learning. Medical Teacher [Internet]. 2012 [cited 2020 Nov 3];34(10):796-801.
3. Overview - Team-Based Learning Collaborative [Internet]. Team-Based Learning Collaborative. 2020 [cited 2020 Nov 3].
4. Whittaker A. Effects of Team-Based Learning on Self-Regulated Online Learning. International Journal of Nursing Education Scholarship [Internet]. 2015 [cited 2020 Nov 4];12(1):45-54.
5. Martirosov A, Moser L. How Team-Based Learning Can Promote the Development of Metacognitive Awareness and Monitoring. American Journal of Pharmaceutical Education [Internet]. 2020;84(11): Article 848112.

Teaching Health Profession Students the Skills Needed to Maintain Wellbeing

by Anna Carroll Harris, PharmD, PGY1 Pharmacy Practice Resident, University of Mississippi Medical Center

Numerous studies have been published exploring burnout among healthcare workers. Health profession students are also prone to burnout due to the vigorous course load required to obtain their degrees. The WHO defines burnout as a syndrome that is directly correlated with an environment that exposes workers or students to chronic stress and where the stressors are not successfully handled. It is characterized by feelings of emotional exhaustion, amplified feelings of negativism towards one’s job, and decreased professional worth.

The occurrence of burnout not only affects those working and learning in the healthcare industry, but also the patients to whom they provide care. For example, pharmacists who are experiencing increased levels of stress and emotional exhaustion may feel a sense of depersonalization towards patients they are caring for. This in turn can lead to medication errors and harmful events for patients.¹ It is imperative that schools and colleges of pharmacy, and other health professional degree programs, help students develop the skillset and positive behavior practices that needed to maintain their wellbeing and prevent burnout throughout their careers.²

Many professional organizations have noted the need to provide health profession students and healthcare practitioners with resources to encourage a state of well-being and prevent burnout. The American Association of Colleges of Pharmacy has published two policies, a 2017 and 2018 statement, in response to the increasing realization that burnout is prevalent. Both statements encourage and hold schools and colleges of pharmacies accountable for effectively promoting wellness and implementing management methods directed to students, faculty, preceptors, and staff.³ In reaction to these statements, schools and colleges of pharmacy across the country are putting programs into place that foster an environment for creating and maintaining well-being. For example, the Ohio State University College of Pharmacy has a “Wellness Corner” dedicated to providing faculty, staff, and students an environment that promotes and protects well-being. They have been recognized across their campus as having a strong culture of wellness by implementing evidence-based wellness strategies and providing tools to achieve a sense of wellbeing.⁴

The University of Kentucky College of Pharmacy completed a study where they “nudged” pharmacy students to adopt well-being behaviors. Over a span of 4 months, a series of optional well-being challenges were embedded in their pharmacy management course. These challenges included limiting smartphone use, emphasizing feelings of gratitude, good sleep hygiene, and engaging in regular exercise. Participants completed a reflection at the end of the course that explored reasons for participation, prior behaviors, and if participants planned to continue to implement the new behaviors after completion of the challenges. The majority of participants indicated that they planned on maintaining the positive behaviors moving forward.⁵

The specific stressors that lead to burnout have been identified in many studies. A study that included pharmacy students in an urban Midwestern region identified strategies that pharmacy students utilized to protect their wellbeing and prevent burnout. Students participating in a social and administrative science course were to write a reflection on factors that they believed had the greatest impact, either positively, negatively or mixed, on their wellbeing during pharmacy school. Four specific themes emerged as strategies that students use to cope with stressors during pharmacy school:⁶

1. Availability and accessibility of institutional resources
2. Personal time management and organizational strategies
3. Personal, mental, and physical health strategies
4. Activities that maintain social relationships

These results provide schools and colleges of pharmacy specific ways they can augment their campuses' attempts to foster wellbeing. Ensuring that institutional resources, such as the medical library and faculty, are readily available to students can help reduce stress and maintain wellbeing. Offering counseling and health services to those who needed them is supportive of students’ wellbeing. A few of the wellness activities mentioned in the Ohio State University College of Pharmacy “Wellness Corner” were a take five-station, a mental timeout area where students could play a game or create a craft, as well as monthly wellness walks. Ensuring that students maintain a healthy balance between schoolwork and leisure activities can reduce stress. Emphasizing the need for students to take time for themselves to socialize with friends and family and maintain hobbies is important.

In the unprecedented times of a pandemic, providing students with resources and teaching them skills to protect their well-being is more essential than ever. With COVID-19 disrupting the lives and wellbeing of so many, health profession students are dealing with the added stressors of helping take care of family members and serving on the frontlines of healthcare, in addition to their demanding coursework.⁷ The loss of person-to-person contact and being isolated away from one’s family has taken a toll on many students. What once provided a means for students to reset and take a break from the rigors of academic coursework is now discouraged.  Schools and colleges need to find creative ways to provide ongoing support to their students, faculty, and staff.  See Table 1.

Table 1: Examples of support during a pandemic

Virtual group exercise	Email check-ins	Virtual mentorships programs
Virtual game nights	Virtual group meditations	PPE drives/mask-making
Virtual book clubs	Virtual dinner dates	Virtual tutoring

As health profession students graduate, they will continue to experience stressful times and emotional exhaustion that can lead down the path of burnout. Health profession programs should work to implement programming and strategies early in their curricula that can provide students with a skillset to prevent burnout. General professional development courses, which are often part of the curriculum, would be a great place to embed lectures about managing stress and including periodic wellbeing challenges for students. These longitudinal courses should be pass/fail due to the nature of the content and should encourage students to adopt and execute tactics that best fit their personal circumstances and needs. Learning about and implementing these healthy habits while in school can help students cope with the stressors they will face throughout their careers.

References

1. World Health Organization. Burn-out an "occupational phenomenon": International Classification of Diseases. Accessed November 18, 2020.
2. Hagemann TM, Reed BN, Bradley BA, et al. Burnout among clinical pharmacists: Causes, interventions, and a call to action. J Am Coll Clin Pharm 2020; 3:832–842.
3. American Association of Colleges of Pharmacy. AACP Statement on Commitment to Clinician Well-being and Resilience. Accessed November 18, 2020. https://accpjournals.onlinelibrary.wiley.com/doi/full/10.1002/jac5.1256
4. The Ohio State University College of Pharmacy. Wellness Corner. https://pharmacy.osu.edu/wellness-corner. Access November 18, 2020.
5. Cain J. Effectiveness of Issuing Well-being Challenges to Nudge Pharmacy Students to Adopt Well-being Protective Behaviors. Am J Pharm Educ 2020; 84(8) Article 7875.
6. Abraham O, Babal, JC, Brasel KV, Gay S. Strategies first year doctor of pharmacy students use to promote well-being. Currents in Pharmacy Teaching and Learning. 2021; 13:29–35.
7. Schlesselman LS, Cain J, DiVall M. THE COVID-19 PANDEMIC ACROSS THE ACADEMY: Improving and Restoring the Well-being and Resilience of Pharmacy Students during a Pandemic. Am J Pharm Educ 2020; 84 (6) Article 8144.

Community Baby Showers: An Innovative Approach to Teaching New Mothers Sleep Safety

by Megan Carter, Doctor of Pharmacy Candidate, University of Mississippi School of Pharmacy

Summary and Analysis of: Ahlers-Schmidt C R, Schunn C, Hervey A M, et al. Redesigned community baby showers to promote infant safe sleep. Health Education Journal 2020; 79(8): 888-900.

When looking through the Health Education Journal, this article piqued my interest because I was aware of the importance of safe sleeping habits for infants.  My mom works for the Alabama Department of Public Health and has been reviewing infant death cases for about 15 years.  Although she doesn't reveal details about the cases she reviews, she has shared a couple of particularly heart-wrenching stories regarding babies who died.  Unfortunately, these stories are not rare, as nearly 3500 infant deaths in the USA are due to sleep-related causes. Unfortunately, although we know much more about sleep safety during infancy, this number has not declined in recent years.¹ These kinds of deaths are preventable if parents are properly educated about how to create a safe sleeping environment.  This study aimed to provide sleep safety education to mothers, specifically mothers from low-income communities, in a relaxed environment. The authors assessed a novel approach that could increase knowledge and health-promoting behaviors.

The authors of this study designed their educational intervention — including recruitment, lesson plans, materials, and assessments — around constructs from the Health Beliefs Model.  The intervention was delivered at community "baby showers" conducted in Sedgwick County, Kansas, and targeted women who were pregnant or who had recently delivered.  Upon arrival at the shower, participants were asked to complete a pre-assessment and information card.  Organizers divided the women into tour groups of 3-5 participants with a volunteer guide who led the women around the various vendor and educational booths for brief presentations.  The last stop for each tour group was the Safe Sleep Crib Demonstration.  Certified nurses or safe sleep instructors used a demonstration crib with safety-approved items to demonstrate their effectiveness as well as examples of unsafe items that are a hazard to infants.  Instructors provided tips for removing hazardous items from the infant's sleep environment.  The sessions were not time-constrained, allowing time for participants to ask questions.  After the shower, participants took home a safety-approved portable crib, blanket, and educational handouts/materials.

These events were held twice yearly (March and October) from Spring 2015 to Spring 2019 and recruited women using a variety of means including fliers at churches and clinics, maternal and child health programs, social media posts, and through partner organizations.  The program specifically targeted locations that served low-income communities, as this was the population that was most at-risk for sleep-related infant deaths.  During the study period, nine “community baby shower” events were conducted. The participants came from a range of racial and ethnic backgrounds: Non-Hispanic Black (30.4%), Non-Hispanic White (30.4%), and Hispanics (25.1%).  Greater than 70% of participants had only a high school education or less and greater than 70% were on Medicaid or uninsured.  It is also notable that less than half of participants received prenatal care from a private provider and about 20% received care from a county health department, community center, or received no routine care. The pre-assessment consisted of true/false statements developed around the Health Belief Model constructs on infant sleep safety and were compared to the responses to the same questions given as a post-assessment.  McNemar's test for paired dichotomous variables was used to analyze differences in pre- and post-assessment responses along with the McNemar odds ratio statistic.  The following true/false questions were included on the pre- and post-assessments:

• My baby is at risk of dying of SIDS
• Loose blankets in the crib can cause infant death
• Sleeping with my baby can cause infant death
• Putting my baby alone, on the back in a crib will help protect her
• My baby will choke on his back
• People tell me different things about how my baby should sleep and I don't know what to do.
• I can't keep my baby warm without blankets.
• I don't have room for a crib in my room.

The results of the study demonstrated statistically significance (p<0.001) improvements in the participants' responses in all but two of the assessment questions.  Responses to questions about knowledge and intentions showed changes in sleep positions, sleep locations, crib items, and plans to discuss safe sleep with others.  Overall, these results appear promising and events such as the community baby shower provide an excellent opportunity to teach sleep safety to mothers.

The results look promising, but as with any study, statistically significant results don't always equate to an improvement in outcomes.  This study did have several strengths, as the participants are representative of the target population and the assessment questions were based on the Health Belief Model and evaluated by the Medical Society of Sedgwick County's Safe Sleep Taskforce.  On the flip side, this study was conducted in one community, so may not be generalizable to other communities.  The study also targeted individuals from low-income areas with lower education, so the results may not apply to mothers in higher-income neighborhoods with greater levels of education.  The recruiting methods did yield a diverse participant population but relatively few dates that the event was held likely limited many women from attending.  Another potential issue was the true/false statements included in the assessment.  Several of the statements are subjective and others are potentially confusing, which may have contributed to some of the nonsignificant results.  Improving the clarity of these statements could improve the accuracy and validity of this study.  While the results were promising, I would be interested to see if the participants put their new knowledge into practice.  Are mothers able to identify hazardous materials in their home? Do they remove or replace these items?  Have the rates of infant death or hospital visits due to unsafe sleeping habits improved in this county as a result of the educational intervention?  Additionally, the results could have been biased as the group who developed the program assessed the results.  Moreover, there was no control group who received instruction in a more “traditional” manner.

Overall, this study proves that educational programs that structure their lesson plans around the Health Belief Model and offered in non-traditional environments can lead to changes in behavioral intentions.  It is important to recognize that instructional programs can be implemented outside of the traditional classroom settings and that informal community events can a venue where patients can learn about important health topics in a fun and engaging way.

References

1. About 3,500 babies in the US are lost to sleep-related deaths each year. (2018, January 09). Retrieved November 30, 2020, from https://www.cdc.gov/media/releases/2018/p0109-sleep-related-deaths.html
2. Ahlers-Schmidt C R, Schunn C, Hervey A M, et al. Redesigned community baby showers to promote infant safe sleep. Health Education Journal 2020; 79(8): 888-900.

The Importance of Post-Exam Quality Assurance

by Karmen Garey, PharmD, PGY-1 Baptist Memorial Hospital – North Mississippi Pharmacy Resident, University of Mississippi School of Pharmacy

From the students’ perspective, once they hit “submit” after completing an exam they think “Thank goodness that’s done!” However, for teachers, there is still some critical work to do. Now it’s time to review the performance data to ensure the examination was fair and measured what was intended. Here are a few tips and strategies to assess the quality of an exam.

Make certain the exam (as a whole) is a “good” one 

Before the exam is administered to students, a good exam should be written with the following goals in mind:^1,2

• An exam should address multiple levels of Bloom’s taxonomy — from knowledge recall to application and analysis.
• The exam should include a variety of questions that test a range of concepts that map back to the learning objectives.
• The consistency of the exam's performance over time is important. An exam should routinely perform the same from year to year despite some changes to the questions.
• An exam should measure the learning outcomes and course material it was designed to test.

Make certain the questions included on the exam are “good” ones

There are two types of questions that should be included on exams: mastery questions and discriminating questions.  Mastery questions are those questions that students are expected to excel on.³ This type of question is typically a “knowledge level” question in Bloom’s Taxonomy. The questions often test factual recall and the recognition of fundamental material.²  These questions might be called “gimmie questions” by the students; however, teachers include these questions to ensure that students have a firm understanding of the basic but super important concepts or facts.  Discrimination questions, on the other hand, are intended to identify students who have a deeper knowledge of the material and separate students into different performance levels (e.g. identify "A", "B", and "C" students).  Higher-performing students are expected to answer these questions correctly more often than lower-performing students.  This type of question often targets the comprehension, application, analysis, synthesis, or evaluation cognitive level in Bloom’s taxonomy. These questions require an in-depth knowledge of the subject matter.²

Next, let’s look at the distractors.  Does each question include appropriate distractors?³ A distractor is an answer choice that, while wrong, sounds and appears like it could be plausible. A good distractor should be clear and concise and should be similar in structure and content to the correct response. Savvy test-takers have learned to spot answers that seem different in some way, so even small variations in the style, subject matter, and length of the answer choices can provide clues. 

Next, is the question stem clearly written.  Is it clear what the learner is being asked?  Or is the question open to interpretation?  When writing questions, it is important to ensure that the question is not misconstrued.  Sometimes students will overthink a question and try to find the hidden meaning when there is none. To avoid this problem, use words that are unambiguous.  Avoid phrasing that could be cryptic.

Finally, is the answer to the question correctly keyed.  If a lot of students selected the “wrong” answer, it's possible that the question was miskeyed.  While this is not something that happens often, it does happen! So it is always a good idea to double-check that the correct answer was selected on the answer key. 

Some other things to consider as you look at the post-exam performance data.  How did the exam scores look last year? While a group of students performing much better or much worse than previous year’s students is not always an indication that the exam is invalid, it should prompt additional questions.

• Was the material taught in a manner that was different from previous years?
• Was the exam formatted or delivered differently?
• Could the students this year have been less (or better) prepared in some way to comprehend the material?
• Is cheating suspected?
• If there are multiple instructors, did students received different messages about the content?

The answers to these questions may not be obvious or even relevant, but it is something to keep in mind.

Use the post-exam statistical analysis to identify problem questions³

As technology becomes a more integral part of exam delivery, it enables a wealth of data that can be used for post-exam quality assurance. Most post-exam statistical analysis tools report similar elements; however, the names may be slightly different. ExamSoft is among the most common exam delivery tools available today and routinely reports these statistics:

• Item Difficulty represents the difficulty of a question. It reports the percentage of students who correctly answered the question. The lower the percentage the more difficult the question. There is not a set number that the item difficulty should be but the number should be used to ensure the intent behind the question matches the number. For example, if the teacher wants the item to be a mastery question, the difficulty should be 0.90 to 1.00 with very few students getting the question wrong.  If the question is meant to separate those who have a firm grasp on the material vs. those who don’t, lower levels are acceptable. An instructor may have a difficulty “cutoff” number in mind where anything below 0.6 (for example) prompts additional analysis of the question.

• Upper/Lower 27%, Discrimination Index, and Point Biserial are each calculated differently but they report a similar concept. Stated simply, they all determine whether the top performers on the exam achieved better results on a question compared to those who did not perform well. If the top performers don’t out-perform the poor performers, the question should be assessed to determine why.
  • Upper 27% / Lower 27% - what percentage of the top 27%  vs. the bottom 27% of performers got the question correct.
  • Discrimination Index – this represents the difference in performance between the best performers vs. the lowest performers.
  • Point Biserial – indicates whether those who answered correctly on a specific item correlates with doing well on the exam overall.  In other words, does performance on this question predict whether a student did well (or not so well) on the exam? 

 

Correlation with Overall Exam was	Point Biserial
Very good	>0.3
Good	0.2-0.29
Moderate	0.09-0.19
Poor	<0.09

So, let’s look at the statistical analysis from two example questions. 

• This was a mastery question — students are expected to do well on this question. It’s a fundamental concept that all students should know.
• The Discrimination index = 0.04 which indicates almost no discrimination between the top and bottom performers. In this case, because it’s a mastery question and we expected all students to perform well on this question.  Thus, we don’t expect this question to discriminate between the best and worse performers.
• The Point Biserial = 0.10 indicating this question only moderately correlate with doing well on the exam overall. Again, the top and bottom performers performed quite similarly on this question, so there won’t be a strong correlation between the performance on this question and the overall exam.
• If this question was not intended to be a mastery question, perhaps the material was taught particularly well … or maybe there was cheating involved

Now let’s take a look at a question where only 66% of the students selected the correct response.

• Item difficulty = 0.66 so 66% of the students selected the correct response. This is not a bad thing but it is important to make sure the students who understood the material were more likely to get this question right.
• This is intended to be a discriminating question, so let’s make certain it’s actually discriminating between the best and worse performers.
• Look at the Upper vs. Lower 27%: 82% of the top performers got this question correct. Only 46% of those who performed the poorest on this exam got this question correct.
• Discrimination Index: 0.36. This question did a good job discriminating between the best and worst performers on this exam.
• Point Biserial = 0.28 Performance on this question has a good correlation with the student’s overall exam performance.

While there are no hard rules for how to analyze an examination, the strategies I’ve outlined in this blog post are some of the best practices every teacher should follow. It is important to follow a systematic process and establish “cut-offs” in advance. The key is to be clear and consistent from exam to exam.

References

1. Brame C. Writing Good Multiple Choice Test Questions. 2013. Accessed December 3, 2020.
2. Omar N, Haris SS, Hassan R, Arshad H, Rahmat M, Zainal NFA, et al. Automated Analysis of Exam Questions According to Bloom's Taxonomy. Procedia - Social and Behavioral Sciences. 2012;59:297–303. Accessed December 1, 2020.
3. Ermie E. Psychometrics 101: Know What Your Assessment Data Is Telling You. Examsoft. 2015. Accessed November 18, 2020.