Using the ‘Answer Until Correct’ Assessment Method to Enhance Learning

by Zemen Habtemariam, Doctor of Pharmacy Candidate, University of Maryland School of Pharmacy

Ultimately the goal of teaching is for students to learn.  Not simply for purposes of doing well on a test or getting a good grade, but to encourage students to engage in learning, to see more than one interpretation of the answer, and to improve retention. However, in my experience, how we are assessed in the classroom is not always reflective of what we are supposed to learn or what we actually have learned. Therefore, using a method of assessment that effectively reflects the desired goal – that is learning — is attractive to me.

Let’s look at an example. In my pharmacy school, a common method of assessment is the multiple choice examination (MCE).  Indeed, it's the most widely used form of assessment at many schools.  During a MCE, students are presented with a series of questions and they are asked to pick the one best answer. While these tests have been used for a very long time and are “objective” measures of knowledge, they have their flaws – and may actually discourage learning. For starters, multiple choice examinations foster the short memorization of facts and not the long term retention of concepts. They often foster “binge and purge” behaviors. In addition, MCEs can provoke anxiety because the learner is often forced to deliberate between two arguably good choices. I know these flaws all too well.

However, MCEs are not the only testing method available. Another testing method that might effectively addresses the “flaws” of the MCE is the Answer-Until-Correct (AUC) method. Under the AUC procedure, examinees can select an option for the multiple-choice question.  If the selected answer is not the best choice, the students is permitted to select another answer.  And they can continue to select other answer choices until the best answer is revealed. This type of testing often awards more points for selecting the best response first and awards fewer points as the students additional choices. This method of testing can enhance learning principally by providing formative feedback. Formative assessment is an approach whereby students receive immediate feedback on their work. This can be a critical element for students to achieve deeper learning.¹ Epstein and colleagues² developed a tool using the AUC method on three multiple-choice examinations in an introductory psychology course. Not only did the study find that students taking these types of exams performed better on the course’s final examination, but students would have received higher scores (when compared to students who had previously taken the course) even if a traditional MCE approach had been used. In this manner, the AUC approach actually helped students to retain their knowledge.

In addition to improving student learning, the AUC method may also reduce test anxiety – something many students experience. One study examined the effect of anxiety on student performance during a series of tests of varying levels of difficulty.³ The study found that although highly anxious students tended to do poorly on examinations, students who were prone to anxiety who received positive feedback from their examinations tended to have big gains in their performance, especially on exams of low difficulty (“easy”). The authors highlighted how items or questions of optimal difficulty on an examination often induced anxiety.  But that’s a Catch-22.  Students can only perform at their optimum when their anxiety is reasonably low.  The take-home message from this study is that students can show improvement in their test performance when anxiety is not highly apparent. Therefore, a testing approach that reduces the risk of selecting a wrong answer and utilizes immediate feedback can be expected to reduce anxiety.  This, in turn, allows learns to perform at their best.

Now that we have covered how the AUC method works and its potential benefits, one may be curious as to when and where it can be applied. Health professionals must be able to perform well on a test, but also be able to apply that knowledge to patient care or in the work environment. A good application of the AUC method would be a pharmacy law course. Pharmacy law, not unlike many topics in a pharmacy school curriculum, is important because every pharmacist must understand and adhere to the laws and regulations that govern pharmacy practice and the sale of medications. Traditional testing methods often require long periods of time to score or grade them.  Thus, students won’t know what answers they got wrong until they get their exam back some days or even weeks later.¹ Thus, students are forced to progress to the next topic in the course without having mastered the previous material and don’t have the opportunity to learn from their mistakes. Thus, educators can apply the AUC method by having students take law exams and answer each question until they reach the correct response. This not only teaches them what the correct answers are, but perhaps more importantly, students begin to learn why the other answers are not correct.

Students and educators will benefit from using the AUC method. This assessment technique has been shown to improve academic performance, alleviate anxiety that can hinder students from learning and test-taking, and addresses some of the limitations inherent in more traditional multiple choice exam formats.

References

1. Persky A, Pollack G. Using Answer-Until-Correct examinations to provide immediate feedback to students in a pharmacokinetics course. Am J Pharm Educ. 2008; 72(4): Article 83.

2. Epstein M, Epstein B, Brosvic G. Immediate feedback during academic testing. Psychological Reports. 2001; 88: 889-94.

3. Rocklin T, Thompson J. Interactive effects of test anxiety, test difficulty, and feedback. Journal Of Educational Psychology. 1985; 77(3): 368-72.  [accessed April 21, 2015]

Saying What the Learner May Not Want to Hear – Providing Feedback

by Regina Ulis, PharmD, PGY1 Pharmacy Practice Resident, Medstar Georgetown University Hospital

Every preceptor will inevitably encounter a learner who must be given “corrective” feedback regarding a less than optimal performance. While it is easy to give praise and reward learners for good behavior, it can be quite difficult to formulate criticism and deliver it in an effective manner. Such criticism may be necessary to help the learner grow professionally or perhaps personally, to protect patients’ health, or for a variety of other reasons.¹ However, it is important to understand that the recipient may see this feedback as a personal attack or may shy away from the learning experience instead of taking the advice in stride and taking action to make a behavior change. So how, then, do preceptors deliver feedback in a positive manner that promotes the learner’s growth?

Several methods have been proposed for giving effective feedback to learners. One of the most frequently used methods is the “Feedback Sandwich”.² A Feedback Sandwich is essentially a negative critique “sandwiched” between positive statements. For example, if a learner needs to be more assertive, the preceptor might say, “You made several excellent recommendations to change the patient’s medication regimen. However, I don’t think you communicated them to the team very effectively.  You seemed unsure of yourself and people could barely hear you. I think that one of your goals for this rotation should be to become more confident in making recommendations to the team. That way the team can take advantage of your clinical knowledge and this will lead to improved patient care.”

The entire purpose of this method is to make the delivery of negative feedback more palatable. This method can be effective because it builds trust, improves comfort, and increases the receptiveness of the listener to the criticisms that are delivered. It has also been documented to increase motivation and engagement²; these qualities are necessary to maximize learning. There are also variations to this technique, such as the “open-face” sandwich, which pairs a negative statement followed by a positive one. Different situations may call for different varieties of “sandwiches,” and it may also become necessary to vary the technique because the learner may begin to anticipate that good feedback is always followed by a negative comment.²

Studies have been conducted to see the effects of the Feedback Sandwich on learning including studies with medical students.²  These results should be applicable to student pharmacists and other health professionals as well. Overall, these studies have shown that this method enhances the learner’s self-esteem. Additionally, using this method allows for more individualized comments, which increases performance even though it may not increase the student's short-term satisfaction. Other results indicated that the number of positive comments provided to a learner predict successful future performance.² This may be a reason why it may be helpful to give positive feedback alongside the negative.

Another common method of giving feedback is the “reflective feedback” technique which focuses on content and its relationship to the receiver. In other words, this message focuses on the message to be delivered and how important it is to the person receiving it. This method consists of a series of 3 steps:³

1.  Ask questions that are clarifying and connected to what was observed
2. Explain the value or potential value to the learner who is receiving feedback
3. Ask a reflective question or put forth a potential action to stimulate thought

The goal of these steps is to make underlying assumptions explicit (step 1), make the recipient feel valued (step 2), and request a response from the recipient and allow them to see others’ point of view (step 3).  One of the positive attributes of reflective feedback technique is that it is intended to come off as less judgmental and more focused on growth and development. Following these three steps opens a conversation between the teacher and learner and allows for a discussion, including justifications and possible observations that each party has made.³

Although these are but two methods that may be used to deliver feedback to learners, both are documented to be effective. They allow for growth and development – often of both parties.^1,3 In health professional education, using these techniques may also lead to improvements in patient care. It is important to note, that many other strategies for delivering feedback exist, and that every situation is unique.  It is important to have a plan prepared for when you have to say what your learner may not want to hear.

References

1.   Hoon A, Olver E et al. Use of the “Stop, Start, Continue” method is associated with the production of constructive qualitative feedback by students in higher education. Assess Eval Higher Educ 2015; 40(5): 755-767.

2.   Parkes J, Abercrombie S, McCarty T. Feedback sandwiches affect perceptions but not performance. Adv Health Sci Educ Theory Pract 2013; 18(3):397-407.

3.   Reilly M. Saying what you mean without being mean. Educational Leadership 2015; 73(4): 36-40.

Teaching Team Work and Cooperation: Critical Skills in Healthcare Today

by Matthew Levit, Doctor of Pharmacy Candidate, Class of 2018, University of Maryland School of Pharmacy

Traditional instructional models stress the importance of individualism, but many educators and schools have begun to emphasize cooperative learning methods.¹ Cooperative learning is an integrative learning model that seeks to educate students through activities that promote social learning and leadership experiences.^2,3 Cooperative learning has its origins in social interdependence theory. According to this theory, the learning outcomes of an individual are determined by their own actions as well as the actions of others.³ Cooperative learning builds on this theory by postulating that learning occurs best through social engagement.⁴

Picture from collegestar.org

Cooperative learning is more than just working together in a group. Students learn by using their social and interpersonal skills to achieve an academic goal.¹ Cooperative learning is commonly used in a variety of practice-based settings including service learning, integrative internships, and continuing education (CE) seminars.⁵ In some of these settings, students receive academic credit and solve real-world problems.⁵ For example, students rotating on an interprofessional team may encounter a patient that suffers from multiple comorbidities (e.g. diabetes, hypertension, and neuropathic pain). One student in the group may be adept at diabetes management.  Another may have experience managing patients with high blood pressure.  And another may be very knowledgeable about neuropathic pain. Collectively, the students learn from each other and are more likely to successfully complete the academic exercise.

Picture from viewinghistorythroughthedigitallens.com

There are five key elements to the cooperative learning model. These include positive interdependence, individual and group accountability, face-to-face promotive interaction, interpersonal and small group skills, and group processing.^1,6 Positive interdependence requires that every member of the group value each other’s contributions.¹ Interprofessional healthcare teams must have positive interdependence in order to fully utilize each member’s unique contributions to the patient’s care. Individual and group accountability requires that each member of the group is accountable for one’s actions and all must contribute to the completion of the assigned task.⁶ An evaluator should discipline students who do not contribute because it is detrimental to the group’s overall success. Interpersonal and small group skills include effective leadership and communication, building trust, making clear decisions, and managing potential conflicts.¹ Face to face promotive interactions require learners to share resources as well as support and encourage their teammates success.⁶ Group processing requires communication among group members. Members should be encouraged to express any concerns they have with each other for the overall benefit of the group.⁶

One example of an instructional method that uses the five key elements of the cooperative learning model is the jigsaw technique.^3,5 In this method, students are assigned a topic and placed into two groups: a home group and an expert group. Students in the home group go to the expert group to learn a particular topic and then go to another group to teach that particular subject.³ This technique promotes contributions from each member of the group (positive interdependence and face to face interaction) as well as accountability for each member’s actions (individual and group accountability). Educators that use this model expect their students to support each other’s learning as well as understand that each member of the group must be able to teach a piece of the subject matter to others. In addition, students are expected to communicate effectively with each other using verbal and non-verbal communication (interpersonal and small group talks and group processing).^3,5  This certain isn’t the only example of cooperative learning.  Several other cooperative learning methods exist [See previous blog essay regarding the Processed Oriented Guided Inquiry Learning (POGIL) method].

Picture from flipclass.com

In healthcare today, interprofessional healthcare teams are practicing cooperative learning.⁷ In the past, different specialties would work independently and had little appreciation or understanding regarding the unique contribution that other members of the health care team could make.  If students start collaborating in the classroom with their peers, then perhaps this will translate to working with other healthcare professionals.⁷ Health professions educators have an important role in helping students become effective members of a team. In order to implement cooperative learning, educators must promote instructional methods that use the 5 key elements of the cooperative learning model. For example, in the classroom setting, an educator can design case-based group exercises (interpersonal and small group skills and group processing). Students should be encouraged to work in small groups so that everyone has a chance to participate (face-to-face promotive interaction) and learn from each other. During these clinical exercises, the instructor can require that each student to write a reflection on how they contributed to the group’s work as well as evaluate peers (positive interdependence and individual and group accountability). This reflection will allow the educator to see how well collaborative learning is working and student comments can be used to make changes. In practice-based settings, preceptors and other healthcare instructors can encourage interprofessional healthcare teams to use this model of learning when “rounding” in the hospital or during interprofessional meetings in clinic.^4,7 Evaluations of their experiences can be used to individualize students’ needs and provide an overall framework for future collaborative work. The ultimate goal every educator should stress when using cooperative learning strategies in healthcare is to promote optimal patient care.

Picture from dental.nyu.edu

Cooperative learning is a model for developing and implementing instructional activities that helps students to develop social learning and leadership skills. ^2,3 Educators and students in the health professions must become proficiency in these skills in order to effectively manage complex patients through team-based collaborations.^4,7

References

1. Salam T, Greenberg H, Pitzel M, Cripps D. Interprofessional education internships in schools: Jump starting change. Journal of Interprofessional Care. 2010; 24(3): 251 – 263.
2. Stavenga de Jong JA, Wierstra RF, Hermanussen J. An exploration of the relationship between academic and experiential learning approaches in vocational education. British Journal of Educational Psychology 2006; 76(1): 155-169.
3. Johnson DW, Johnson RT. An Educational Psychology Success Story: Social Interdependence Theory and Cooperative Learning. Education Educational Res 2009; 38(5): 365 – 379.
4. Budgen C, Gamroth L. An overview of practice education models. Nurse Education Today 2008; 28(1): 273 – 283.
5. Schul JE. Revisiting an Old Friend: The Practice and Promise of Cooperative Learning for the Twenty-First Century. Soc Studies 2011; 102(1): 88 – 93
6. Basak T, Yildez D. Comparison of the effects of cooperative learning and traditional learning methods on the improvement of drug-dose calculation skills of nursing students undergoing internships. Education Educational Res 2014; 73(3): 341 – 350.
7. Mitchell P, Wynia M, Golden R, et al. Core principles and values of effective team based health care. 2012; Discussion Paper, Institute of Medicine, Washington, DC.