Creating Valid Multiple-Choice Exams

by Scott Ross, PharmD, PGY1 Pharmacy Practice Resident, Mississippi State Department of Health

With increased class sizes and teacher load, multiple-choice exams have become the primary method for evaluating health professions students. There are many pros and cons to using multiple-choice tests. This article aims to investigate the cons and improper techniques and offer potential solutions to improve the quality of questions and enhance student learning.

Multiple-choice exams remain a popular form of assessment because they have several advantages, including ease of grading, standardization, the objectiveness of scoring, and the ability to test many discrete concepts. And teachers can administer multiple versions of the same test.^1,2 Furthermore, evidence suggests that a well-constructed multiple-choice exam is just as effective as a short-answer test in terms of promoting the retention of material.^3,4 However, a poorly constructed multiple-choice exam will not accurately measure learning and can lead to frustration. There are several best practices that many teachers fail to follow, including using "all of the above" or "none of the above" answer choices, writing "throw-away" answers as potential choices, asking students questions that focus on their ability to memorize and recall trivial details, and authoring stems that are unclear/vague.^1,5

While seen commonly on multiple-choice exams, the "none of the above" answer choice leaves students wondering what the correct answer is. Indeed, it could be argued that the best possible answer is not among the answer choices, and thus "none of the above" would always be the best option. In many cases, students view "none of the above" as a throw-away answer that can be ignored. Similar but different issues arise with the "all of the above" answer choice. Using the "all of the above" answer choice may have the benefit of determining if the student is aware that more than one choice is correct, but this quickly results in guessing and relying on partial knowledge of the material to answer correctly.⁶ When the "all of the above" choice frequently appears on an exam, students will likely pick up on trends and will lean towards selecting this answer even when they lack an understanding of the material. Thus, "none of the above" and "all of the above" answer choices should be avoided.⁶ Instead, consider using "select all that apply" questions because they thwart guessing — but admittedly, they are more difficult. To discourage guessing, some instructors award points for each correct response but take off points if a student selects an incorrect answer or does not select a correct response.¹ While this is undoubtedly more challenging, it is more efficient and less cognitively demanding than asking students to respond to series of open-ended questions.

Another common issue when constructing multiple-choice tests is including "throw-away" answers — answers that are so obviously incorrect that even those who do not know the subject matter can quickly eliminate them. Including these answer choices is harmful because it increases the odds of guessing correctly. It is a best practice to include at least 3 but no more than 5 plausible answer choices.⁷  The key word here is plausible – at least they should seem reasonable to the learner who is not sufficiently knowledgeable about the subject matter.

Some critics of multiple-choice testing state that exam scores using this format do not always correlate to the learner's understanding of the material — the method simply asks students to memorize and recall information.⁵ This is important to keep in mind when forming questions as many instructors rely too heavily on "recall" or knowledge-based type questions. While they are more challenging to write, it is possible to create questions that require critical thinking. Forming thoughtful questions that require students to analyze, apply, and evaluate is vital to ensuring they develop the skills needed in their future careers.

Another common problem is forming misleading or vague question stems or answer choices that lead to confusion or misunderstandings. It is also best to avoid negative phrasing (e.g., "which of the following is not true …") in exams since this can cause students to misread the question. If a question truly cannot be phrased positively, it is best to make the negative wording stand out by using italics, capitalization, or bolding of the word(s). Having clear answer choices is just as important as forming clear question stems. An excellent way to ensure that questions and answers are worded clearly and concisely is to send the material to someone else to review. It is crucial to keep in mind how the answer choices relate to each other. Answer choices should be homogenous in the sense that they relate to the same content and have a similar sentence structure and length. This is to prevent giving clues to students as to what the correct answer is. Another strategy to prevent clues is to always present the choices in numerical or alphabetical order.

Perhaps the biggest concern with multiple-choice tests is the format itself. Most choices will not be provided to the health professional.  Rather they must recall and weigh the potential options themselves.  Thus, multiple-choice exams are not authentic assessments — they do not reflect real life. Real-life decision-making comes from generating choices for ourselves and formulating our own answers by considering multiple pieces of information and then making a judgment. Thus, relying solely on multiple-choice assessments to determine a student's progress does not accurately reflect whether a student is competent.  Other forms of assessment, including objective structured clinical exams (OSCE), evaluations of authentic work products, and observations during field-based activities, must also be used.

Creating valid multiple-choice exams is a vital skill that all teachers should master to ensure their students have mastered the material. However, there are several common problems that should be avoided, and multiple-choice assessments have several limitations. Using a combination of assessment strategies is essential to get a comprehensive view of each student's knowledge, skill, and abilities.

 

References

1. Weimer M. Multiple-choice tests: Revisiting the pros and cons [Internet]. Faculty Focus. 2019 [cited 2021Sep19].
2. Medawela RMS, Ratanayake DRDL, Abesinghe W, et al. Effectiveness of "fill in the blanks" over multiple choice questions in assessing final year dental undergraduates. Educación Médica 2017, 19 (2): 72-76.
3. Khan JS, Mukhtar O, Tabasum S, et al. Relationship of Awards in multiple choice questions and structured answer questions in the undergraduate years and their effectiveness in evaluation. Journal of Ayub Medical College 2010; 22 (2): 191-195.
4. Haynie W. Effects of Multiple-Choice and Short-Answer Tests on Delayed Retention Learning [Internet]. Journal of Technology Education 1994; 6 (1): 32-44.
5. Fors K. Opinion: Multiple choice tests don't prepare students [Internet]. The Utah Statesman. 2020 [cited 2021Sep19].
6. Butler A. Multiple-choice testing: Are the best practices for assessment also good for learning? [Internet]. The Learning Scientists. 2017 [cited 2021Sep19].
7. Butler AC. Multiple-choice testing in education: Are the best practices for assessment also good for learning?. Journal of Applied Research in Memory and Cognition 2018; 7 (3): 323-331.