Reuse, Reduce, Recycle...Test Questions?

by Hana Kim, Pharm.D., PGY-1 Resident, Kaiser Permanente of the Mid-Atlantic States

As an instructor, it is difficult to create quality test questions, let alone make new ones every year.  The question as to whether to recycle questions is an issue any organization administering tests or examinations must address. When a question is reused, it can increase the possibility that it will become more public to test takers and can give an advantage to those who have access to the question.  There is also concerned about repeat examinees, those who fail an examination and are required to retake it.  Does reusing test questions appropriately challenge students to learn the material?  Does it provide an unfair advantage to a select few students who have access to the recycled questions?  And, what can instructors do to alleviate these concerns?

Timothy J. Wood studied the role of reused test questions on repeat examinees who took the Medical Council of Canada (MCC) Evaluation Examination (MCCEE).¹ The MCCEE is a basic medical knowledge test for International Medical Graduates that contains 324 multiple choice questions. The MCCEE is offered 3 times a year, 4 months apart. Wood investigated whether prior exposure to test questions enhanced performance among 130 repeat examinees. The examinees were presented 36 repeat questions from the previous examination.  The examinees had no knowledge that these questions would be reused.  The investigators found that the examinees scores on the reused and non-reused questions increased equally, a finding that was consistently with previous studies. He noted that this might be due to increased knowledge of the subject matter, hopefully due to an increased amount of studying in preparation for the re-take exam. Wood concluded that prior exposure to test questions had little impact on the performance. Although there was no difference in examinee performance on repeat questions, the exam was given 4 months apart and the examinees did not have access to the questions in between test administration, so the results may not be representative of what we’d see in most academic settings.

Similarly, Wagner-Menghin and colleagues conducted a study to evaluate the effect of reusing written test questions. The authors specifically utilized the Rasch model, which is a probabilistic psychometric framework measurement model that estimates item difficulty and ability measures. The study noted four conceptual factors that should be taken into consideration:

• Reuse expectation: passing items along to new test takers can be beneficial, especially when reusing items is expected
• Cheating attitude: many studies on cheating have not focused on cheating with the reuse of test questions
• Exam’s consequences: there are consequences if a student fails an examination; therefore, the pressure to pass may promote cheating
• Item content: reusing questions that require student application of knowledge, not simply recall, may diminish the test validity²

This study was designed to quantify the reuse of test questions based on an item’s level of difficulty. Specifically, the authors introduced a new written multiple-choice course exam to assess clinical skills in 671 medical students. To assist the students, a “representative” set of multiple-choice questions was included in the official study materials. Looking at item content, there was a larger effect on those that tested application of knowledge versus recall questions.  There was a 50% increase in student scores on application test questions that were reused compared to 20% increase when recall-type test questions were re-used.  Although the re-used material did not result in increased overall scores, the authors postulate that the lack of benefit may be due to a ‘deficit in study organization and time management for late test takers.’  A test with 30-45% reused questions (particularly with a large number of recall questions) is, therefore, unlikely to substantially benefit students.²

When questions are pass down from one student to another, it creates a potential unfair advantage as some students have access to the items while others do not.  What can professors do to mitigate the problem of questions being passed down? One possible solution is to make all previous examinations available to students so there is fair opportunity for all students. This is exactly what a law professor at George Mason University has been practicing in his classes. He develops new questions for every exam and makes past exams and answers available to students.³  This strategy certainly improves fairness but increases faculty workload the workload as new questions need to be formulated every year.

While some studies demonstrate a statistically significant difference in scores when questions are reused, one of the biggest concerns is that questions will get passed down from year to year, increasing the possibility of “cheating.”  These situations are inevitable but instructors should consider safeguards to help mitigate this problem. Some options include creating new test questions ever year or allowing a sufficient amount of time (2-3 years) between question re-use. Although there are several suggested solutions, the question as to whether test items should be reused and recycled remains an unanswered one.

References:

1. Wood TJ. The effect of reused questions on repeat examinees. Adv Health Sci Theory Pract. 2009; 14(4): 465-73.

2. Wagner-Menghin M, Preusche I, Schmidts M. The Effects of Reusing Written Test Items: A Study Using the Rasch Model. ISRN Education 2013; Article ID 585420.  Accessed 17 November 2013.  
3. Somin I. The Perils of Reusing Questions from Past Exams. The Volokh Conspiracy. http://www.volokh.com/2011/01/18/the-perils-of-reusing-questions-from-past-exams/. Accessed 27 October 2013.