Reuse, Reduce, Recycle...Test Questions?

by Hana Kim, Pharm.D., PGY-1 Resident, Kaiser Permanente of the Mid-Atlantic States

As an instructor, it is difficult to create quality test questions, let alone make new ones every year.  The question as to whether to recycle questions is an issue any organization administering tests or examinations must address. When a question is reused, it can increase the possibility that it will become more public to test takers and can give an advantage to those who have access to the question.  There is also concerned about repeat examinees, those who fail an examination and are required to retake it.  Does reusing test questions appropriately challenge students to learn the material?  Does it provide an unfair advantage to a select few students who have access to the recycled questions?  And, what can instructors do to alleviate these concerns?

Timothy J. Wood studied the role of reused test questions on repeat examinees who took the Medical Council of Canada (MCC) Evaluation Examination (MCCEE).¹ The MCCEE is a basic medical knowledge test for International Medical Graduates that contains 324 multiple choice questions. The MCCEE is offered 3 times a year, 4 months apart. Wood investigated whether prior exposure to test questions enhanced performance among 130 repeat examinees. The examinees were presented 36 repeat questions from the previous examination.  The examinees had no knowledge that these questions would be reused.  The investigators found that the examinees scores on the reused and non-reused questions increased equally, a finding that was consistently with previous studies. He noted that this might be due to increased knowledge of the subject matter, hopefully due to an increased amount of studying in preparation for the re-take exam. Wood concluded that prior exposure to test questions had little impact on the performance. Although there was no difference in examinee performance on repeat questions, the exam was given 4 months apart and the examinees did not have access to the questions in between test administration, so the results may not be representative of what we’d see in most academic settings.

Similarly, Wagner-Menghin and colleagues conducted a study to evaluate the effect of reusing written test questions. The authors specifically utilized the Rasch model, which is a probabilistic psychometric framework measurement model that estimates item difficulty and ability measures. The study noted four conceptual factors that should be taken into consideration:

• Reuse expectation: passing items along to new test takers can be beneficial, especially when reusing items is expected
• Cheating attitude: many studies on cheating have not focused on cheating with the reuse of test questions
• Exam’s consequences: there are consequences if a student fails an examination; therefore, the pressure to pass may promote cheating
• Item content: reusing questions that require student application of knowledge, not simply recall, may diminish the test validity²

This study was designed to quantify the reuse of test questions based on an item’s level of difficulty. Specifically, the authors introduced a new written multiple-choice course exam to assess clinical skills in 671 medical students. To assist the students, a “representative” set of multiple-choice questions was included in the official study materials. Looking at item content, there was a larger effect on those that tested application of knowledge versus recall questions.  There was a 50% increase in student scores on application test questions that were reused compared to 20% increase when recall-type test questions were re-used.  Although the re-used material did not result in increased overall scores, the authors postulate that the lack of benefit may be due to a ‘deficit in study organization and time management for late test takers.’  A test with 30-45% reused questions (particularly with a large number of recall questions) is, therefore, unlikely to substantially benefit students.²

When questions are pass down from one student to another, it creates a potential unfair advantage as some students have access to the items while others do not.  What can professors do to mitigate the problem of questions being passed down? One possible solution is to make all previous examinations available to students so there is fair opportunity for all students. This is exactly what a law professor at George Mason University has been practicing in his classes. He develops new questions for every exam and makes past exams and answers available to students.³  This strategy certainly improves fairness but increases faculty workload the workload as new questions need to be formulated every year.

While some studies demonstrate a statistically significant difference in scores when questions are reused, one of the biggest concerns is that questions will get passed down from year to year, increasing the possibility of “cheating.”  These situations are inevitable but instructors should consider safeguards to help mitigate this problem. Some options include creating new test questions ever year or allowing a sufficient amount of time (2-3 years) between question re-use. Although there are several suggested solutions, the question as to whether test items should be reused and recycled remains an unanswered one.

References:

1. Wood TJ. The effect of reused questions on repeat examinees. Adv Health Sci Theory Pract. 2009; 14(4): 465-73.

2. Wagner-Menghin M, Preusche I, Schmidts M. The Effects of Reusing Written Test Items: A Study Using the Rasch Model. ISRN Education 2013; Article ID 585420.  Accessed 17 November 2013.  
3. Somin I. The Perils of Reusing Questions from Past Exams. The Volokh Conspiracy. http://www.volokh.com/2011/01/18/the-perils-of-reusing-questions-from-past-exams/. Accessed 27 October 2013.

Cognitive-Moral Development in Pharmacy School

by Stephanie M. Callinan, Pharm.D., PGY1 Pharmacy Practice Resident, VA Maryland Health Care System

"I promise to devote myself to a lifetime of service to others through the profession of pharmacy. In fulfilling this vow:

• I will consider the welfare of humanity and relief of suffering my primary concerns.
• I will apply my knowledge, experience, and skills to the best of my ability to assure optimal outcomes for my patients.
• I will respect and protect all personal and health information entrusted to me.
• I will accept the lifelong obligation to improve my professional knowledge and competence.
• I will hold myself and my colleagues to the highest principles of our profession’s moral, ethical and legal conduct.
• I will embrace and advocate changes that improve patient care.
• I will utilize my knowledge, skills, experiences, and values to prepare the next generation of pharmacists.

I take these vows voluntarily with the full realization of the responsibility with which I am entrusted by the public.”

-       American Pharmacists Association, “Oath of a Pharmacist”

During the commencement ceremony, the pharmacy school class in all their pride and joy recites the “Oath of a Pharmacist” as an affirmation of their dedication to serving others through the profession of pharmacy.  One of the key precepts of this oath is to uphold the highest “moral, ethical, and legal conduct.” Although appropriate ethical and legal principles can be taught in a classroom setting, what can we do as educators to “teach” moral behaviors? How can we insure that every student in that graduating class meets this requirement?

Cognitive-moral development is the process of developing one’s self-awareness regarding ethical and professional behaviors. The process of cognitive-moral development was first described by the great constructivist, Jean Piaget, who observed that children develop and enhance their knowledge through their personal and environmental interactions.¹  The moral development process was further described by American psychologist Lawrence Kohlberg as six stages grouped into three levels: pre-conventional, conventional, and post-conventional morality.

Kohlberg’s Stages of Moral Development2
Level	Explanation	Stage
1. Pre-Conventional (Individual)	Person sees only right or wrong and is driven to complete tasks based upon rewards or avoiding punishments.	1. Obedience and punishment 2. Instrumental purpose and exchange
2. Conventional (Group)	Persons conform to social expectations/roles and respect authority as part of a team.	3. Interpersonal relationships 4. Maintenance of Social Order
3. Post-Conventional (Professional/ Institutional)	Persons adhere to institutional rules, however recognize that personal morals may conflict with these regulations.	5. Social contract and individual rights 6. Universal ethical principles

Kohlberg’s stages of moral development can be seen in the educational setting through these examples:

• Individual: Bob always interrupts class to inject his own opinions. He is frequently seen arguing grades with the professor stating, “You didn’t tell us to study this for the exam!” He often competes with classmates to enhance his self-image.
• Group: Susie is a listener; she respects the leader(s) in her group project and reflects on their suggestions to create her own recommendation(s). She actively implements changes based on feedback from her teachers.
• Professional: Joe is a “B” student, involved in several committees, and volunteers his time in the community. He is able to prioritize his responsibilities, while also maintaining his grades.  Joe asks for feedback as he recognizes that he is not “perfect” and sets goals to improve.

As healthcare shifts to a team-based, patient-centered approach, techniques to mold “individual” students into “professional” students are crucial within pharmacy school curricula. A 1999 study that compared the moral reasoning skills of pharmacy students, pharmacy practitioners, and other professions (i.e. medicine, nursing, and law) found that pharmacy students and practitioners had a lower level of moral reasoning when compared to staff nurses, medical students, and law students.⁴ Although this data is somewhat dated, this is an alarming finding.  Are we, as educators, doing enough in pharmacy schools to promote high-level, post-conventional cognitive-moral reasoning?

A recent review suggests that cognitive-moral development in the classroom can be facilitated by problem-based learning, team (or group)-based learning, in-class discussion, and discussion of topics regarding moral or ethical decisions.⁵  By using these strategies, the educator adopts the role of a facilitator, instead of a lecturer or “expert.” Students are required to collaborate with peers, defend their recommendation(s) based on their own knowledge, and “experience” difficult ethical scenarios that occur in practice, but are rarely discussed within the pages of a book.  By having students develop and defend their decisions, they can improve individual-level moral reasoning skills by recognizing the “rules,” (group level) and begin to identify and plan for the exceptions to the rules (professional level).

Although cognitive-moral reasoning can be cultivated in the classroom, there is no substitute for practice and experience. As described by Piaget, moral reasoning is chiefly promoted through interactions with other people and the environment.  As pharmacists in a teaching environment, we must act as role models for pharmacy students — leading by example and holding students accountable, not only for patient care duties but other responsibilities too. Although literature in this area is sparse, some studies suggest the teacher-prompted personal reflection exercises can encourage moral growth.^6,7 In my own experience, personal reflection encouraged me to ask for feedback on characteristics/skills which I would like to improve.

We can unknowingly impede our student’s cognitive-moral development by modeling “individual” level behaviors such as self-centeredness and black and white thinking (i.e. refusal to accept the validity of multiple answers). The ideal pharmacy preceptor should reflect on his/her own personal behaviors in order to model high moral character and assess their students’ level of moral reasoning.  Moreover, the ideal preceptor should implement practices that help student become true professionals, aware of the “rules” that should govern our conduct but also able to identify and sort through conflicts between the “rules” and what’s “the right thing to do” for patients and the society in which we live.

Think back on the students you have educated and precepted. What have you done to develop their moral reasoning? Have you inadvertently, at times, sent the wrong messages? What can you do to prevent this in the future? Addressing these questions is essential because you have an important role to play in your students’ cognitive moral development.  When you look at that graduating class each year, do you think we have prepared students to uphold the highest standards of “moral, ethical, and legal conduct?”  I’ll leave that question for you to answer.

References

1. Piaget, J. The moral judgment of the child. New York: The Free Press; 1965.

2. Moral development and moral education. In: Kohlberg, L. & Turiel, editors. Psychology and education practice. Upper Saddle River, NJ: E. Scott Foresman; 1971.

3. Duncan WC. Strategies for incorporating cognitive & moral development. PowerPoint presented at: 2013 AACP Teacher’s Seminar: Strategies for Effective Teaching and Learning. American Association of Colleges of Pharmacy Annual Meeting; 2013 July 13; Chicago, IL.

4. Latif DA and Berger BA. Cognitive moral development and clinical performance: implications for pharmacy education. Am J Pharm Educ. 1999;63(1):20-27.

5. Schuitema J, Ten Dam G, Veugelers W. Teaching strategies for moral education: a review. J Curric Stud 2007;40(1):69-89.

6. Vertress SM, Shuman AF, Fins JJ. Learning by doing: effectively incorporating ethics education into residency training. J Gen Intern Med 2013;28(4):578-82.
7. Daboval T, Moore GP, Ferretti E. How we teaching ethics and communication during a Canadian neonatal perinatal medicine residency: an interactive experience. Med Teach 2013;35(3):194-200.