Less is More: Learning in Small Groups

by: Yevgeniya Kogan, Pharm.D., PGY-2 Health-System and Administration Resident, University of Maryland Medical Center

College. Freshman year.  Day one.  First class. If those factors weren’t intimidating enough, imagine hurrying through a huge college campus, and locating the tucked away lecture hall.  When you enter it, before you unfold a sea of stadium-like seats occupied by about 300 hundred unfamiliar faces. You make your way all the way up to the top of the hall looking for an empty seat. Your professor is about the size of a fly from your birds-eye perspective and you are grateful for his annoyingly loud voice and 10 foot tall power point presentation.  Now, fast-forward almost 10 years and, in stark contrast, you find yourself in front of a computer (maybe in your pajamas), on your own, with faceless participants, and an unfamiliar voice permeating from the telephone leading a discussion. You are in your very first online class. Questions come to mind; which is better? What environment provides the most opportunity for growth?  Which set up will I benefit from the most? The answer might seem obvious. After all, who doesn’t love the comfort of their own pajamas over a humongous lecture hall crammed next to students, some of whom have obviously given up showering in favor of long nights of partying. Are these two extremes the only way?

Hamann¹ and colleagues took on the challenge of assessing just that question in a study where they compared discussions conducted in large face-to-face classes, online classes, and (the alternative) small face-to-face classes. Based on a survey of students who were exposed to all three environments, they found (no surprise) large class discussions were rated least favorable by the majority of students. The interesting findings emerged when they compared online and small-group discussions. While online- classes tend to yield the most satisfaction when it comes to participation and the ability to express one’s thoughts, the small-group face-to-face discussions out-perform the online environment in terms of getting to know your classmates, stimulating interest, and overall satisfaction.

The authors explore participation, which is known to enhance learning and stimulate creativity, even further by looking at gender differences. It is common knowledge that male participants tend to dominate discussions. When Hamann¹ and colleagues explored this phenomenon they saw the following. As expected, in the face-to-face classes, both large and small, males tended to participate more frequently than their female counterparts. However, in the online environment, this difference is virtually eliminated and equal participation emerged. The increased participation from the female students might be attributable to a less intimidating environment.  There is also evidence to support that smaller groups tend to level the playing field for students of different ethnic backgrounds.²

During my undergraduate years I was in many large-classrooms in a science-focused school for most of my science and math courses.  Indeed, most colleges and universities teach the  introductory science, technology, engineering and math (STEM) courses in large, stadium-like lecture halls. According to Jason Koebler³, the large classroom is a way higher-education uses a “weed-them-out” process that teaches students to sink or swim.  Luckily, I swam. During graduate school I was in both the traditional large classroom and the smaller group setting where discussions were mandated and you were held accountable for the taught content. As I reflect back, one thing stands out the most are the many faces engrossed in their laptops – such a studious bunch!   But on closer inspection, the sea of brightly lit screens were displaying social media, news, movies, and online chats.  Very few screens projected the course material.  Perhaps the large-classroom wasn’t as studious as I thought.  In contrast, my experience in small groups, while sometimes forced, yielded greater interaction, discussion, and exchange of ideas.  True, some small groups were more effective in generating discussions than others.  In larger small-groups (more than 15 people), more people took the back seat while in smaller small-groups (8 students or less) everyone was engaged. When does a small group become too large?

Class size has been a hot topic for many years.  The US Federal government allocated $12 billion (over a seven-year period) to reduce class size and states like California spent another $3.6 billion.⁴  There are some conflicting results when it comes to class size.  Maasoumi⁴ and colleagues’ analysis indicates that a reduction in class size from 20 or more students down to less than 20 students generally increases test scores for those students who initially scored below the median test score but decreases test scores for those who scored above the median.  Conversely, Konstantopoulos⁵  found that all students reap the benefit from being in small classes when they examined Stanford Achievement Test scores in  mathematics, reading, and science.  The greatest benefit was seen in students who were low achievers and those who spent the longest duration of time in a small class setting. Nye⁶ reported that when students transitioned from small classes to large ones, the academic benefits persists for two full years.

As the pressures on academia increases to produce competent and confident students, it is important to consider the benefits of small groups and the impact they have on student satisfaction, understanding of concepts, and ability to work together.  Learning in small, face-to-face groups seems to have many advantages over large, content driven, sink or swim classrooms and independent, self-directed, lonely cyberspace classes.

Reference:

1. Hamann K, Pollock P, Wilson B. Assessing Student Perceptions of the Benefits of Discussion in Small-Group, Large-Class, and Online Learning Contexts. College Teaching [serial online]. Spring 2012; 60(2):65-75. Accessed December 15, 2013.

2. Pollock P, Hamann K, Wilson B. Learning through Discussions: Comparing the Benefits of Small-Group and Large-Class Settings. Journal of Political Science Education [serial online]. January 1, 2011;7(1):48-64. Available from: ERIC, Ipswitch, MA. Accessed December 17, 2013.

3. Koebler J. Experts: 'Weed Out' Classes Are Killing STEM Achievement. US News & World Report.  April 19, 2012. Online. Accessed December 14, 2013.

4. Maasoumi E, Milliment D, Rangaprasad V. Class Size and Educational Policy: Who Benefits from Smaller Classes? Econometric Review [serial online]. November 2005;24(4):33-68. Accessed December 17, 2013.

5. Konstantopoulos S, Chung V. What are the Long-Term Effects of Small Classes on the Achievement Gap? Evidence from the Lasting Beneftis Study. American Journal of Education [serial online]. November 1, 2009;116(1):125-54. Accessed December 14, 2013.

6. Nye B, Tennessee State Univ.  Small is Far Better. A Report on Three Class-Size Initiatives: Tennessee’s Student Teacher Achievement Ration (STAR) Project (8/85-8/89), Lasting Benefits Study (LBS:9/89-7/92) and Project CHALLENGE (7-89-7/29) as a Policy Application (Perliminary Results). Pager No. 5. [serial online]. November 13, 1992.  Accessed December 14, 2013.

Tips for Teaching Non-Native English Speakers

By Ittiporn Chuatrisorn, PGY-1 Pharmacy Practice Resident at the University of Maryland Medical Center

The non-native English speaking adult population in the United States is large and diverse.  Educational programs serve learners from very different backgrounds and with very different needs.^1,2  From my experience as an international student, I have found some teachers to be more understandable and have more effective techniques for teaching non-native English speakers (NNES) than others. This blog essay has been assembled as a resource for you - the instructors and preceptors who are encountering more and more NNES students.  I hope to provide you with some ideas on how to teach lectures and lead small groups in a manner that improves listening comprehension for NNES and ways to get your non-native speakers more involved in class discussions.

Monitor how you speak and what you say.^2-6  Pause for a couple of extra beats between sentences.  By reducing the speed of your speaking, even just alittle, you are giving the audience (NNES and English speakers alike) a chance to absorb what you say.  You do not have to cut your rate of speech in half; simply become aware of your speech and slow down! When a professor speaks slowly and clearly this helps NNES understand the concepts and, later, they do not have spent extra hours trying to find the concept in the textbooks or other resources.

Avoid idiomatic expressions and slang.^3-6 If you use idioms, slang, or long series of adjectives, define them and repeat the concept in more formal terms.  For example, one professor, when explaining how to approach a job interview, said “Do not shake hands like a wet fish when you greet the interviewer.” I had absolutely no idea what the professor meant until the idiomatic expression was explained to me!

Clarify examples that refer to cultural events or norms that may be unfamiliar to people from another country.^6-7 Some professors use “Ameri-centric” examples to explain concepts (e.g. references to U.S. history or popular culture). NNES who are unfamiliar with the examples may not understand the concept. You should take a few minutes to provide brief background information.  During one lecture regarding drug therapy reimbursement, for example, a professor talked about Blue Cross and Blue Shield, Aetna, and Humana. I had no idea that these were health insurance companies! At the end of that lecture, I did not understand anything. What may seem everyday and commonplace to the lecturer may be alien to the NNES student.  Of course, this is not unique to NNES students.  English speakers who are not from the US would have similar difficulties – but NNES students likely face even greater challenges when Ameri-centric examples are used.

Use visual aids.  Write key words on the board, slides, or otherwise provide visual cues to help NNES process meaning more readily.  This can include gestures, pictures, and concrete objects.^2-4 I think it is better to start early with visual cues in the class.  When the professor repeated key terms, wrote key terms on the board, and prepared a handouts in advance, this really helped me follow the lecture.  One of my pharmacy professors had a section of the blackboard site set aside for key terms and an outline of the day’s lesson plan.  He posted this information on board prior class. This gave me a refer to if I lost the thread of the discussion.

Provide written instructions about homework assignments and examinations.^2,6  Asking if everyone understands the assignment or knows when it is due may not be enough. NNES may be too embarrassed to speak up or may not even understand that you are discussing an assignment.^7-8 When I studied in pharmacy school, the professor asked if everyone understood the assignment, no one said a word. However, a Chinese colleague, who had only been in the country for 2 months, did not understand.  She misunderstood and presented a wrong topic. As you can imagine, she was very embarrassed!  Thus, oral explanations, without written support, could create problems for NNES.

Allow NNES students to record lectures.² They can listen as often as necessary to fill in their notes. Moreover, they can concentrate solely on trying to understand what the professor is saying during class, knowing that they can make notes later from the recording. In my first semester of school, I was shocked to discover that the professor did not begin at chapter one and continued through the textbook in a step-wise manner, chapter by chapter.  And they use a lot of materials outside the textbook. This can create confusion and obstacles for NNESstudents. Access to notes, class recordings, and a list of helpful outside sources helps NNES keep up the class.

Provide NNES students more opportunities to talk about the material.  Give NNES time to reflect before asking them to speak.^7-8 Many Americans think aloud, but people from many other cultures do not. What this means is that NNES rarely have time to reflect and respond before a native English speaking student (or, worse, the professor) has answered the question and taken over the conversation.  In some parts of the world, students are taught to be deferential, never challenging the teacher’s point of view or offering innovative ideas.⁸ For example, many Asian student would prefer to conform to tradition without trying to present novel ideas. Furthermore, you may experience difficulty-eliciting opinions from Korean, Japanese, Chinese, Thailand, and other students who come from educational systems where rote learning is the rule. Thus, you should give NNES a few minutes to write in response to a question before asking students to participate in the discussion.

Some NNES are very self-conscious about their imperfect English.^7,8 They may be frustrated by not being able to accurately articulate their complex thoughts. They may be concerned that their native English-speaking peers will think less highly of them. If the response is slightly off, try to do something positive with it. You should rephrase the response but don’t point out grammatical errors.²  Ask clarifying questions and elaborate on their response.

To summarize, NNES students are just as smart as native English speakers and will learn from you if you employ a few simple techniques. These students face many challenges navigating a new academic setting with different expectations, studying in a new language, and adapting to a new culture.  These techniques will help you to make your lectures and small group discussions more accessible to them, improve their listening comprehension, and enhance the learning experience for everyone.

Check out these additional resources:

1)   Responding to non-native speakers of English (University of Minnesota)

2)   Teaching Nonnative Speakers (Baruch College)

3)   ESL Instructional Resources (University of Washington)

References

1.   National Center for Education Statistics. Participation of adults in English as a second language classes: 1994-1995. Washington DC. Assessed 1 December 2013.

2.   Stevens LP, Jefferies J, Brisk ME, Kaczmarek S. Linguistics and Science Learning for Diverse Population: An Agenda for Teacher Education. In: Bruna KR and Gomez K, editors. The Work of Language in Multicultural Classrooms – Talking Science, Writing Science. 2^nd ed. New York: Taylor and Francis Publishers; 2009. p291.

3.   Wright WE. Foundations for Teaching English Language Learners: Research, Theory, Policy, and Practice. 1^st ed. Philadelphia. Caslon Inc. Publisher; 2010.

4.   Common Classroom Practices for All English Language Learner Educators. In: Wagner S, King T. Implementing Effective Instruction for English Language Learners: 12 Key Practices for Administrators, Teachers, and Leadership Teams. Philadelphia. 1^st ed. Caslon Inc. Publisher; 2012. p107.

5.   Tapia AT. Non-Native English Speakers Setting New Standard. New America Media Commentary. Assessed 1 December 2013.

6.   Lee DS. What Teachers Can Do to Relieve Problems Identified by International Students. New Directions for Teaching and Learning. 1997; 70: 40-51.

7. Nelson GL. How Culture Differences affect Written and Oral Communication; The Case of Peer Response Groups. New Directions for Teaching and Learning. 1997; 70: 77-84.

8. Hodne BD. Please Speak Up: Asian Immigrant Students in American College Classrooms. New Directions for Teaching and Learning. 1997; 70: 85-92.

Patients Counseling: Applying Gagne’s Nine Events of Instruction

by Chris Dobroth, Doctor of Pharmacy Candidate, University of Maryland School of Pharmacy

“Excuse me. Where are the paper towels? Oh, and I have a prescription to pick up.” Or "Really? We have to talk to another person before we can be discharged?” Not an auspicious start to a conversation, much less one regarding someone’s health. This indifference by many patients combined with the many demands for the pharmacist’s time calls for counseling sessions that are quick and efficient. What makes these sessions both quick and efficient as opposed to just quick?

The American Society of Health System Pharmacists (ASHP) in Guidelines on Pharmacist-Conducted Patient Education and Counseling states that patient counseling and education should be approached by pharmacists as “interrelated activities.”¹  According to the American psychologist Robert Gagne, there are nine events that should be included in every instructional activity to maximize its effectiveness.²  These events are the “why” behind the “how” of ASHP’s four steps to patient interaction.  Examining them more closely will help us develop a more complete understanding of what needs to happen during a patient counseling session. After all, at its core, effective patient counseling is a form of instruction.

Here are AHSP’s four step of patient counseling and how Gagne’s nine events of instruction related to each:^1,2

Step 1. Establish caring relationship, explain pharmacist role, and ask for permission to counsel. Gagne’s first and second event of instruction are to gain the learner’s attention and inform them of the objectives.

Step 2. Assess the patient’s knowledge and capabilities. Gagne’s third event is to stimulate recall of prior learning.

Step 3. Provide information to fill in the gaps in the patient’s knowledge. Gagne’s fourth and fifth events are to present content and provide learning guidance.

Step 4. Verify patient’s knowledge and understanding. Here Gagne’s sixth through ninth events are to elicit performance, provide feedback, assess performance, and enhance retention and transfer.

Notice how efficiently ASHP’s four steps contain all of Gagne’s nine events of instruction?

Let’s consider the Indian Health Services Three-Prime Questions:³ 

1. What did the doctor tell you this medication was for?
2. How did the doctor tell you to take this medication?
3. What did the doctor tell you to expect?

Gagne’s nine events show that these questions are an integral part of the learning process.² Here the use of opened questions serve to grab learners’ attention and engage them in conversation.  Now that you’ve got their brain running and you’re stimulating prior learning through the three questions, it’s your turn to assess their “knowledge and capabilities” as Gagne suggests and then to present the content and provide guidance.  You have already elicited the patient’s understanding, so now the holes can be filled in based on their level of health literacy. While the Indian Health Service’s Three-Prime questions lay the groundwork for an effective counseling session, they do not offer guidance from start to finish the way ASHP’s Four-Steps do.  Understanding Gagne’s nine events of instruction allows for an effective closer to the counseling session by utilizing the sixth through ninth steps: eliciting performance, providing feedback, assess performance, and enhance retention, and finally transfer to future situations.  Without these final steps, the patient will have endured a deluge of information but you won’t know if they really understood. While your intentions were good and the information thorough, the “teach back” technique is the only way to ensure the patient is truly ready to use their medication in an appropriate manner.

As the demands on our time increase, we must seek ways to be more efficient. By reminding ourselves that patient counseling sessions are a form of instruction, we can use Gagne’s nine events of instruction to master these techniques. Effective patient counseling leads to better outcomes and may decrease the amount of time spent re-answering the same questions at a later encounter.  So before your next patient encounter, take a few minutes to reflect on how you would approach it and remember Gagne’s nine events of instruction.

References:

1.  American Society of Health Systems Pharmacists. ASHP Guidelines on pharmacist-conducted patient education and counseling. Am J Health-Sys Pharm. 1997; 54:431-434. [accessed 2013 Sept 22].
2.  Buscombe, C. Using Gagne’s theory to teach procedural skills. The Clinical Teacher. 2013; 10: 302-307. [Accessed 2013 Nov 16^th].

3.  Indian Health Services: The Federal Program for American Indians and Alaska Natives [Internet]. Patient-Provider Communication Toolkit. Tool 9.  Pharmacist Consultation.  Rockville: Indian Health Services. [accessed 2013 Nov 16^th].

Can You Use Games to Train Your Brain?

by Allison Holllis, Pharm.D., PGY1 Pharmacy Practice Resident, University of Maryland Medical Center

Ever have trouble recalling the location of objects, remembering people’s names soon after they’ve introduced themselves, maintaining focus on important tasks all day, calculating figures in your head, or determining the best course of action?  Of course you have.  We all have!  We’ve spent years cramming entirely too much information into our heads and will spend the rest of our lives digging through the trenches of our brains trying to pull out stored information. If only there was a fun app to help keep our minds sharp, recall important information at the drop of a hat, improve our attention span, and increase our problem solving ability. Maybe there is!

Lumosity^TM is a web-based application that presents a variety of game-like exercises with the intent of improving your cognitive abilities. According to the Lumosity website, setting aside a few minutes each day to complete the tasks provided on their app can make you "smarter, sharper, and brighter."¹  The exercises designed by the folks a Lumosity are intended to improve specific brain functions such as sustaining attention, thinking before acting, visual and auditory processing, listening and reading.  Can playing games improve your brain power?  Sounds too good to be true!  Does brain training really work? The evidence is controversial.

A 2008 study by the psychologist Susanne Jaeggi found that memory training increased intelligence and implied that a person could boost their IQ by a full point per hour of training.² However, when a group of psychologists working at Georgia Tech tried to replicate the findings, with tougher controls, there was no evidence that it increased intelligence.³

A group of researchers in San Francisco examined whether Lumosity led to improvements in visual attention and working memory.  Participants were given initial cognitive assessments, randomly assigned to a training intervention group or waitlist control group, and then cognitive assessments where performed again following the intervention periods.⁴  The training intervention consisted of cognitive exercise sessions (20 minutes per day) using the Lumosity app. They found that the trained group improved significantly over the control group in the areas of visual acuity and working memory.

Could these brain games be useful in people with dementia and other cognitive impairments?  A 2013 study of brain training exercises in older adults with mild cognitive deficits found no statistically significant difference in the treatment and control groups.  But there was a trend toward better performance in the treatment group in those with the least impairment at baseline.⁵

The largest study ever conducted on brain training involved 11,430 participants who trained several times each week on cognitive tasks designed to improve reasoning, memory, planning, visuospatial skills, and attention.⁶  Although improvements were observed performing each of the cognitive tasks that participants were trained to do, there was no evidence of transfer to tasks they were not trained to do — even tasks that are cognitively related. The researchers found that regular players of brain games got better at the games themselves but did not experience marked improvement in fluid intelligence (the ability to solve novel problems and adapt to new situations). Researchers attributed the improvements not to increasing memory and skills but rather to learning how to play the games better and memorizing the answers.

Is there a role for brain training and apps like Lumosity in our classrooms? Applications such as Lumosity can be a fun way for students to engage in the learning process without even realizing they are learning! Lessons like math, spelling, and vocabulary can be taught via brain apps that quiz students and they can reinforce topics discussed in class.  Educators can also teach specific skills by playing games.  If these games present realistic cognitive tasks that are reasonably similar those needed in the “real world,” it can perhaps help students develop the necessary skills to be better practitioners.  Even if lessons aren’t targeted toward specific skills that might be used in practice, brain-training apps may be a useful way to help students build memory, perform calculations, and remain focused.

So the next time you are having trouble remembering where you left your keys or want to get better at Sudoku, consider Lumosity (and similar cognitive training tools) to help train your brain.  While it may or may not help you analyze clinical trial data or make important life decisions, it’s a fun way to keep you entertained during your next road trip!

References

1.   Lumosity [Internet]. [cited 2013 Sept 25]

2.   Jaeggi S, Buschkuehl M, Jonides J, Perrig W. Improving fluid intelligence with training on working memory. PNAS Early Edition. 2008:10:1-5.

3.   Redick TS, Shipstead Z, Harrison TL, Hicks KL, Fried DR, Hambrick DZ, Kane MJ, Engle RW. No evidence of intelligence improvement after working memory training: a randomized, placebo-controlled study. J Exp Phychol Gen 2013:142:359-79.

4.   Hardy J, Drescher D, Sarkar K, Kellett G, Scanlon M. Enhancing visual attention and working memory with a web-based cognitive training program. Mensa Research Journal 2007: 42:13-20.

5.   Zhuang JP, Fang R, Feng X, Xu XH, Liu LH, Bai QK, Tang HD, Zhao ZG, Chen SD. The impact of human-computer interaction-based comprehensive training on the cognitive functions of cognitive impairment elderly individuals in a nursing home. J Alzheimers Dis. 2013:1:36:245-51.

6.   Owen AM, Hampshire A, Grahn JA, Stenton R, Dajani S, Burns AS, Howard RJ, Ballard CG. Putting brain training to the test. Naure 2010:456:775-8.

Reuse, Reduce, Recycle...Test Questions?

by Hana Kim, Pharm.D., PGY-1 Resident, Kaiser Permanente of the Mid-Atlantic States

As an instructor, it is difficult to create quality test questions, let alone make new ones every year.  The question as to whether to recycle questions is an issue any organization administering tests or examinations must address. When a question is reused, it can increase the possibility that it will become more public to test takers and can give an advantage to those who have access to the question.  There is also concerned about repeat examinees, those who fail an examination and are required to retake it.  Does reusing test questions appropriately challenge students to learn the material?  Does it provide an unfair advantage to a select few students who have access to the recycled questions?  And, what can instructors do to alleviate these concerns?

Timothy J. Wood studied the role of reused test questions on repeat examinees who took the Medical Council of Canada (MCC) Evaluation Examination (MCCEE).¹ The MCCEE is a basic medical knowledge test for International Medical Graduates that contains 324 multiple choice questions. The MCCEE is offered 3 times a year, 4 months apart. Wood investigated whether prior exposure to test questions enhanced performance among 130 repeat examinees. The examinees were presented 36 repeat questions from the previous examination.  The examinees had no knowledge that these questions would be reused.  The investigators found that the examinees scores on the reused and non-reused questions increased equally, a finding that was consistently with previous studies. He noted that this might be due to increased knowledge of the subject matter, hopefully due to an increased amount of studying in preparation for the re-take exam. Wood concluded that prior exposure to test questions had little impact on the performance. Although there was no difference in examinee performance on repeat questions, the exam was given 4 months apart and the examinees did not have access to the questions in between test administration, so the results may not be representative of what we’d see in most academic settings.

Similarly, Wagner-Menghin and colleagues conducted a study to evaluate the effect of reusing written test questions. The authors specifically utilized the Rasch model, which is a probabilistic psychometric framework measurement model that estimates item difficulty and ability measures. The study noted four conceptual factors that should be taken into consideration:

• Reuse expectation: passing items along to new test takers can be beneficial, especially when reusing items is expected
• Cheating attitude: many studies on cheating have not focused on cheating with the reuse of test questions
• Exam’s consequences: there are consequences if a student fails an examination; therefore, the pressure to pass may promote cheating
• Item content: reusing questions that require student application of knowledge, not simply recall, may diminish the test validity²

This study was designed to quantify the reuse of test questions based on an item’s level of difficulty. Specifically, the authors introduced a new written multiple-choice course exam to assess clinical skills in 671 medical students. To assist the students, a “representative” set of multiple-choice questions was included in the official study materials. Looking at item content, there was a larger effect on those that tested application of knowledge versus recall questions.  There was a 50% increase in student scores on application test questions that were reused compared to 20% increase when recall-type test questions were re-used.  Although the re-used material did not result in increased overall scores, the authors postulate that the lack of benefit may be due to a ‘deficit in study organization and time management for late test takers.’  A test with 30-45% reused questions (particularly with a large number of recall questions) is, therefore, unlikely to substantially benefit students.²

When questions are pass down from one student to another, it creates a potential unfair advantage as some students have access to the items while others do not.  What can professors do to mitigate the problem of questions being passed down? One possible solution is to make all previous examinations available to students so there is fair opportunity for all students. This is exactly what a law professor at George Mason University has been practicing in his classes. He develops new questions for every exam and makes past exams and answers available to students.³  This strategy certainly improves fairness but increases faculty workload the workload as new questions need to be formulated every year.

While some studies demonstrate a statistically significant difference in scores when questions are reused, one of the biggest concerns is that questions will get passed down from year to year, increasing the possibility of “cheating.”  These situations are inevitable but instructors should consider safeguards to help mitigate this problem. Some options include creating new test questions ever year or allowing a sufficient amount of time (2-3 years) between question re-use. Although there are several suggested solutions, the question as to whether test items should be reused and recycled remains an unanswered one.

References:

1. Wood TJ. The effect of reused questions on repeat examinees. Adv Health Sci Theory Pract. 2009; 14(4): 465-73.

2. Wagner-Menghin M, Preusche I, Schmidts M. The Effects of Reusing Written Test Items: A Study Using the Rasch Model. ISRN Education 2013; Article ID 585420.  Accessed 17 November 2013.  
3. Somin I. The Perils of Reusing Questions from Past Exams. The Volokh Conspiracy. http://www.volokh.com/2011/01/18/the-perils-of-reusing-questions-from-past-exams/. Accessed 27 October 2013.