Using Audience Response Systems

by Adrian Hui, Pharm.D., PGY1 Pharmacy Resident, MedStar Union Memorial Hospital

How many of you have used an audience response system? Please use your devices to select an answer. 

A.    Yes

B.    No

C.   Not sure

D.   What is an audience response system?

Many of you may be familiar with this scenario, commonly seen in classroom settings today.  With their increasingly widespread use, audience response systems have made their way into television game shows such as “Who Wants to be a Millionaire” where the audience can be polled to help the contestant answer a question.  If you are not familiar with the term “audience response system,” perhaps one of the following alternatives may ring a bell: personal response system, electronic voting system, or student response system.¹

What exactly are “audience response systems (ARS)?”  ARS are tools used to quickly gather data from people participating in a poll.  ARS made their first appearance in the 1960s at Cornell and Stanford Universities² but were costly, cumbersome, and frequently malfunctioned.¹  Eventually, ARS became commercially available in the 1990s, and with several technological improvements (e.g. infrared capability, portability), it soon became widely used in various educational settings in the early 21^st century.¹  Nowadays, ARS are simple and easy to use.  They only require the following components:³

• Presentation software (i.e., PowerPoint with Turning Point)
• Hardware to transmit information (i.e., “clickers”, mobile computers, mobile phones)
• Wireless receiver or Internet connection to a computer to receive the data

I first experienced ARS as an undergraduate student when I was required to purchase my first “clicker” for an introductory genetics class.  The device was about the size of a small TV remote and had buttons for the numbers 0-9, the letters A-J, and a few other miscellaneous functions.  We were required to bring the “clickers” to every class and use them to answer multiple choice quiz questions interspersed throughout the lectures.  Our answers on these “quizzes” counted towards 10% of our final grade.  This mode of learning was unlike anything I had ever experienced before, and although I initially felt apprehensive about ARS, I eventually came to appreciate its utility. 

There are several benefits to using ARS.  Once a question is posed, responses are anonymous and summarized graphically for the audience.  However, the instructor has the flexibility to track each individual response and can collect the data for a grade or to give credit for attendance.  Anonymity can have a positive impact on learning, often enhancing participation, engaging the students, and promoting greater interactions between the students and their peers/teachers.¹  In fact, a study by Clauson et al.⁴ showed that 93.2% of pharmacy students appreciated the anonymity of ARS.  To gauge how well the students understand the material being covered in class, the instructor can use ARS intermittently throughout a lecture.  Based on the results, the instructor can determine whether it's appropriate to re-explain concepts more in-depth if the class seems to be struggling or continue on to the next topic.  Thus, ARS, if used effectively, can improve the quality of learning and provide both the instructor and students valuable feedback.  ARS may help improve test results as well.  In a study by Cain et al., final grades for a Physiological Chemistry and Molecular Biology (PCMB) II course in 2008 (using ARS) were significantly higher than those in previous years without ARS (p < 0.05 for 2008 vs 2007 and p < 0.001 for 2008 vs 2006).  Also, students taking PCMB II with ARS in 2008 had significantly higher final grades compared to when they took PCMB I the previous semester without ARS (p < 0.001).  Although this study has some limitations, the findings suggest that ARS may be a useful tool to improve student performance on exams.³  See Table 1 for a summary of the benefits of ARS.¹

Table 1: Summary of ARS Benefits

Classroom environment benefits

Attendance: Students go to class more

Attention: Students are more focused in class

Anonymity: All students participate anonymously

Participation: Students participate with peers more in class to solve problems

Engagement: Students are more engaged in class

Learning benefits

Interaction: Students interact more with peers to discuss ideas

Discussion: Students actively discuss misconceptions to build knowledge

Contingent teaching: Instruction can be modified based on feedback

Learning performance: Learning performance increases as a result of using ARS

Quality of learning: Qualitative difference when learning with ARS (e.g., better explanations, thinking about important concepts, resolving misconceptions)

Assessment benefits

Feedback: Students and teachers like getting regular feedback on understanding

Formative: Assessment is done that improves student understanding and quality of teaching

Compare: Students compare their ARS responses to class responses

 Adapted from Kay et al (2009)¹

Although ARS has some benefits, there are some challenges too.  From my experience, the “clickers” were a source of stress when students lost/forgot them or had technical difficulties.  This was especially the case when the “clickers” were used to answer quiz questions that counted toward a student’s grade or attendance.  Cain et al.³ found that the most common complaint about ARS (shared by 82% of pharmacy students) was the cost of the devices.  It should be noted, however, that some schools “cover” the cost of the devices and recirculate them to students in different classes.  But let’s face it, even when the school buys the devices to be shared by students, the cost of this technology is, in the end, bore by students in the form of tuition and fees.  Technical problems with ARS can also be frustrating for instructors that have integrated ARS questions into their lectures.  Furthermore, ARS may be a challenge for relatively inexperienced instructors unable to adapt their teaching to address unexpected responses from the students.  Developing ARS questions can be time consuming and takes skill and experience in order to create truly effective ones.  Effective questioning should not target simple recall of the material being presented but rather should cultivate conceptual, analytical, and problem solving skills.⁵  Such thoughtful questions are more likely to lead to enriching discussions and stimulate higher-level cognitive processes instead of serving as one-dimensional questions simply used to keep the student awake.  For a summary of the challenges with ARS, see Table 2.¹

Table 2: Summary of ARS Challenges

Technology-based challenges

Bringing remotes: Students forgot/lost remotes and could not participate in class

ARS did not work: Remote devices did not function properly

Teacher-based challenges

Responding to student feedback: Less experienced teachers cannot adjust to student feedback

Coverage: Cover less course content if ARS is used

Developing questions: Time consuming to create ARS questions

Student-based challenges

New method: Students find it difficult to shift to a new way of learning

Discussion: Discussion leads to confusion or wasting time

Effort: Too much effort is required by students when using ARS

Summative assessment: Using ARS for tests may not be popular with students

Attendance for grades: Students do not like ARS used for monitoring attendance

Identifying students: Students want to remain anonymous

Negative feedback: Students feel bad when receiving negative feedback

 Adapted from Kay et al (2009)¹

How can we optimize the use of ARS in the classroom?  Since our attention span often fades after about 20 minutes,¹ one strategy is to provide ARS questions every 20 minutes to break up a lecture.  Along with using effective questions, a small incentive to increase participation and attendance may be helpful.  Using ARS responses for 5% of a student’s grade is sufficient to increase motivation and attendance.¹  Lastly, although ARS may take away time to cover material, I would argue that ARS enables the instructor to explore concepts in greater depth and thus provides greater value in terms of student learning.  Overall, the benefits of ARS seem to outweigh its challenges, and several studies have shown that the majority of students favor its use in the classroom.  Yet, more research is needed to better understand the utility of ARS and to hone best practices.¹ 

References:

1. Kay RH and LeSage A. Examining the benefits and challenges of using audience response systems: a review of the literature. Computers and Education. 2009; 53:819-27.

2. Nelson C, Hartling L, Campbell S et al. The effects of audience response systems on learning outcomes in health professions education. A BEME systematic review: BEME Guide No. 21. Med Teach. 2012;34(6):e386-405.

3. Cain J, Black EP, Rohr J. An audience response system strategy to improve student motivation, attention, and feedback. Am J Pharm Educ. 2009; 73(2): Article 21.

4. Clauson KA, Alkhateeb FM, Singh-Franco D. Concurrent use of anaudience response system at a multi-campus college of pharmacy. Am J Pharm Educ. 2012;76(1): Article 6.

5. Dangel HL and Wang CX. Student response systems in higher education: Moving beyond linear teaching and surface learning. Journal of Educational Technology Development and Exchange 2008; 1(1): 93-104.

Pharmacy Education 2.0: Incorporating Social Media

by Deanna Tran, Pharm.D., Assistant Professor, University of Maryland School of Pharmacy

“OATUS, IMHO that class was so much fun LOL! T2UT”

Do you understand that sentence? I do! As a member of the Millennial generation, information technology and multimedia are second nature to me. This is also true for many current college students. According to USAToday, ninety-five percent of college students use Facebook for an average of 238 minutes per week!¹ Millennials prefer learning environments that incorporate technology, active learning, teamwork, and multitasking.² And now that I’ve started a career in academia I can’t help but wonder whether we should incorporate social media platforms, such as Facebook, Twitter, YouTube, and Google+ into our classroom instruction.

Social media platforms play such a large role in popular culture. By incorporating them into the classroom, we could potentially increase student involvement, interest, and enthusiasm in the course content. When social media platforms were used in a graduate-level medical humanities elective at Penn State College of Medicine, the students expressed satisfaction with its use, and the instructors noted increased interest in classroom activity.³

Using different social media platforms can also help enrich student learning. In the medical humanities elective at Penn State, YouTube was used to demonstrate cross-cultural perspectives on aging and mental health.³ Skype allowed experts from across the United States to share their experiences with the students.³ These two social media platforms provided the students unique opportunities to connect with experts and real-life patients, something that the instructors could not have provided otherwise. The instructors also used Twitter. Real-time observations were solicited from the students while they were at off-site outreach events. The instructors also noticed that students learned how to phrase thoughts and ideas more succinctly when they used Twitter.³ This is a skill that not only needs to be taught to our future physicians, but also student pharmacists. Additionally, Twitter could be used to make lectures more dynamic; for example, discussion questions could be posted on Twitter during lectures. Lastly, Twitter might be a useful educational tool because it can serve as an additional channel for dialogue between the students and instructors. Twitter might encourage engagement with the quieter students who may be afraid to speak up during class as well as students who may need more time to formulate responses.

Using social media may also help to address different learning styles. Twitter, Facebook, and Google+ have the capability to host small group discussions. This would probably resonate with the “Doers” of the class, whom according to David Kolb, tend to like small group discussions and self-directed work.⁴  Online text-based discussions might appeal to the “Watchers”, who prefer observation and reflection, the time to read and ponder on what’s being discussed prior to responding. Lastly, online discussions would allow the “Feelers” of the class to learn from their peers.

Social media platforms can facilitate many student-directed activities such as posting blog entries and making videos. These projects are consistent with the principles of andragogy. By making students responsible for their own learning, students would not only be more invested, but also have a deeper understanding of the content.

Lastly, social media could capture the attention of the class since technology is something many students find interesting. If it is used at the beginning of class session, social media might help the instructor gain attention – the first step in Gange’s nine events of instruction.⁵

Even though there appear to be many benefits to integrating social media into the classroom, there are some limitations and concerns. There are currently no best practices guidelines on how to most effectively use these technologies and there a very limited number of studies.⁶   Thus, more evidence is desperately needed.  Social media is a public form.  Therefore, there are concerns that students might post unprofessional content online.  And privacy and confidentiality issues need to be considered.  Social media use can also be distracting in the classroom. In a Pharmacy Practice Development, Management, and Evaluation class at Auburn University Harrison School of Pharmacy, students were required to post at least two tweets during class.² Surveys at the end of the cources indicated that students felt that Twitter distracted them from taking notes. Also, there is no mechanism to prevent students from sending private tweets to one another during class. Lastly, we can’t forget that the instructor has to feel comfortable using the technology too! I’ve had a few experiences where professors were not familiar with the available technology in the classroom.  I felt frustrated and distracted as the instructor fumbled along trying to get everything to work. This makes for a less than ideal learning environment!

I believe the potential advantages of social media outweigh the disadvantages and that we should selectively incorporated these new tools into the classroom. Many of the disadvantages could be overcome.  When using social media, instructors need to set clear expectations in order to prevent any unprofessional behaviors.³   Clearly more research is needed and we should all be will share our experiences so that best practices will emerge.  There is no doubt that social media is here to stay and as an educator I believe we should not ignore it but embrace it!

References:

1.   Marklein, Mary Beth. Teachers embrace social media in class. USAToday. 2012 Mar 4 [Updated 2012 Mar 4; cited 2012 Oct 14].

2.   Fox B, Varadarajan R. Technology in Pharmacy Education Use of Twitter to Encourage Interaction in a Multi-CampusPharmacy Management Course. Am J Pharm Educ 2011;75(5):1-8.

3.   George D, Dellasega C. Use of social media in graduate-level medical humanities education: Two pilotstudies from Penn State College of medicine. Med Teach. 2011;33:e429-e434.

4.   Kolb David. Experiential learning: experience as the source of learning and development. Englewood Cliffs, New Jersey: Prentice Hall. 1984.

5.   Gagné, R. The Conditions of Learning and the Theory of Instruction4th edition. New York: Holt, Rinehart, and Winston. 1985.

6.   Copland, D. For Social Media In the Classroom To Work, Instructors Need Best Practices. ReadWriteWeb. 2012 May 2 [updated 2012 May 2; cited 2012 Oct 14]. 

Self-Assessment versus Self-Monitoring

by Sarah A. Pierce, Pharm.D., PGY1 Pharmacy Practice Resident VA Maryland Health Care System

Most pharmacy residents are familiar with the use of ResiTrak^TM to complete self-evaluations, an arduous process made more difficult by having to recall performance over a long period of time.  Is this method of self-evaluation effective?

In both pharmacy education and residency training, self-assessment is a commonly utilized tool intended to encourage a learner to evaluate his or her performance, identify strengths and weaknesses, and note areas for self-directed learning and growth. In its accreditation standards for Doctor of Pharmacy programs, the Accreditation Council for Pharmacy Education (ACPE) discusses the importance of self-assessment for students, faculty, and staff.¹  The theme of self-assessment and self-directed learning persists into post-graduate residency training. The American Society of Health-System Pharmacists (ASHP) includes “resident self-assessment of their performance” as a requirement in their accreditation standards for PGY1 pharmacy residency programs.²   The ASHP standards require “summative evaluations” at the end of each learning experience (aka “rotation”) and  encourages optional spontaneous “snapshot” self-evaluations too.

Implicit in these requirements is an assumption is that self-assessment is valuable and accurately reflects a person’s strengths and weaknesses. However, according to Eva and Regehr, there is substantial body of literature which suggests that learners often cannot accurately self-assess their strengths and weaknesses and that self-assessment correlates poorly with actual performance.³  However, there may be a distinction between self-assessment and self-monitoring: “self-assessment as a cumulative evaluation of overall performance, and self-assessment as a process of self-monitoring performance in the moment” [emphasis added].³

Eva and Regehr discuss the results of two studies which explored self-monitoring and self-assessment.^3,4 In each study, participants answered sixty trivia questions divided into six categories.  Participants were asked to evaluate their performance at different times during the testing. To measure “self-assessment” (that is, a cumulative evaluation of overall performance), the researchers had participants predict their overall score for each category both before and after completing all ten trivia questions in that category. To measure “self-monitoring” (that is, an evaluation of performance while in the moment), the researchers had participants rate their confidence in a given answer immediately after answering the question. The results showed that a “self-monitoring” measure demonstrated a higher correlation with actual performance when compared to the cumulative “self-assessment” measure. 

They hypothesized a potential explanation for these findings.   Self-monitoring likely requires a “fundamentally different cognitive process” than self-assessment. With self-monitoring, learners have many inputs and sources of information at their disposal to predict potential success or failure on a moment-to-moment basis. However, with self-assessment, the learner must rely on memory to aggregate information of multiple past events in order to determine the overall success or failure.³ The concept of self-monitoring was replicated and expanded in work by McConnell and colleagues.⁴

What are the potential implications of these findings? In the Educational Theory and Practice course, the idea of self-directed learning was introduced.⁵   Self-assessment is a tool used to facilitate self-directed learning. However, if self-assessment is not as accurate as one may hope, then perhaps self-assessment is not the best tool to facilitate self-directed learning. I would argue that more attention should be directed to self-monitoring on a moment-to-moment basis, rather than on cumulative self-assessments.  Through  self-monitoring, individuals would develop a more accurate picture of their abilities and this could lead to more focused self-directed learning needs. As self-assessment is by far the most common self-evaluation tool used in pharmacy education and residency training today, new, creative ideas to transition to self-monitoring are needed.

Self-monitoring could be implemented in pharmacy education and residency training in several ways.  In pharmacy school, an early and consistent emphasis on self-monitoring could prove to be more effective than intermittent, reflective self-assessments.  By asking students to identify their strengths and weaknesses in real-time, this may motivate them towards focused self-directed learning.   For example, students taking an exam on the pathophysiology of diabetes, the pharmacology of diabetes medications, and diabetes management could be required to rate their confidence on each test question. After the exam, students could receive a report with their self-monitoring responses and a breakdown of their actual performance in each domain (e.g. pathophysiology, pharmacology, and patient management). In this way, students would be better at identifying areas they struggled with, gradually improve the accuracy of self-monitor their performance, and note areas that require further study. Regularly repeating this process may help students develop stronger self-monitoring skills and help them become independent practitioners after graduation.

A similar argument could be made for pharmacy residency programs. I believe “snapshot” evaluations should be used more frequently.  During my internal medicine rotation, I was asked to complete a snapshot evaluation related to my data gathering skills and treatment plan for a specific patient. This was much more focused than the summative self-evaluation done at the end of my rotation whereby I had to assess my overall performance related to several different goals and objectives. I believe I gained more insight into my strengths and weaknesses with the snapshot evaluation.  I was able critical examine my performance on narrow set of tests “in the moment” rather than having to search my memory for past events that related to my performance.

While self-assessment is certainly a necessary component of pharmacy education and helps facilitate self-directed learning, I believe there should be a greater emphasis on “real-time” self-monitoring.   Self-monitoring should be taught and required in Doctor of Pharmacy programs and frequent self-monitoring “snap shots” should be a mandatory component of pharmacy residency training.

References

1.  Accreditation Council for Pharmacy Education. Accreditation standards and guidelines for the professional program in pharmacy leading tothe doctor of pharmacy degree. Chicago: Accreditation Council for Pharmacy Education; 2011.  97 p.  [cited 2012 Oct 8]

2.  ASHP Commission on Credentialing. ASHP accreditation standard for postgraduate year one (PGY1) pharmacy residency programs. Bethesda (MD): American Society of Health-System Pharmacists; 2005. 23 p.  [cited 2012 Oct 8]

3.  Eva KW, Regehr G. Exploring the divergence between self-assessment and self-monitoring. Adv Health Sci Educ Theory Pract 2010;16(3):311-29. [cited 2012 Oct 8]

4.  McConnell MM, Regehr G, Wood TJ, Eva KW. Self-monitoringand its relationship to medical knowledge. Adv Health Sci Educ Theory Pract 2011;17(3):311-23. [cited 2012 Oct 8]

5.  Kaufman DM. Applyingeducational theory in practice. BMJ 2003; 326: 213-6. [cited 2012 Oct 8]