Narrative Evaluation System

by Caroline Kim, Doctor of Pharmacy Candidate, University of Maryland School of Pharmacy 

Years ago, I attended elementary school in South Korea. My report card consisted of a letter grade accompanied by a short narrative from my teachers.  These comments stuck with me more than the grades.  They gave me insight into my performance beyond numbers.  They explained areas where I needed to improve — even if I earned a high grade — and areas where I did well despite a poor grade.  The finality of my letter grades was inescapable, but the comments were a source of motivation for further learning.

The purpose of evaluation in education is to guide student development, promote excellence, aid in defining the successful completion of a program, and provide performance evidence.¹  The traditional grading system might be effective in promoting high standards, but it fails to capture abstract details related to student performance. Traditional grading systems also provide little room for feedback to guide the future learning process.  I would argue that there is a need for a more effective method of evaluation beyond numerical and letter ‘grade’ classification systems.

The Narrative Evaluation System (NES) is a nontraditional grading system, which provides constructive feedback on student’s performance using a narrative format.  In construct, NES draws heavily from Walter Fisher’s narrative paradigm theory. ²  The theory states that narration and storytelling are the basis of all human communication. ²  In the NES system, the evaluator takes into account the objectives of the course when evaluating the strengths and weaknesses of a student’s performance.  Like traditional evaluation models, NES also recognizes noteworthy student performance.  In addition, NES encourages cooperation and collaboration by personalizing descriptions of student performance. ³ This helps to promote closer relationships between students and instructors.

NES does have some drawbacks.  NES requires a larger time commitment from educators. ³  In order to summarize student performance into a narrative, educators are required to keep detailed notes on each performance and provide suggestions on how students can improve in the future.  NES explores more dimensions of the evaluation process and educators must devote more time and energy to constructive criticism.  Simply put, NES is more difficult than  the simple categorization required in traditional grading systems.  Even after providing narrative comments, some institutions force educators to use traditional grading systems — allowing a comparison between students using the same grading scale.  The NES system supplements traditional evaluation.³   Students who have been evaluated using the narrative evaluation system often request letter grades — in one study more than two thirds did so.⁴ The most common reason for requesting letter grades was to understand the narrative evaluation by comparing their performance to grades that they are familiar.⁴ 

Very few studies have analyzed the impact of NES on future success.  One study examined whether graduates from a program that used NES remained competitive when it came to admissions into graduate and professional programs.³ When comparing students who graduated from the University of California at Santa Cruz (UCSC) — which uses a narrative grading system — with other University of California campuses which assign traditional grades, the investigators found that UCSC students generally performed as well if not better in terms of acceptance rates into graduate and professional schools.³ 

In another study, a survey of UCSC students and alumni explored their attitudes towards NES.  The survey respondents indicated they preferred NES because it pointed out strengths and weaknesses, and helped them grow personally and professionally.  They appreciated how NES promoted cooperation, scholarship, and creativity. ⁴ However, 62% of undergraduate students surveyed and 54% of alumni surveyed reported some quibbles with NES.⁴ The problems included the length of time it took to receive the narrative, failure to receive a constructive narrative, or evaluations that were similar to and provided little addition information beyond letter grades. ⁴ Despite these challenges, 82% of undergraduate students and 78% of alumni favored NES over a traditional letter grade.<sup>4 

Figure 1. This table summarizes a surveyed undergraduate and alumni opinions of NES.4</sup>

Survey Questions	Undergraduate (%)		Alumni (%)
	Yes	No	Yes	No
NES influenced decision to attend the school	58	42	48	52
Favored NES over letter grades	82	14	78	18
Requested letter grades	32	68	N/A	N/A
Believed NES worked in their favor	N/A	N/A	62	37
Experienced a problem with NES	62	38	54	46

Another study compared the effect of NES and a conventional grade system on learning and motivation.⁵  During the study students were asked to perform certain learning tasks during 3 distinct teaching sessions.  Students were randomized to receive either narrative or numerical grades when completing the learning tasks in sessions 1 and 2, and then took a post-test after session 3.⁵ Results showed that students who received narrative evaluations had the highest post-test performance.  Moreover, traditional grades depressed creativity and weakened student interests in the subject matter.⁵ 

Currently, at least seventeen colleges and universities in United States and Canada use narrative evaluation (NES).  Some have used this system for over fifty years.  NES can guide students in their learning and help them improve on their weaknesses.  The NES model provides a more complete picture of the student and helps students to distinguish themselves as individuals.  Although it may be time-consuming, educators should consider implementing NES – perhaps in conjunction with traditional grades — to enhance the learning process. 

References

1. Hanson J, Rosenber A, Lane L.Narrative Descriptions should Replace Grades and NumericalRatings for Clinical Performance in Medical Education in the United States. Frontiers in Psychology.  Nov 2013; 4: 1-10.
2. Fisher W. The Narrative Paradigm: In the Beginning. Journal of Communication. Dec 1985;35(4): 74-89.
3. Affirmation of Accreditation Self-Study Report. University of California at Santa Cruz. [Internet].  1994 Nov: 22-29. [cited 2015 Mar 8].
4. Wong M. Assessment and Evaluation of Past and Present Student Attitudes toward the UC Santa Cruz Narrative Evaluation System. University of California at Santa Cruz. [Internet]. 1992 Jun. [cited 2015 Mar 9]
5. Butler R,Nisan M. Effects of no Feedback, Task-related Comments, and Grades on Intrinsic Motivation and Performance. Journal of Educational Psychology. Jun 1986; 78(3): 210-216.

Self-Determination Theory: Supporting Students’ Intrinsic Motivation

by Adrienne Kowcz, PharmD., PGY1 Community Pharmacy Resident, University of Maryland School of Pharmacy

From day one, we are innately curious. Children constantly explores the world around them, and as we grow, we gravitate toward careers and hobbies that interest us.  We are intrinsically motivated to learn more. Educators should capitalize on this desire when designing instruction.  Unfortunately, external motivators play far too great a role in education today.² Overly prescriptive supervision and standardized evaluations can thwart the learning process, and cause learners to lose interest.

Self-determination theory (SDT) suggests that teachers can capitalize on internal motivation by supporting each student’s natural tendency to be curious and desire for autonomy.  That is, teaching should be guided by students’ interests. If we can find ways to support autonomous motivation in the delivery of instruction, we can achieve optimal learning. Although SDT has been around for 4 decades, Edward L. Deci and Richard Ryan expanded on the theory by refining the differentiation between intrinsic and extrinsic motivation as well as proposing three key intrinsic needs involved in self-determination.¹

Motivation is often examined from the perspective as to how we convince others to change their behavior. External factors, such as rewards, punishments, grades, evaluations, and other’s opinions often motivate people. However, SDT explores the intrinsic motivation, or how people are motivated from within when there is no tangible reward or external push. Deci and Ryan postulated that an individual needs intrinsic motivation as well as three intrinsic psychological needs in order to initiate these behaviors and maintain good psychological well-being and self-determination. These universal needs are autonomy, competence, and relatedness. Below is a model that depicts SDT in higher education. These components together form the self-determination model which emphasizes supporting student autonomy in order to achieve positive learning outcomes.

Autonomy relates to people feeling as though they have power over their behaviors. Giving students opportunities for growth and the ability to make their own choices, increases their sense of autonomy and reduces coercion / controlled. One study examined the learning outcomes of college students in a science course where the teachers used two different outcome expectations: one group was told they had to teach the material to others (autonomy supported) and the other group was told they had to pass an exam (controlling behavior).³ Those who had the expectation of teaching others had a deeper conceptual understanding of the material.  Autonomy can be supported in the classroom by teachers reducing the number of evaluations and encouraging active student participation, fostering positive feelings that what students say has purpose in the classroom. Also, teachers should provide a clear rationale for the learning activity. This has been shown to improve student’s effort to learn.

Competence refers to a person’s effectiveness at performing the skill or task. When someone feels competent that they can perform a task, they are more likely to continue to use what they have learned and strive to achieve more. A way to support competence in an educational setting is introducing challenging activities where students can use their previous knowledge and skill. When students perform well, providing feedback about the process can be beneficial to their growth. Not congratulating a job well done, but rather letting the student know that their effort was recognized.  In addition, providing feedback on how to master more has been shown to make students continue to strive for greater results after a compliment.⁶

Relatedness is the last need that Deci and Ryan believe should be satisfied in order to support self-determination. In a classroom, when students have a sense of belonging, that those around them truly understand and value them, they will more likely have intrinsic motivation to perform the tasks at hand. Acknowledging student’s feelings can help improve that connection. In an education setting, studies have shown that students who feel connected to teachers do better in school than those who are disconnected and do not have a relationship with someone who truly cares about them in the school.⁴ As professors, we have the ability to get to know students, as well as teach in an inviting environment where we encourage participation and provide positive feedback to encourage growth.

Creating an autonomy supporting environment is not only beneficial in the classroom, but in the clinical environment too. In the study titled Reducing the Health Risks of Diabetes: How Self-determination Theory May Help Improve Medication Adherence and Quality of Life the investigators applied the SDT model to predict medication adherence, quality of life, and physiological outcomes among patients with diabetes.⁵ Patients were surveyed assessing their perceived autonomy-support from their providers, autonomous self-regulation for medicine use, perceived competence to perform self-management, medication adherence, and quality of life. Results showed that when clinicians elicit patients’ perspectives, just as teachers elicit student’s responses, and support autonomy, patients have higher quality of life, improved medication adherence, and better health outcomes.

Regardless of setting, supporting autonomy, competence, and relatedness leads individuals — students and patients — to become better learners motivated by their internal desires. When these needs are supported, people gain self-determination and their intrinsic motivation to learn is enhanced.

References

1. Deci EL, Ryan RM. The “what” and “why” of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry 2000; 11: 227-268.
2. Williams G., Deci E. The Importance of Supporting Autonomy in Medical Education. Ann Intern Med. 1998;129:303-308.
3. Niemiec CP, Ryan, RM. Autonomy, competence and relatedness in the classroom:Applying self-determination theory to educational practice. Theory and Research in Education. 2009; 7(2): 133-144.
4. Vallerand RJ, Reid G. On the causal effects of perceived competence on intrinsic motivation: A test of cognitive evaluation theory. Journal of Sport Psychology 1984; 6: 94–102.
5. Williams, G. Patrick, H. et al. Reducing the Health Risks of Diabetes: How Self-determination Theory May Help Improve Medication Adherence and Quality of Life. Diabetes Educ 2009; 35 (3): 484-492.
6. Dweck, Carol. The power of believing that you can improve. Dec 2014. 

Embracing Mobile Technology – A Gateway to Learning

by Joanna Yala, PharmD, PGY1 Pharmacy Practice Resident, Sinai Hospital 

In an increasingly technological world, teachers and students are being pushed to adapt. Smartphones, tablets, and laptop computers are quickly replacing the conventional blackboard and chalk. I believe that transitioning from traditional note-taking and lectures to incorporating mobile devices and on-line information sharing into our classrooms can positively impact the learner’s experiences.  In this discussion it is helpful to understand what learning and memory entails. Learning is the acquisition of skill, knowledge as well as attitudes, while memory is the expression of what the learner has acquired.¹ 

As today’s technology constantly develops, we face a dilemma — risking turning away from what is already tried and tested to venturing out into the unknown and untested. As educators, we want to fulfill our duty to prepare our students with the knowledge and skills necessary to equip them for their future endeavors, but we also want to deliver the learning experiences in a controlled, safe environment. Thus, the decision for institutions to shift towards online classrooms and the use of mobile devices in tandem with the face-to-face instruction is critical.

There have been multiple studies that have evaluated the use of the mobile devices (e.g. smartphone, tablet, and computer) as an instructional tool. Social media allows a reflective output for ideas and computer applications have allowed for file-sharing.  These advances have expanded our capabilities and created virtual centers of learning.

An observational study of undergraduate students’ adoption of a mobile note-taking tool was conducted at the University of Chester in the United Kingdom.²  The software tool allowed educators to provide mobile support to students’ learning and provided an array of functions for the gathering and management of information.  The tool allowed students to easily record ideas using voice notes as well as save pictures and handwritten notes. There was positive feedback and recommendations from the test subjects reading the use of this particular platform.  Users felt the tool positively impacted their organizational skills.

In another study conducted by the Center for Teaching Excellence at the United States Military Academy, the best practices for teaching using iPads were explored.³  All the students in this setting had iPads available to them. The participants (teachers and students) deemed the devices beneficial, but the results also revealed concerns with their use such as the need to have access to the Internet, appropriate software applications for peer-to-peer and student-professor interactions, and user competency. Despite these concerns, the student interest in the subject matter seems to be one of the most important factors to consider when selecting the best teaching method. Thus, software developers have continuously tried to design applications that make the learning experience more enticing, convenient, and user-friendly.  Features common to mobile devices include capturing photos, videos of lectures, and hand-written notes as well as accessing electronic documents.

In light of the current capabilities of mobile devices as a tool to enhance instruction, I think they can be effectively used to improve learning. Studies suggest positive behavior changes when students use them for independent study under the guidance of an instructor. Social learning is also cultivated through peer-to-peer interaction with online discussion boards.

The world has evolved so much in the past two decades. Students were once dependent on every word a lecturer said, hurriedly scribbling them down in notebooks. Now we are privileged to access information with just a few taps on a screen – anywhere, anytime. By conditioning ourselves to embrace new methods of presenting and organizing information, educators can provide a gateway to limitless knowledge and possibilities.

References:

1. Kazdin A. Learning and Memory. Adapted from the Encyclopedia of Psychology [Internet]. c2000 [cited 2015 2 Feb]. 4128p.
2. Shepman, A, Rodway P, Beattie C, Lambert J. An Observational Study of Undergraduate Students’ Adoption of (mobile) Note-takingSoftware. Computers in Human Behavior. 2012 ;28 (2): 308-317.
3. Beskow D, Deb A. Increasing Learning with iPads and Social Media [Internet]. Center for Teaching Excellence, United States Military Academy, West Point, New York; 2013 [cited 2015 Feb 2]. 11p.

Applying Intersectionality to Health Professions Education

by Caitlin Leach, Pharm.D., PGY1 Community Pharmacy Practice Resident, University of Maryland School of Pharmacy

Health professions are comprised of a distinct group of people with specific domains of inquiry possessing language, methods, and skills within a focused framework.  While health professions education encourage the acquisition of an impressive depth of knowledge, they limit the scope of practitioners’ expertise.  A common solution to this issue involves promoting interprofessional education. Theoretically, this exposes learners to other areas of specialization in order to provide a more holistic approach to the task at hand.

I argue that this is not an adequate solution.

Interprofessional approaches involve temporary overlap of profession-specific frameworks to provide insight and new knowledge regarding a particular task. Unfortunately, this categorizes information outside the practitioner’s framework as the responsibility of other professions. This approach is appropriate when complex tasks have access to an interprofessional perspective. However, often in pharmacy practice, particularly in settings where patients face barriers-to-care and lack-of-access, complex cases must be solved independently of additional practitioners’ perspectives. To address this gap, pharmacy students need more exposure — not to interprofessional education, but to intersectional education.

“Intersectionality” is a sociological theory coined by Kimberlé Crenshaw in 1989.¹ The theory postulates that multiple axes of identity contribute simultaneously to a person’s experience of oppression and discrimination.  The typical manifestations of oppression - sexism, racism, classism, biphobia, homophobia, transphobia, and belief/cultural-based prejudice – cannot be examined independently because they contribute to inequality in a way that is interconnected and interrelated. Instead, intersectionality offers a holistic framework to evaluate the impact of multiple identities contributing to a person’s experience of inequality.¹ Borrowing the concept of intersectionality, it can be applied in health professional education and used to help students see the multiple determinants contributing simultaneously to a patient’s experience of health.

This figure illustrates various axes of human identity converging to a point of intersection

Image from: http://www.clydefitchreport.com/2015/01/intersectionality-going-forward/

Similar to identity, the origin of poor health is multifactorial and cannot be examined through a singular lens. Pathophysiologic and psycho-social determinants of health are interconnected and interrelated. While interprofessional approaches provide a different perspectives, from ability (physical therapy), family and income status (social work), to mental health (psychology, psychiatry), and others, it is imperative for pharmacists to be able to incorporate these health determinants when making therapeutic decisions in the absence of consulting other professions. In healthcare education, intersectionality can be applied to patient cases with the realization that health is not determined by any one thing. Through addressing psycho-social health determinants and discussing their impact on pharmacotherapeutic decision-making, intersectionality can provide students with a more complex understanding of health in order to address the needs of diverse patient populations.²

Intersectional case studies could explore a variety of questions including:

• What does lifestyle modification look like in a patient with diabetes on a fixed-income, with limited access to reliable electricity and heat, and sporadic medication adherence due to transportation issues to the pharmacy?
• In a patient with metabolic syndrome, depression, and anxiety that loses insurance coverage and can only afford half of her medications, which 5 of 10 medications are the most important in maintaining her health?
• What preventative care and screening recommendations do you make for male, female or transgender patients lacking primary care providers?

Adding layers of complexity regarding the human experience to patient cases can aid students in making the transition from the classroom to practice. Using intersectionality to address health disparities and improve health outcomes will also lead to more encompassing, holistic healthcare.³ Intersectionality provides a framework for pharmacy education that takes into account the complexity of health and human experience rather than reinforcing barriers-to-care as the responsibility of other health professionals and, in effect, perpetuates health disparities.

References:

1. Sumi C, Crenshaw K, McCall L. Toward a Field of Intersectionality Studies: Theory,Applications, and Praxis. Signs: Journal of Women in Culture & Society. Summer 2013;38(4):785-810.
2. McGibbon E, Rochester C. Applying Intersectionality & Complexity Theory to Address the Social Determinants of Women’s Health. Women's Health & Urban Life. May 2011;10(1):59-86.
3. Hankivsky O. Women’s health, men’s health, and gender and health: Implications of intersectionality. Social Science & Medicine. June 2012;74:1712-1720.

Maximizing e-Learning by Maximizing Interaction

by Ryan D’Angelo, B.S., Pharm.D., PGY1 Pharmacotherapy Resident, University of Maryland

I have always believed in welcoming technology into the classroom. Why not bring the wealth of information that exists today and increase what you can present to your students? It seems with increasing acceptance of technology and changing pedagogy techniques, the tide is turning. More and more institutions and teachers are relying on electronic learning (e-learning) to reach their learners in an effort to improve access and learning outcomes.

Most e-learning platforms use web-based systems to make information available to users anywhere in the world, on their own time, and at their own pace.^1,2 This can help decrease the amount of time lecturing to students and use in-class time to engage in active discussions. One thing to keep in mind when deciding to employ e-learning — there are a variety of implementation strategies and some are better than others. When e-learning techniques fail to engage learners, the results can be potentially worse when compared to learning outcomes observed in a traditional classroom environment.

One key in making interactive e-learning effective is providing learners with easy, fast, and instinctual access to content.^1,3,4 This can be achieved by utilizing diverse multimedia methods (audio, video, text) to deliver the content. Allowing for random access to information gives learners the ability to select content of their interest and is more likely to capture the mental attention of users.4 Allowing random access to content (rather than prescribing what content should be accessed and the sequence) requires interactivity by forcing the user to select what content they want to review, instead of forcing them to start at a particular point.^1,4,5

Interactive e-Learning Strategies

 

• Interactive Text: Terminology within text can be made interactive by providing definitions of text when the cursor is placed over the word, hyperlinks to images or video, and creating space where users can annotate within the margins of the page.
• Audio and Video: Providing video content in segments that are indexed allows for random access to different portions of the video. Many on-line continuing education courses utilize this technique by allowing viewers to pick the sections they would like to watch in the order they prefer.
• Simulation: Simulation can allow learners to put their knowledge to the test in “real-life situations”. Simulation can be created for nearly any subject including hard sciences (chemistry and physics), healthcare (patient cases), and personal interactions (psychology). While its technically more difficult to create high quality simulations, it can escalate interactivity to the next level!
• Learning by Asking (LBA): Allowing learners to assess their understanding by creating online questions and answers is an effective means of increasing interaction with the material. After the answer is selected, the correct response can be displayed to provide learners with real-time reinforcement and a detailed explanation. In addition, allowing students to propose questions related to the content can encourage learners to reflect on what they had just read, seen, or heard.
• Asynchronous Discussion Boards: Similar to learning by asking, asynchronous discussion boards can encourage critical thinking and examination of content.  Writing about the material forces learners to provide their interpretation of content. This can stimulate a discussion among students regarding the content and can inspire further inquiry.

Many studies have evaluated the impact interactivity on learning effectiveness and have found improvements in learner satisfaction, increased attention and better test performance compared to traditional classroom methods like lecturing.^1,4,5,6 This may be in part to enhanced self-directed and self-paced learning allowing students to be in a more focused and motivated mind-set when they engage with the material. One of the common limitations of these studies is that they generally evaluate only one form of interactivity. However, McLaughlin and colleagues evaluated a application of diverse set interactive tools using a multi-media website. The website was designed to provide instruction in a neurologic pharmacotherapy module — part of a required course in a doctor of pharmacy curriculum.3 The multi-media site blended video, assessment questions with immediate feedback, an interactive pop-up glossary within text, and allowed students to pose questions within each section of the text. The questions were answered by the instructor within 24 hours. This format was compared to a conventional 38-page downloadable text version of the material. When student performance on the final exam was compared, those with access to the interactive media site achieved significantly higher scores (80% vs. 74.5% p-value = 0.04). In addition, students with access to the media-site reported higher satisfaction with the module but this difference was not statistically significant. In class quiz scores were similar between the two groups (83.5% vs 84.1% p-value 0.89).

One of the major difficulties with implementing e-learning is the time it takes to develop interactive multimedia. Most faculty lack the knowledge to program the software to include these interactive approaches. Development of e-learning content requires a large investment of time, at least initially. But one thing to keep in mind is that the materials are enduring and can be updated and built on. Resources (people, equipment, and software) available through information and technology (IT) departments may also be able to assist with the initial development of e-learning materials. This can ease the time burden.

E-learning can be a useful tool to enhance learning and allow the more effective use of classroom time by using it for discussions and other “flipped classroom” activities. Building interactive multi-media requires dedicated time but when developed appropriately it enhances learning outcomes. E-learning has tremendous potential because the diversity of methods can meet the needs of students with different learning styles. Teachers should not be dissuaded from using interactive e-learning strategies because it requires us to learn new ways of delivering instruction. Rather, it is an opportunity to increase our skills to reach our learners.


References

1. Sun PC, Tsai RJ, Finger G, Chen YY, Dowming Y. What drives a successful e-learning? An empirical investigation of the critical factors influencing learner satisfaction. Computers and Education. 2008;50:1183-1202.
2. Zhang D, Nunamaker F. Powering e-learning in the new millennium: an overview of e-learning and enabling technology. Inf Sys Front. 2003;5(2): 207-18.
3. McLaughlin JE, Rhoney DH. Comparison of an interactive e-learning prepatory tool and conventional downloadable handout used within a flipped neurologic pharmacotherapy lecture. Currents in Pharmacy Teaching and Learning. 2015;7:12-19.
4. Zhang D. Interactive multimedia based e-learning: a study of effectiveness. American Journal of Distance Education. 2010;19(3):149-62.
5. Zhang D, Zhou L, Briggs RO, Nunamaker JF. Instructional video in e-learning: assessing the impact of interactive video on learning effectiveness. Information and Management. 2006;43:15-27.
6. Chumley-Jones HS, Dobbie A, Alford CL. Web-based learning: sound educational method or hype? A review of the evaluation literature. Acad Med. 2002;77:S86-93.