The Importance of Self-Assessment

by Taylor Hayes, PharmD, PGY1 Pharmacy Practice Resident, Baptist Memorial Hospital – Golden Triangle

Self-assessment is a practice that encourages students to reflect on their learning or performance so that they can identify strengths and weaknesses and make improvements. Teaching a student to effectively engage in self-assessment brings to mind the parable “If you give a man a fish, you can feed him for a day. However if you teach a man to fish, you feed him for a lifetime”.¹Teaching self-assessment helps students to become more autonomous in their learning by being able to self-identify what went right or wrong. From this, students can tailor their learning habits, strategies, and materials so that have a positive effect on their performance.

Self-assessment can come in many forms – from students scoring their own projects using a rubric, reflective assignments, and exam wrappers. Exam wrappers are designed to make students look beyond their score of the exam and take a deeper dive.  An exam wrapper asks students probing questions about the exam and the student’s preparation. Some example questions of the exam wrapper include how much time the student spent preparing for the exam, the part of the exam that the student believes they did not perform the best on, and what the student believes the teachers can do to help in their preparation for the next exam.² Having students ponder on these questions prompts self-reflection and gets them to consider ways they might better prepare for the next exam.

Self-assessment is a key element of metacognition, the mental processes where one develops awareness of the processes one uses when learning new material or problem-solving. Metacognition makes students more conscious of their thinking and how their cognitive strategies help them succeed. Being self-aware of one’s performance helps students take ownership of their learning.^3,4

However, self-assessment is often subjective and students often struggle with identifying the areas where they need to make improvements. These students are unconscious in their shortcomings and may not realize the need for adjustments (or how to make adjustments). A preceptor once asked for me to place myself into a category – unconsciously incompetent, consciously incompetent, consciously competent, or unconsciously competent. These categories are known as the four stages of competency. When you are unconsciously incompetent, you are unaware of a knowledge gap. When you are consciously incompetent, you are aware of a knowledge gap and recognize the importance of filling this gap. For those who are consciously competent, they know the information but they need to put forth conscious effort to recall the information or perform the task. Finally, unconsciously competent refers to knowing the information and being able to easily perform the skill without much conscious effort or thought.⁶ It is hard for students that are unconsciously incompetent to be aware of what they do not know.  Thus, continually practicing self-assessment can help the learner develop the skills needed to identify areas that need improvement. Self-assessment can, at first, be facilitated by teachers giving students feedback on their performance and then asking the students to reflect on how they think they performed (or vice versa). This helps students gain a sense of direction on the things they can improve, while also prompting them to independently think about how they can improve.

Source: The Four Stages of Competence [Internet]. Timothy S. Bates. 2014. Available from: https://tsbates.com/blog/four-stages-competence/

One study looked at the impact of self-assessment on academic performance in students. Eighty-nine students took a test and then self-assessed their performance by grading their exams under the supervision of a teacher. Following this, the teachers also graded the test and provided feedback to the students. A second test was given on the same topic and was graded solely by the teachers. From this, the two scores from both the student-graded test and the teacher-graded test were then calculated. The study found that 74% of students scored higher on the second test. This helped to show that after the students had self-assessed their own performance, they were able to identify the areas of shortcomings in order to improve on them for the next exam.

This same study, however, also showed some of the pitfalls that may occur with self-assessments. An analysis of the first student-graded test was performed to assess the difference in scoring between the student’s score versus the teacher’s score.  The majority of the students (74%) gave themselves significantly higher scores than what the teacher had given them. This highlights that self-assessment is subjective, and that being able to accurately assess one’s performance is difficult for some students. Ways to combat this include giving students a rubric to follow, showing an example of good performance and comparing it to a not-so-good performance, or grading a paper together as a group. In the study, the student’s and teachers’ perceptions about the self-assessment process were gathered using questionnaires. The teachers believed that having the students perform the self-assessment was effective in promoting student self-learning. The students found the process beneficial but time-consuming. While as teachers we can never give back time, we can reiterate the importance of the task as a worthwhile investment of time. Reminding the students that self-assessment will help them in future learning and performances will help the student understand why the self-assessment activity is being done. The authors of the study concluded that self-assessment can serve to increase the motivation for students to both want to perform better and help develop self-directed learning skills.⁶

It might be beneficial for students to develop a list of their “successes” and “failures” in order to reflect on them. When were times they were disappointed in their performance, and how could they avoid these same disappointments from happening in the future? When was a time they were proud of their work, and what were the steps they took in order for this to happen? If other people have provided feedback on the student’s performance, it might be beneficial for them to reflect on this in their self-assessment as well. The student needs to really reflect and narrate on their experience to improve from it, rather than just regurgitate a list. Of course, it’s important to remember when writing a self-assessment that there is always room for improvement. Self-assessment isn’t remediation, only for those who are performing poorly.  Even when a student is performing well, there are still things to learn from that experience that can benefit the student in future exams and experiences.^7,8

References:

1. Loveless B. Helping students thrive by using self-assessment [Internet]. Education Corner.
2. Lovett M. Exam Wrappers [Internet]. Eberly Center - Carnegie Mellon University.
3. Mcdaniel R. Metacognition [Internet]. Vanderbilt University. 1970.
4. Burch N. The Four Stages of Competence [Internet]. Mercer County Community College.
5. Assessment Resource Centre [Internet]. Centre of Enhancement for Teaching and Learning.
6. Hertzberg K. How to Write a Self-Evaluation [Internet]. Grammarly; 2020.
7. How to write a performance evaluation self-assessment [Internet]. Business News Daily.

Self-Assessment versus Self-Monitoring

by Sarah A. Pierce, Pharm.D., PGY1 Pharmacy Practice Resident VA Maryland Health Care System

Most pharmacy residents are familiar with the use of ResiTrak^TM to complete self-evaluations, an arduous process made more difficult by having to recall performance over a long period of time.  Is this method of self-evaluation effective?

In both pharmacy education and residency training, self-assessment is a commonly utilized tool intended to encourage a learner to evaluate his or her performance, identify strengths and weaknesses, and note areas for self-directed learning and growth. In its accreditation standards for Doctor of Pharmacy programs, the Accreditation Council for Pharmacy Education (ACPE) discusses the importance of self-assessment for students, faculty, and staff.¹  The theme of self-assessment and self-directed learning persists into post-graduate residency training. The American Society of Health-System Pharmacists (ASHP) includes “resident self-assessment of their performance” as a requirement in their accreditation standards for PGY1 pharmacy residency programs.²   The ASHP standards require “summative evaluations” at the end of each learning experience (aka “rotation”) and  encourages optional spontaneous “snapshot” self-evaluations too.

Implicit in these requirements is an assumption is that self-assessment is valuable and accurately reflects a person’s strengths and weaknesses. However, according to Eva and Regehr, there is substantial body of literature which suggests that learners often cannot accurately self-assess their strengths and weaknesses and that self-assessment correlates poorly with actual performance.³  However, there may be a distinction between self-assessment and self-monitoring: “self-assessment as a cumulative evaluation of overall performance, and self-assessment as a process of self-monitoring performance in the moment” [emphasis added].³

Eva and Regehr discuss the results of two studies which explored self-monitoring and self-assessment.^3,4 In each study, participants answered sixty trivia questions divided into six categories.  Participants were asked to evaluate their performance at different times during the testing. To measure “self-assessment” (that is, a cumulative evaluation of overall performance), the researchers had participants predict their overall score for each category both before and after completing all ten trivia questions in that category. To measure “self-monitoring” (that is, an evaluation of performance while in the moment), the researchers had participants rate their confidence in a given answer immediately after answering the question. The results showed that a “self-monitoring” measure demonstrated a higher correlation with actual performance when compared to the cumulative “self-assessment” measure. 

They hypothesized a potential explanation for these findings.   Self-monitoring likely requires a “fundamentally different cognitive process” than self-assessment. With self-monitoring, learners have many inputs and sources of information at their disposal to predict potential success or failure on a moment-to-moment basis. However, with self-assessment, the learner must rely on memory to aggregate information of multiple past events in order to determine the overall success or failure.³ The concept of self-monitoring was replicated and expanded in work by McConnell and colleagues.⁴

What are the potential implications of these findings? In the Educational Theory and Practice course, the idea of self-directed learning was introduced.⁵   Self-assessment is a tool used to facilitate self-directed learning. However, if self-assessment is not as accurate as one may hope, then perhaps self-assessment is not the best tool to facilitate self-directed learning. I would argue that more attention should be directed to self-monitoring on a moment-to-moment basis, rather than on cumulative self-assessments.  Through  self-monitoring, individuals would develop a more accurate picture of their abilities and this could lead to more focused self-directed learning needs. As self-assessment is by far the most common self-evaluation tool used in pharmacy education and residency training today, new, creative ideas to transition to self-monitoring are needed.

Self-monitoring could be implemented in pharmacy education and residency training in several ways.  In pharmacy school, an early and consistent emphasis on self-monitoring could prove to be more effective than intermittent, reflective self-assessments.  By asking students to identify their strengths and weaknesses in real-time, this may motivate them towards focused self-directed learning.   For example, students taking an exam on the pathophysiology of diabetes, the pharmacology of diabetes medications, and diabetes management could be required to rate their confidence on each test question. After the exam, students could receive a report with their self-monitoring responses and a breakdown of their actual performance in each domain (e.g. pathophysiology, pharmacology, and patient management). In this way, students would be better at identifying areas they struggled with, gradually improve the accuracy of self-monitor their performance, and note areas that require further study. Regularly repeating this process may help students develop stronger self-monitoring skills and help them become independent practitioners after graduation.

A similar argument could be made for pharmacy residency programs. I believe “snapshot” evaluations should be used more frequently.  During my internal medicine rotation, I was asked to complete a snapshot evaluation related to my data gathering skills and treatment plan for a specific patient. This was much more focused than the summative self-evaluation done at the end of my rotation whereby I had to assess my overall performance related to several different goals and objectives. I believe I gained more insight into my strengths and weaknesses with the snapshot evaluation.  I was able critical examine my performance on narrow set of tests “in the moment” rather than having to search my memory for past events that related to my performance.

While self-assessment is certainly a necessary component of pharmacy education and helps facilitate self-directed learning, I believe there should be a greater emphasis on “real-time” self-monitoring.   Self-monitoring should be taught and required in Doctor of Pharmacy programs and frequent self-monitoring “snap shots” should be a mandatory component of pharmacy residency training.

References

1.  Accreditation Council for Pharmacy Education. Accreditation standards and guidelines for the professional program in pharmacy leading tothe doctor of pharmacy degree. Chicago: Accreditation Council for Pharmacy Education; 2011.  97 p.  [cited 2012 Oct 8]

2.  ASHP Commission on Credentialing. ASHP accreditation standard for postgraduate year one (PGY1) pharmacy residency programs. Bethesda (MD): American Society of Health-System Pharmacists; 2005. 23 p.  [cited 2012 Oct 8]

3.  Eva KW, Regehr G. Exploring the divergence between self-assessment and self-monitoring. Adv Health Sci Educ Theory Pract 2010;16(3):311-29. [cited 2012 Oct 8]

4.  McConnell MM, Regehr G, Wood TJ, Eva KW. Self-monitoringand its relationship to medical knowledge. Adv Health Sci Educ Theory Pract 2011;17(3):311-23. [cited 2012 Oct 8]

5.  Kaufman DM. Applyingeducational theory in practice. BMJ 2003; 326: 213-6. [cited 2012 Oct 8]

Peer Assessment: More Than Busy Work

By Anh Tran, Pharm.D., PGY1 Pharmacy Practice Resident, Medstar Union Memorial Hospital

Take a moment and think about a time when you were in high school or college and you were asked to assess your peers on their work.  Or vice versa.  I remember a time when I had just turned in a paper in an undergraduate English class.  The professor then informed us that we would be grading each other’s papers!  The first thought that went through my mind was, “This is just busy work!”  Actually, peer assessment can be a very effective learning tool. 

Peer assessment is the process whereby students receive a critical evaluation and feedback of their work from a similarly experienced individual, peer, or colleague.  This practice is commonly used in various settings, including pharmacy education.  For example, peer assessment can be used to evaluate a patient counseling session conducted by a student pharmacist or a pharmacotherapy presentation by a pharmacy resident.  Peer assessment plays a vital role in a pharmacist’s professional development, whether during school, experiential rotations, postgraduate training, or career. Furthermore, the practice of peer assessment promotes active learning, group work, and complex problem solving.

In addition to promoting these great aspects of learning, peer assessment has other distinct advantages.  Peer assessment enables faster and more detailed feedback.¹  How many times have you turned in an assignment and waited for weeks for the professor to grade it and provide feedback?  Most likely, you forgot your thought process through that assignment and thus, the feedback is no longer useful to you.  Instead, having peers grade each other’s assignments provides more timely feedback, which is more useful because the assignment and the students’ thoughts are still fresh in their minds.  In addition, since assignments are being reviewed simultaneously by multiple graders, there is the potential for more detailed and in-depth feedback.

Peer assessment might have some advantages from a teaching and learning point of view, but what are students’ attitudes towards it?  In a study conducted by Wu and colleagues, 91.9% of PharmD students surveyed believed that peer assessment is a skill that they will use in their pharmacy career.  In terms of student-to-student peer evaluation, 80% of students were comfortable providing an honest assessment to their partner and 95.7% of students were comfortable receiving it. Furthermore, only 34.4% of the students believed that the assessment of students is solely the responsibility of faculty and not students.²  In another study, Basheti and colleagues demonstrated that anonymous peer feedback in a pharmacy course is an effective means of providing constructive feedback on performance.  The study found that 78.1% of students felt that their participation in the peer assessment process helped them to deepen their understanding of the course content and 78% of students would endorse the use of this practice in other courses.³  Thus, students felt comfortable with peer assessment and perceived it as a valuable tool in their education.

Peer assessment is consistent with the principles of andragogy.  In other words, peer assessment takes evaluation from “teacher-driven” to “learner-driven”.  By taking assessment out of the teacher’s hands, students have yet another learning opportunity.¹  Peer assessment can lead to a deeper understanding of a topic by evaluating the work of others.³  For example, when I evaluated the English paper of an undergraduate peer, I was pleasantly surprised what I learned just from reading it!  We had written on the same topic, but we had different views and opinions.  By practicing peer assessment, students can discover other perspectives on a topic which can broaden their understanding.

Finally, peer assessment fosters metacognition, which is a knowledge or awareness of one’s own learning processes.¹  By participating in peer assessment, students are in a better position to understand the grading criteria.  Thus, they can then internalize this understanding and apply it to their future work and to improve their own performance.  For example, in a practice patient counseling session, a pharmacy student grading a peer would develop a better understanding of best practices and can then apply these criteria to his/her future counseling sessions.

While peer assessment has many great qualities, there are some concerns.  Can peer assessment truly serve as a substitute for the teacher’s assessment?  Are these assessments valid?  Falchikov and colleagues attempted to answer these questions by performing a meta-analysis comparing peer and teacher assessments in higher education.  The meta-analysis showed a mean correlation over all the studies to be r = 0.69, indicating reasonably good agreement between peer and teacher assessments.⁴  Similarly, Sadler and colleagues conducted a study to determine the agreement between the grades given by a teacher and those given by a peer.  This study showed that peer-grades were highly correlated with teacher grades (r =905)!¹

Assessment and evaluation are essential components of instructional design and peer assessment is a good way of engaging students in the classroom. Studies have identified ways to implement peer assessment by educators.   It’s important to provide training on the evaluation process to students and to provide clear criteria for peer feedback in order to avoid superficial comments.  In addition, professors should blind the reviews in order to reduce bias, since friendships may affect the accuracy of peer assessment.¹

When educators implement structured, unbiased approachs to peer assessment, it can play an exceptional role.  Not only is it an effective learning tool, but peer assessment can foster team work, active learning, and metacognition.  Students realize the importance of peer assessment and are comfortable participating in such a process.  So the next time your professor announces that you’ll be grading your peers, embrace it and further your learning!

References

1.   Sadler PM, Good E. The impact of self- and peer-grading on student learning. Educational Assessment. 2006; 11(1):1-31.

2.   Wu K, Davison L, Sheehan A.  Pharmacy students’ perceptions of and attitudes towards peer assessment within a drug literature evaluation course. Am J Pharm Educ. 2012; 76(4):62.

3.   Basheti IA, Ryan G, Woulfe J, Bartimote-Aufflick K. Anonymous peer assessment of medication management reviews. Am J Pharm Educ. 2010; 74(5):77.  

4.   Falchikov N, Goldfinch J. Student peer assessment in higher education: a meta-analysis comparing peer and teacher marks. Review of Educational Research. 2000; 70(3):287-322.

360-Degree Feedback

by Andrea Passarelli, Pharm.D., PGY1 Pharmacy Practice Resident, The Johns Hopkins Hospital

Preceptors play an integral role in the development of pharmacists.  Most schools of pharmacy provide formalized training and education to their preceptors in order to help them master effective precepting techniques, but literature surrounding this topic is scarce.  Preceptors should use a variety of teaching strategies throughout a rotation and these should be tailored to the learner’s stage or professional development.¹  As learners grow in their ability and confidence, they should be performing more tasks independently, such as medication reconciliation, discharge counseling, and "rounding" with the medical team.  In these situations, the preceptor might use an assessment technique known as 360-degree feedback to evaluate the student’s performance.  The 360-degree feedback technique generally involves the preceptor asking other healthcare professionals as well as patients about the student’s performance.  This evaluation technique is gaining popularity in healthcare and other sectors.  I wanted to research this technique further so that I could use this assessment strategy during my career.

One paper published by Joshi et al. in Academic Medicine described the successful implementation of 360-degree feedback for obstetrics and gynecology residents at Monmouth Medical Center.² Residents were assessed on their interpersonal and communication skills by nurses, faculty members, allied health professional staff, medical students, patients, and co-residents.  In addition, each resident completed a self-assessment.  The researchers found good correlation between evaluations within each group of evaluators as well reasonably strong agreement among evaluators regarding each resident's rank among the peer group. Interestingly, there was a negative correlation between the rankings by faculty, staff, and medical students with the rankings given by peers.  The highest-rated residents (based on faculty, staff, and student evaluations) received low marks from their co-residents and vice versa.  This may perhaps be due to some perceived competition or a desire to “get ahead” by rating high-achieving residents poorly.  On self-assessment, junior residents typically rated themselves highly while senior residents rated themselves average or low.  This may have been because senior residents were setting higher standards for themselves or have increased self-awareness later in the curriculum. A potential advantage of 360-degree feedback employed at Monmouth Medical Center: evaluators were eager participants because their feedback was anonymous. The 360-feedback technique has been widely described in medical education journals, but has been most often been used in residency training rather than student education.^3-6  360-degree feedback can be an extremely effective tool for a preceptor, as learners will often communicate differently in the presence of a preceptor and when independently communicating with peers, staff (who may be perceived to be at different social ranks within the organization), and patients.

When I was a student I had one experience with 360-degree feedback during my first acute care rotation.  On the last day of the rotation, my preceptor allowed me to round independently with the team.  After the team finished rounding, my preceptor asked the medical interns, attending physicians, and nurse case-manager about my performance.  In subsequent advanced practice experiences, both as a student and resident, preceptors have occasionally sought feedback from physicians regarding my performance.  But most based my evaluation solely on their direct observations.

While my experience with the 360-degree technique from the perspective of a learner is limited, I believe there are some important points that should be considered prior to implementation.  First, it is important to inform the learner that this technique will be used.  Although the goal is to assess the learner’s performance without influence from the preceptor, learners might be taken aback if their performance is discussed with other individuals without their knowledge.  Further, it is important for preceptors to remember that in order to effectively implement this technique they must ask a breadth of individuals (with different roles) to participate in the process.  This was done during my first rotation, but not in subsequent rotations as preceptors only asked physicians for their input.  I have never had a preceptor ask one of my patients about their perception of my abilities.  And with the notable exception of my first rotation, my preceptors haven’t asked medical interns, medical students, or nurses for feedback.

360-degree feedback appears to be an incredibly valuable assessment tool that can enhance the quality of evaluations provided to both pharmacy students and residents.  This technique allows preceptors to more accurately assess a learner’s communication skills, especially in the absence of preceptor supervision.  Research has shown that this is an effective and accurate evaluation technique when used in medical residency training, but its use has not (yet) been described in the pharmacy education literature.  Preceptors utilizing this technique should be familiar with its fundamentals, and should ask individuals in multiple roles, including patients, for feedback regarding the learner’s performance.  When used appropriately, 360-degree feedback allows the preceptor a unique opportunity to obtain a complete picture of the learner’s strengths and can help them identify areas for improvement.

References

1.    McDonough RP, Bennett MS. Improving communication skills of pharmacy students through effective precepting. Am J Pharm Educ. 2006 ;70(3): Article 58.

2.    Joshi R, Ling FW, Jaeger J. Assessment of a 360-degree instrument to evaluate residents’ competency ininterpersonal and communication skills. Acad Med 2004;79:458-63.

3.    Meng L, Metro DG, Patel RM.Evaluation professionalism and interpersonal and communication skills:implementing a 360-degree evaluation instrument in an anesthesiology residencyprogram. J Grad Med Educ 2009;1 (2):216-20.

4.    Wood J, Collins J, Burnside ES et al. Patient, faculty, and self-assessment of radiology resident performance: a360-degree method of measuring professionalism and interpersonal/communicationskills. Acad Radiol 2004;11(8):931-9.

5.    Sorg JC, Wilson RD, Perzynski AT et al. Simplifying the 360-degree peer evaluation in a physical medicine andrehabilitation residency program. Am J Phys Med Rehabil 2012; 91(9):797-803.

6.    Chandler N, Henderson G, Park B et al. Use of a 360-degree evaluation in the outpatient setting: the usefulness ofnurse, faculty, patient/family, and resident self-evaluation. J Grad Med Educ 2010;2(3):430-4.