“In the same way we talk about evidence-based medicine, we should be making evidence-based decisions about our students and courses,” said Edward Krupat, director of the HMS Center for Evaluation and an HMS associate professor of psychology in the Department of Psychiatry at Beth Israel Deaconess Medical Center. Like a car on a maintenance schedule, faculty take their courses and clerkships for regularly planned checkups to the center, which Krupat calls “the assessment/evaluation conscience of the Medical School.” The center warehouses voluminous data—from student and peer assessments to outside consultants’ critiques—for the faculty and administration to use. And it provides consultation to faculty seeking better methods of testing students. The center also oversees the Objective Structured Clinical Exam (OSCE) at the end of the second and third years, in which students practice their examination skills on actors posing as patients.
Among a spate of recent or impending projects, the center has worked with the faculty on a new evaluation form for clerkships to render student grading less subjective. “There really had not been very precise, objective criteria for distinguishing between a high honors grade and honors, for instance,” said Krupat. The new form bases grades on identification of specific competencies achieved.
Krupat’s center and the Academy Center for Teaching and Learning are planning a workshop this fall to hone faculty skills in writing more specific feedback on student clerkship evaluation. “To say that someone is ‘hard working’ doesn’t tell the student much about him- or herself.” The two centers also have developed a new program under which the dean for medical education recognizes outstanding tutors.
The Center for Evaluation has initiated peer reviews of basic science courses and is assembling a committee of senior faculty to look at the first-year Human Body course. It has done similar reviews of clerkships for years and currently has the neurology clerkship under its spotlight, compiling assessments by faculty, outside consultants, and students while conducting on-site meetings with students and clerkship directors.
To ensure that curriculum reform is not a faith-based initiative, the center will take its measuring tape to that as well. Krupat has begun surveying current second- and third-year students—those who began their medical education under the old curriculum—to learn how well that curriculum met their needs. And he will do the same for students in the first two years of the new curriculum. All those surveyed will be followed up until their graduation and perhaps beyond.
The center has begun measuring the clinical performance of students in the Principal Clinical Experience, the newly formatted approach to single-site clerkships. The oldest PCE—the pilot integrated clerkship at Cambridge Hospital—will be the subject of an article by Krupat and the Cambridge directors in Academic Medicine. Their conclusion? “Although the numbers were small, we found that the pilot students did at least as well, if not better, on Harvard and national board performance measures of knowledge and skills. Further, the students told us that they feel well prepared.”
Newer PCEs at the Harvard-affiliated teaching hospitals—involving specialty rotations even though students get a longitudinal experience at one hospital—are still being refined, but students report that they are much better prepared by staying in one institution, enjoying the continuity with faculty and feeling that they get a great deal of support, Krupat said.
As part of the second-year reforms, the center is helping course directors design a “week of integration” at the end of each semester, during which students will be tested to see if they have made connections among the content and concepts of the different courses that they have had—the whole point of integration.
