Summary of VUSM Course and Program Evaluation
Highlighting Student Protections and Obligations
Version: September 2019
For the formal Program Evaluation Policy, please visit the VUSM Catalog.
In this Summary, the term “evaluation” is used to refer to the measurement of effectiveness of the curriculum and school programs, not individual student performance.
The purpose of evaluation at Vanderbilt University School of Medicine (VUSM) is to provide timely information that can be used to determine the effectiveness of the VUSM learning system in accomplishing its vision and to support continuous improvement of all of its components.
Vision: The VUSM Learning System is the collection of people, processes, and technology that creates a learning environment in which a range of innovative formal and informal learning opportunities are available that fully integrate learning with patient care and reflect the results of translational research in clinical care and education. It is expected that the VUSM learning system will promote the professional growth of medical students into leaders and scholars who will be able to function effectively in and contribute to the complex health care system of the 21st century.
Structure of Evaluation at VUSM
1. Curriculum improvement is a community effort. All VUSM students, staff and faculty have a role in identifying needs and implementing strategies for improvement. Each medical student must participate in every VUSM developed program evaluation survey. Medical students are invited, but not required, to participate in VUSM developed program evaluation focus groups. See the Student Responsibilities section below for specific information.
2. The Program Evaluation Team within the Office of Undergraduate Medical Education (OUME) determines evaluation strategies, collects and summarizes data, and provides interpretation. Curriculum leaders including course directors and Deans use this information to formulate and monitor organizational improvement strategies.
3. OUME provides staff support to coordinate program evaluation activities. This includes initiating evaluation activities, monitoring responses, closing evaluation activities, summarizing the data and implementing process improvements.
4. The Student Curriculum Committee (SCC) provides representatives to all evaluation committees. Student representatives gather data from colleagues as needed, provide reports and interpretations, and make recommendations for improvement. SCC leaders meet regularly with Deans and course directors to initiate rapid-cycle improvements during all required courses.
5. The Undergraduate Medical Education Committee (UMEC) is the faculty committee charged with central oversight of the curriculum. UMEC regularly reviews reports describing the performance of the system, making recommendations for improvement and monitoring the impact of changes.
6. To manage a potential survey burden on students, all surveys initiated by educators to solicit information about the performance of the VUSM learning system and/or students must be reviewed and approved by the Program Evaluation Team before distribution to students.
Types of Evaluation
1. Program evaluation examines the impact of overall curriculum and the interaction of its components and the learning environment on medical student learning. At VUSM, program evaluation activities include but are not limited to:
a. the Annual Learning System Survey (ALSS);
b. the Matriculating Student Questionnaire of the Association of American Medical Colleges (AAMC);
c. the Year 2 Questionnaire (Y2Q) of the AAMC;
d. the Graduation Questionnaire (GQ) of the AAMC;
e. the PGY-1 program director survey;
f. regularly scheduled “rapid response” meetings between the Associate Dean for Undergraduate Medical Education and members of the Student Curriculum Committee;
g. focus groups of students that examine specific issues that are called as needed; and
h. data from student assessment, which is considered in program evaluation to balance the data obtained from surveys and examine relationships between student performance and the activities, characteristics, and outcomes of the learning system.
2. Course evaluation examines the effectiveness of courses in meeting stated objectives and providing appropriate learning opportunities and assessment activities. Every required course is evaluated. There are two parts of each course evaluation. The first part contains items that are consistent across all courses, allowing for comparison across all courses. The second part contains items of interest to course directors.
3. Evaluation of educators. The performance of faculty members, residents, and fellows who teach in a course is evaluated.
Data Security
In recognition that students are providing feedback about a system in which they are also being assessed (graded), we take specific steps to protect students from any real or perceived bias related to the provision of quality improvement information. Data that is collected in evaluation activities developed by VUSM is stored on secure servers managed by the Office of Undergraduate Medical Education.
There are two levels of access to evaluation data stored in secure locations.
1. Individually identifiable data – Program Manager for Curriculum Evaluation and Student Assessment only; the Program Manager for Curriculum Evaluation and Student Assessment receives non-disclosure training and does not share identifiable data.
2. De-identified data – Senior Associate Dean for Health Sciences Education, the Associate Dean for Undergraduate Medical Education, the Director of Evaluation and members of UMEC.
Routine access to evaluation data will be managed by the Program Manager for Curriculum Evaluation and Student Assessment; non-routine access will be referred to Director of Evaluation and if necessary, Associate Dean for Undergraduate Medical Education.
Data that is collected in the Matriculating Student Questionnaire, the Year 2 Questionnaire and the Graduate Questionnaire developed by the Association for Medical Colleges (AAMC) is received by VUSM in aggregate form in PDF documents. When data that was used to produce the report is requested from AAMC for analysis purposes, de-identified data is received in delimited excel format from the AAMC that is stored on a secure OUME server.
Student Responsibilities
Each medical student must participate in every VUSM developed program evaluation survey and each assigned VUSM developed program evaluation focus group.
1. VUSM evaluation activities will use the student identification number assigned by VUSM Enrollment Services to track evaluation and assessment data throughout a student’s medical school experience. Tracking allows higher level analysis of aggregate data to support quality improvement efforts and will not be used in ways that will expose student identity. Protection of student anonymity is described in the section above, “Data Security.”
2. If a student does not participate in an assigned evaluation activity and does not obtain permission from the Associate Dean for Undergraduate Medical Education to be excused, it is considered a deficiency in the Systems-Based Practice competency. Multiple deficiencies may be reported to the Promotions Committee.
3. The participation requirement does not currently cover Educator Evaluations. Because the potential number of faculty evaluations would be very large for each student, a sampling process is being developed to reduce the number of faculty evaluations an individual student would have to complete. When the sampling process is implemented, this policy will cover evaluation of faculty as well.
4. Students may opt-out of any VUSM-developed evaluation survey and/or any assigned VUSM-developed evaluation focus group. Any student wishing opt out must request a meeting with the Associate Dean for Undergraduate Medical Education to discuss the student’s reasons for wanting to be excused from the requirement to respond to a survey or participate in a focus group. If the Associate Dean for Undergraduate Medical Education excuses a student from the requirement to respond to a survey or participate in a focus group, the Dean will inform the Program Manager for Curriculum Evaluation and Student Assessment, who will remove the student’s name from the panel for that survey or focus group.
5. Students are encouraged to contact the Associate Dean for Undergraduate Medical Education at any time to offer quality improvement suggestions or to express concerns about student protections in this process.