Universal Student Ratings of Instruction (USRI) Review
The University of Calgary has embarked on campus-wide consultations around the current USRI questionnaire and process. In November 2018, a working group of the GFC Teaching and Learning Committee (TLC) was struck with a responsibility to review and provide recommendations on the administrative processes, data collection, and communication and education plans currently in place with the USRI.
The group began its work in December 2018 and completed a literature review of student ratings of instruction in summer 2019. Wide-spread and thorough consultations began in February 2020 and will continue throughout the summer and fall of 2020. This includes gathering input from students and academic staff from across campus within all units. Facilitated consultations are supported by educational development consultants within the Taylor Institute.
USRI Questionnaire and Report Changes
In the fall of 2019, the USRI working group recognized that an interim change to the current USRI would help address some of the pressing issues with the questionnaire and reports. These changes are intended to improve the experience and use of the current USRI questionnaire, while campus-wide consultations and subsequent recommendations on what we want in a new form, take place. These recommendations were discussed at Deans’ Council, Student Legislative Council, Teaching and Learning Committee of GFC, GFC Executive Committee and approved at GFC in December 2019.
Remove the overall instruction question (question #2.1) from the USRI questionnaire and report
This affects both current and historical reports
Rationale: It is not clear what this question is measuring and the rating score for question #2.1 is commonly lower than the ratings for Qs 2-12. A recent review of the question #2.1 rating scores for courses across the institution found this was the case for two-thirds of UCalgary courses. This highlights issues around the validity of this question, leading to the recommendation to remove it from the current USRI was made.
Remove rating comparisons
Rationale: Comparing different people and courses can be misleading due to factors that can influence ratings and students’ perceptions of their experience that are not related to the quality of teaching and learning. Removal of the comparators helps focus the use of USRI on the course being rated, which is the intended use of student ratings.
Replace mean scores with mode
Rationale: Mean scores can be affected by outlier ratings and may not be representative of the most frequent response selected by students. Reporting the mode helps illuminate the most frequent response for each question and help focus attention on the distribution of scores across the questions and the trends of responses.
Proposed Consultation Process/Timeline
Feb. to Aug. 2020
USRI consultations and focus groups occurred between February and August with all faculties, undergraduate and graduate students, and other campus groups in face-to-face and online sessions.
Sept. to Dec. 2020
Analyze collected data and complete recommendation report based on responses from the academic community.
Jan. to April 2021
Present recommendation report for a revised USRI to GFC through the UCalgary governance process for feedback and discussion.
Working Group Membership:
- Leslie Reid, vice-provost (teaching and learning), co-chair;
- Andrew Estefan, Faculty of Nursing, co-chair;
- Dawn Johnston, Faculty of Arts;
- Wendy Benoit, Faculty of Science;
- Barb Brown, Werklund School of Education;
- VP Academic, Students’ Union;
- VP Academic, Graduate Students; Association;
- Vacant, TUCFA representative;
- Jacqueline Lambert, Office of Institutional Analysis;
- Natasha Kenny, Taylor Institute for Teaching and Learning.