USRI Report and Recommendations (1998)

  

Changes implemented after publication of the following document includes:

*Fall 2010, USRI course evaluation instrument no longer collects student id numbers* 

Table of Contents:

• Introduction
• Background 
• Implementation Task Force
• Costs
• Recommendations
• Purpose and Expectations 
• Form of the Instrument
• Control
• Administration
• Reporting
• Publication
• Guidelines for Instructor Feedback
• Other Use of the Data
• Implentation
• Future Testing
• Archiving 
• Summary
   

1998-05-06

Introduction

The history of discussions at General Faculties Council (GFC) regarding Student Ratings of Instruction is a lengthy one, going back at least to 1992 (see Chronology of GFC Decisions, Student Ratings of Instruction, 1997-04-24 in Appendix #1).  This report will not attempt to provide all the details of that history.  Rather, it will emphasize those decisions which are pertinent to the implementation of a Universal Student Ratings of Instruction Instrument (hereinafter referred to as the Ratings Instrument).  In addition the report will also provide pertinent details of the work of the Academic Program Committee's (APC) Implementation Task Force and an estimate of costs for administering the Ratings Instrument annually.
 

  ( Top / Bottom of  Report )

Background 

At its 369th meeting (1992-03-12) GFC approved student evaluations as "one factor on which the evaluation of teaching shall be based," and asserted that "student evaluations shall be required of all academic appointees."  Subsequently, at its 389th meeting (1992-03-27) the GFC Executive Committee established the Task Force on Teaching Effectiveness, Evaluations and Procedures.  Part of its mandate was to review logistical issues involved with student evaluation forms and departmental procedures on evaluation of teaching.

The report of the Task Force was presented to GFC at its 395th meeting (1994-06-16).  GFC approved a set of both General Principles as well as Operating Principles.  Two of the General Principles are of significance for this report: 1.) that there should be regular and systematic evaluation and 2.) that it is acknowledged that "students are an essential source of insight into the effectiveness of educators."  The Operating Principles directed the VP(A) to work with Deans to develop a systematic set of procedures for administering student teaching evaluations and directed Deans to develop written plans for evaluating teaching in their Faculties.  To facilitate this process the VP(A) sent a proposal to all Deans (August 24, 1994) describing a set of five principles upon which the process of evaluation would be based:

 1. evaluation of teaching effectiveness will be multi-faceted;
 2.  evaluation of teaching effectiveness will be used to foster excellence in teaching;
 3.  the purpose of evaluation shall be to provide formative and summative evaluation for the improvement of teaching and learning;
 4. the evaluation process will identify strengths and areas in need of improvement in instruction;
 5.  evaluation will be an essential element in promotion and tenure decisions.

The proposal also outlined a multi-faceted process consisting of three elements.

 1.  a mandatory universal set of questions to be administered every time a course is taught by all instructors;
 2.  the option available for each Faculty or department to include additional questions;
 3.  the inclusion from time to time of another method of evaluation   chosen by the Faculty and approved by the VP(A).

The response from the Deans was encouraging and indicated that the idea of a multi-faceted process was acceptable.  A committee was struck to develop and refine a questionnaire.  It included representatives from the Office of the VP(A), TUCFA, Human Resources, the Students' Union, and the Associate VP(Student Affairs).  The committee consulted with students, University Computing Services, the Teaching Development Office, the Freedom of Information and Privacy Commissioner's Office and national experts.  Universal questionnaires from other Canadian universities were also consulted.

As a result of the consultations, the following points were made about the process:

1.  The questions would be constructed in order to have students rate   instruction.  It was felt that this is where students are competent and can contribute the most.
2.  The questions would be similar in content to the questions appearing in the annual Student Exit Survey.  This would help validate responses in both processes.  It would also provide Deans with a closer look at student satisfaction, helping them to understand better the provincial KPI Student Satisfaction data collected on each Faculty.
3. Faculty members would receive rating reports which would include means, medians, standard deviations, comparisons to Departmental averages, Faculty averages and total University  averages.  Over time, trend data could be provided.
4.  For the first year of rating students would not be asked to identify themselves.  In year two identification would be an option.
5.  Students would be asked to contribute to the process of administering and collecting the ratings.
6.  Ratings would be collected on optically scanned paper or in the most cost-effective way.
7. Deans requested that the responses not be made public but be made available to the Department Heads for inclusion in the FPC process.

The result was an Instrument composed of fifteen questions which was tested by the Department of Chemistry and by several classes in the Faculty of Education throughout 1995.  Arrangements were made to increase the number of pilot Faculties for 1995/96 when the Students' Union brought forward a proposal to make the ratings public.  Further pilots were halted until this issue was resolved. The GFC Executive Committee at its 434th meeting (1996-10-01) directed the Appointment, Promotion and Dismissal Committee (APD) to review the issue of student ratings of instruction including the possibility of a universal instrument and the release of the results of ratings.  The APD Committee reported to the GFC Executive Committee at its 437th meeting (1997-01-07) that consensus could not be reached and recommended that the issues be debated on the GFC floor.  The Executive Committee recommended that GFC approve a universal instrument for student ratings of instruction and release of the results of such ratings.  At its 419th meeting (1997-01-23) GFC approved the use of a universal core set of questions for student ratings of instruction and forwarded the matter of revising the core set of questions to the Academic Program Committee (APC).  GFC postponed the debate on publishing the results from the student ratings of instruction until receipt of recommendations on the core set of questions from APC.

A subcommittee of APC met to revise the core set of questions working from the original fifteen question instrument.  The subcommittee revised these questions and reduced the number to twelve.  The twelve questions were presented to APC at its 127th meeting (1997-02-04).  APC made further revisions to the questions and recommended the revised set to the GFC Executive Committee for approval.  At its 439th meeting (1997-02-04) the GFC Executive Committee recommended that GFC approve the twelve questions as the universal core set of questions for student ratings of instruction.  At its 420th meeting (1997-02-20) GFC debated and revised the questions and approved the twelve revised questions for purposes of piloting (for the twelve questions see Appendix #2).

The matter of the release of results from the student ratings of instruction was debated by the GFC at its 421st meeting (1997-03-20).  GFC approved in principle the release of results from the student ratings of instruction subject to satisfactory pilot testing and a GFC approved implementation plan which addresses satisfactorily alternatives regarding matters such as release of information to students, resource needs and costs and impact on faculty and students.

  ( Top / Bottom of Report )

Implementation Task Force

APC appointed a Task Force to pilot the items and to develop an implementation plan.  Members of the Task Force were:

 T. Barnfather (UCS), replaced by Jeremy Mortis (FIS)
 A. Colijn (TUCFA), replaced by A. Stalker
 P. Galbraith (SLC), replaced by Adrienne Miller
 D. Kline (Psych), co-chair
 R. Neufeldt (RELS), co-chair
 B.A. Samuels (Director of Planning)
 R. Schulz (LIDS)
 R.E. Woodrow (APC)
 M. Wylie (Deans' Council Executive)

 The Task Force has met weekly since April 4, 1997 to deal with the appropriate process for piloting the core set of items (a total of 50 meetings), to address the implementation issues mentioned in the GFC motion at its 421st meeting, and to begin the process of developing recommendations for both the implementation of a Ratings Instrument and the publication of results.  At its initial meetings the Task Force established a set of principles and identified the tasks that lay ahead.  Since then the focus of the work of the Task Force has been the development of an appropriate testing process, and the development of recommendations for the implementation of the Ratings Instrument and the publication for students of the information derived from the Ratings Instrument.  Throughout the business of discussing and developing recommendations the Task Force has tried to be sensitive to the concerns raised in the debates over the universal instrument, particularly issues such as impact, privacy and confidentiality.

At the 3rd meeting, April, 23, 1997, the Task Force established a Psychometric Properties Subcommittee (PPS) to deal with issues related to piloting.  Members of this committee were:

 R. Aggarwala (Math & Stats)
 W. Chin (Management)
 D. Kline (Psych), Chair
 T. Rogers (Psych)
 J. Wallace (Sociology)

On the recommendations of the PPS the Task Force decided on a two phase piloting process. The first involved student interviews and a Faculty questionnaire to identify important instructional issues and to refine the rating items.  The second involved in-class psychometric evaluation of the Ratings Instrument across a broadly representative range of courses and types of classes.  With ongoing input from the Task Force, the PPS developed written specifications for the two phases, including protocols for the Phase 1 student interview and Faculty questionnaire analysis of the instrument, as well the specifications for the Phase 2 evaluation of the Ratings Instrument by both faculty and students.  These specifications formed the basis for a solicited research proposal from Creating Organizational Excellence (COE), a research unit in the Faculty of Social Sciences.

Phase 1 was completed during the summer term 1997.  This included both student interviews and a Faculty questionnaire.  Some 30 student interviews were carried out in July, 1997.  Students were selected randomly from courses taught in the summer session.  Course sampling was done to ensure representation of Faculties, Departments and course level (200, 300, 400).   For the Faculty questionnaire a stratified random sampling procedure was used to ensure adequate Departmental and Faculty representation.  Three hundred questionnaires were sent out.  Of the initial three hundred 83 were returned, a response rate of 28%.  While the response rate is lower than the Task Force would like, it is a respectable rate representing a good range of faculty opinions.  Revisions to the rating items were recommended by COE on the basis of the results of the interviews and questionnaires.  The recommended revisions were brought by COE to the Task Force and were refined further in discussions between COE and the Task Force.  The COE Report recommended replacing the original item #4 (the instruction enabled effective learning) with two items which were intended to capture the elements which respondents thought the original item #4 was intended to address.  These items are "I learned a lot from this course" (strongly recommended by faculty responses) and "The support materials used in this course helped me to learn."  The Report recommended collapsing item #9 with item #10 since responses indicated that the two items overlapped.  The Report also recommended dropping the phrase "considering class size" on the grounds that answers are normally given with class size in mind (see Appendix #2, COE Phase 1 Report, Summary of Recommendations).  The COE Report on the first phase along with the suggested revisions was presented to APC for discussion and advice on September 16, 1997.  The Task Force reported to APC that there would likely be further minor revisions to the items based on the feedback from the second phase of testing.

Phase 2 was carried out by COE in two parts.  To ensure a representative sample of Faculties, Departments and course level, a stratified random sampling of classes, students and faculty was used (it should be noted that graduate courses are not represented in the sampling nor is the Faculty of Continuing Education).  Part 1 provided a qualitative process in which an attempt was made to receive feedback from faculty and students on the meaning and clarity of the twelve item scale (as revised based on the results of Phase 1), the rating scale for the global question on instruction, aspects of instruction not captured by the twelve items, and the instructions to students on the rating form.  Part 2 was designed to examine the psychometric properties of the twelve revised items. The revised core set of items was piloted in the Fall Term, 1997 in 132 classes (5,603 students).   The recommendations resulting from the second phase of the piloting process have resulted in minor changes to the previously revised Ratings Instrument.
Specifically, for the sake of clarity minor changes were made to the wording of some of the items.  The anchor in item #1 was changed from "terrible" to "unacceptable" (see Appendix #3, COE Phase 2 Report, Summary of Recommendations).

The Task Force and COE are of the view that a-more-than-adequate range of Faculties, Departments, classes, faculty and students was represented in the piloting process.  The conclusion of COE, supported by the data is that the Ratings Instrument is a very good instrument for the intended purpose, that is, the universal ratings of instruction at the University of Calgary.  It should also be pointed out that instruments of this sort have been used and tested in university settings for years.  As a result there is a considerable fund of literature pertaining to the use of such instruments.  For the most part the literature supports, with qualifications, the use of such instruments for the evaluation of instruction.   Nonetheless, concerns have been and continue to be raised about the use of such instruments.  With the help of COE, and submissions made by interested Faculty members, the Task Force attempted to isolate  commonly raised concerns both about the use of such instruments and the publication of the results of student ratings of instruction.  The Task Force commissioned a literature search to be carried out in relation to these concerns.  This is included in Appendix #4.  In addition to the literature search, the Task Force also sought advice on practices pertaining at other universities in Canada.  The information on other universities is found in Appendix #5.  To supplement the information derived from the Phase 2 piloting of the Ratings Instrument, during the Fall Term, 1997 the Task Force investigated the applicability of the Ratings Instrument for non-traditional courses (e.g. distance education, practica, labs, tutorials).  This was done through Task Force interviews with  P. Sokol (Medicine), G. Rogers (Social Work), A. Marini (Education), C. Lucy (Chemistry), E. Enns (Germanic, Slavic and East Asian Studies), J. Rankin (Nursing) and J. Chua (Management).  The Task Force also received several independent submissions from individual Faculty members.  The interviews and faculty submissions indicated that with minor editing many of the items would be applicable to courses using non-traditional forms of delivery.  It was also clear that it would be useful to add a N/A column to the items to allow for responses for items which could not be edited sufficiently to make them useful for non-traditional forms of delivery.

In addition to seeking advice concerning non-traditional forms of course delivery, the Task Force also sought advice from the University's Legal Counsel and the Office of the Privacy Commissioner, with respect to potential legal issues arising from the Freedom of Information and Privacy Act.  The Task Force has received some advice from the University's Legal Counsel but very little from the Office of the Privacy Commissioner.   To our knowledge the only ruling available on the use of student ratings instrument is the one issued for Simon Fraser University.  The ruling seems to suggest that a course evaluation resulting in a computer generated statistical report can be published without infringing on the protection of privacy.  Evaluations of instructors may, however, be problematic.  The University of British Columbia seems to have abandoned a Senate approved universal survey, seemingly in response to the enactment of the Freedom of Information legislation.  The University of Victoria has initiated discussions between administration and faculty about the use of surveys for evaluative purposes.  The picture across Canada with respect to current practice is mixed (see Appendix #5).  Practice ranges from the use of universal surveys and student access to the results of the surveys, to the use of surveys without publication of any results, to no universal surveys at all.  A few universities are planning on making available the results of student ratings on the web.  Our own Legal Counsel has advised that the Act is weighted in favor of disclosure, but has also advised caution.  How the use of student ratings of instruction will play out in the context of the proposed Freedom of Information and Privacy Act for Alberta's Universities is, at this point, not at all clear.  The proposed Act was not written with universities in mind.  Exceptions may be made for the application of some parts of the Act to the University, but this is not at all certain.  Our best information at this point is that the Commissioner's Office will deal with the Ratings Instrument only if a formal complaint is registered.  Given the uncertainties surrounding the issue of publication the Task Force has made recommendations regarding publication, but has attempted to exercise appropriate caution in its recommendations on publication.  The recommendations on publication reflect the input from both the University's Legal Counsel and the University's Archivist and Information and Privacy Officer.

The Task Force did not see as part of its mandate the development of a reporting system for the publication of the results of the Instrument.  The technical details will have to be done by someone else, preferably the University Computing Services.  Nonetheless, the Task Force did spend considerable time in developing recommendations which have to do with the content and format of the reported and published results as well as principles which should govern the publication of results.  An overriding concern has been to ensure that safety measures can be put in place in order to restrict access to students for the intended purpose, that is, to assist students in course selection.  The concerns are reflected in the recommendations dealing with publication of results, guidelines for publication and the development of a reporting system.  In the view of the committee the preferred medium for the publication of results is the electronic medium rather than hard copy.  As noted above some universities are planning on making the results of universal surveys available on the web.

  ( Top / Bottom of Report )

Costs

The Task Force was faced with two possibilities with respect to the form of the Ratings Instrument and generating  results from the Ratings Instrument, that is, using forms that could be optically scanned or using plain forms.  In the latter case data entry would be necessary.  According to cost estimates provided by UCS, the most efficient and cost effective method is to use forms which can be scanned optically.  The annual cost estimates provided include time and costs associated with the development and handling of the forms, reporting and annual maintenance costs for the system.  The estimated time related to departmental administration has been separated to indicate departmental labour requirements.  The actual impact on units will depend on the administration of the Universal Student Ratings Instrument in conjunction with revised Faculty instruments.  This may reduce incremental effects, whether in terms of support staff time, or casual labour costs.  In-class administration by student volunteers could drastically reduce the financial impact on units and impact on support staff.  The cost estimates also include the development of a reporting and publication system as a one-time cost (see Appendix #8).
 

  ( Top / Bottom of  Report )

Recommendations

 

PURPOSE AND EXPECTATIONS

The purpose of evaluation of teaching at the University of Calgary is, and has been, to help foster excellence in teaching by providing information of both a formative nature (to identify areas of strength and areas in need of improvement), and a summative nature (for merit, promotion and tenure purposes).  Faculties and Units across campus have developed and used instruments tailored to their own needs.  New in this context is the decision by GFC at its 419th meeting to approve the use of a Universal Student Ratings Instrument.  It is the view of the Task Force that such unit specific instruments should continue to be used and that the Universal Student Ratings Instrument should not be seen as a replacement for those instruments.  Furthermore, it is the view of the Task Force that evaluation should continue to be a multi-faceted process, both to preserve the integrity of the process and to address concerns that are at times raised about evaluations.  The Ratings Instrument proposed therefore is, and should remain, a small instrument to supplement instruments and procedures already in use.  Accordingly, the Task Force makes the following recommendations concerning expectations.
 

  ( Top / Bottom of  Report )

EXPECTATIONS

1.     Evaluation of Teaching at the University of Calgary shall include the Universal Student Ratings of Instruction and other forms of evaluation designed by individual Faculties/Units.  Under no circumstances shall the Universal Student Ratings Instrument be the only source for evaluation of teaching.

2.     It is expected that Faculties/Units will supplement the Universal questions with their own forms of evaluation consistent with their own policies or guidelines.  Other forms might include teaching dossiers, faculty/department/unit instruments, evaluation by the Head, evaluation by peers, interviews with students, and documented instances of achievement and/or problems in the classroom.  It is expected that Deans and Heads will use the information provided by the Student Ratings of Instruction Instrument and these other sources for both summative and formative purposes.

3.     All faculties or units are to ensure that credit courses are assessed with the Universal Student Ratings Instrument each time they are offered by an instructor, subject to the exemptions and qualifications listed below.  (The term "instructor" includes all instructors of record whether these are Continuing, Contingent Term, Limited Term, or Term Certain.  The term "course" includes all undergraduate and graduate courses.)

  ( Top / Bottom of  Report )

FORM OF THE INSTRUMENT

As has been pointed out in the Report above, the twelve questions approved by GFC at its 420th meeting have been subjected to a thorough piloting process.  The process has resulted in some revisions to the questions.  These revisions are reflected in the proposed instrument (see pp. 20-21).  During the piloting process student response was also sought on the proposed preamble which includes a request for the student ID. The purpose of the request for the ID is twofold.  It adds an element of seriousness and ownership to the evaluation process.  It provides the means to access important demographic information for reporting and comparative purposes (i.e., age, gender, major, etc.).  The following recommendations apply to the evaluation items and to the face sheet of the proposed instrument.

4.     The Universal Student Ratings of Instruction shall take the form of the 12 item sample Instrument on pp. 20-21.

5.     The Universal Student Ratings of Instruction Instrument shall contain the following preamble:

The information you provide on this rating instrument is intended for use by students to aid them in selecting courses, by Deans and Department Heads to assess instructors, and by instructors to assist them in improving instruction, and by the University for administrative research purposes.  Therefore it is essential that you answer the questions carefully.  If you have concerns about the course or instruction that are not addressed by this instrument, please speak to the Department Head or Dean.

You are asked to provide your ID number in order that the University can access relevant statistical data (e.g. major, year, gender, age) and to ensure student registration in the course.  Your ID will NOT be revealed to the instructor(s) receiving the summary information from the responses to the items on the instrument.  Instruments which do not include an identification number that indicates that the respondent was a student registered in the course or course component, will not be considered.

Your participation in this evaluation process is gratefully acknowledged by the Students' Union, Instructors and Administration of the University.

6.    The Universal Student Ratings of Instruction Instrument shall include the following questions for background information (items not available through the use of the ID number).

• whether the course is a required or optional course
• whether the course is part of the student's major
• percent of classes attended
• workload compared to other courses taken
• expected grade in the course

  ( Top / Bottom of  Report )

CONTROL

The Task Force has, throughout its deliberations operated on the assumption that this is to be a University Ratings Instrument, not a Students' Union Ratings Instrument.  The legal advice received by the Task Force indicates that the rating forms are tangible property capable of possession and that the University can retain rights to those sheets and the information on the sheets.  On the basis of the assumption stated above and the legal advice given, the Task Force offers the following recommendations.

7.     The central administration of the University shall undertake the financing of the Universal Student Ratings of Instruction.

8.     The central administration shall take responsibility for archiving the data from the Student Ratings of Instruction and the publication of the results for purposes of course selection.

9.     The University of Calgary has copyright ownership of the form of the rating sheets and any data and reports compiled therefrom.
 

  ( Top / Bottom of  Report )

ADMINISTRATION

As indicated in the statements on purpose and expectations, the Task Force sees this Ratings Instrument as an addition to evaluation instruments and procedures already in use.  The Task Force was aware that there are a variety of procedures employed for the administration of existing evaluation instruments.  The Task Force took the view that the University should not insist on a single procedure for the administration of the Ratings Instrument, but that Faculties/Units can adapt their own procedures as long as they are consistent with the principles of the administration of this Instrument (see Recommendation #11).  The Task Force was also concerned to build in appropriate exemptions to respond to sensitivities surrounding the use of such instruments and the need to protect the confidentiality of student responses particularly with respect to the reporting of demographic information.  At the same time there was a concern to protect the idea of universality.  The exemptions are intended to strike a balance between the concern for universality and the concern to build in needed protection.  Accordingly, the Task Force offers the following recommendations with respect to administering the Instrument and reporting the results of the Instrument.  It should be noted that reporting in this context does not include the publication of the results for student use.

10.    Where feasible, the Universal Student Ratings of Instruction Instrument shall be administered along with other Faculty, Department/Unit instruments.

11.    The procedures for the administration of the Ratings Instrument shall be determined by each Faculty/Unit.  However the following principles should be observed:

• The instructor shall not be present during the administration of the Ratings Instrument.
• Those involved at any stage in the administration of the Ratings Instrument shall be at arms length from the Instructor.
• Appropriate measures shall be in place to ensure that there can be no tampering throughout the process.
• Instructors shall not have access to the results until after the grades have been submitted.
• The Instrument shall be administered during regular class meetings where such are held.  Alternative arrangements may be made for courses which do not have regularly scheduled meetings.

 

12.   Normally the Universal Student Ratings of Instruction Instrument shall be administered during the last quarter of the Instructor's involvement with the course.

 At the discretion of the Head and course coordinator, in cases of multi-instructor courses it may be administered more than once during the term.

13.    Appropriate instructions shall be given to those receiving the Instrument.   To ensure this, there shall be training given to those administering the Instrument.  The training shall include:

 -  information about the basic principles of the process (e.g., confidentiality, use of the data)
 - specific identification of the course or course component being evaluated.

14.    The Universal Student Ratings of Instruction Instrument shall be administered each time a course is offered subject to the following exceptions:

a.  Unless requested by the instructor, the Ratings Instrument shall not be administered in classes with an enrolment of fewer than eight (8) students.
b.  In the case of low enrolment courses (fewer than 8), instructors have the option of having the Ratings Instrument administered and the data aggregated across multiple course sections and/or terms for the same course.
c.  With the written consent of the Head or Equivalent, the Ratings Instrument need not be administered in cases where there are serious extenuating circumstances (e.g., extended serious illness, extended personal duress) that could demonstrably affect the quality of instruction.

  ( Top / Bottom of  Report )

REPORTING TO INSTRUCTORS, DEANS, HEADS OR HEAD EQUIVALENTS

It bears repeating in this context, that the Ratings Instrument is intended as only one aspect of the evaluation of formative and summative instruction at the University of Calgary.  Heads and Deans need to make sure that reporting with respect to course evaluation includes forms of evaluation other than the results of this Instrument.  Of particular concern to the Task Force was the use of the data from the Ratings Instrument with respect to small classes and, with respect to difficult classes.  Student responses from small classes are less reliable than responses from larger classes.  Heads and Deans should be aware that data from small classes and difficult classes needs to be treated with due caution (see Appendix #6).

15.   Upon making a written request Instructors will be provided with copies of the original completed data sheets of the 12 item scale for purposes of verification.  The written request must be made within three (3) months of the distribution of the reports.

Discrepancies between the reported version and the data sheets shall be brought to the attention of the person (group) made responsible for the administration of this system.

16.   Student demographic background information shall not be reported in cases where such reporting will compromise the anonymity of students.

17.   The results of the Ratings Instrument shall be reported to Heads, Head Equivalents, and Deans except under the following circumstances.  (Please note that the reports will always be given to the instructor.) 

a. Once the response rates are known, and before the results are made known to the Instructor, the instructor may request that the results not be reported in cases where the respondents fall below 20% of enrollees or fewer than eight (8) respondents, whichever number is greater (e.g., for a class of 40 or fewer students there would have to be eight (8) or more respondents; for classes of over 40 students, the 20% rule applies).
b.  In cases where the instructor is attempting radically new instructional approaches and/or is teaching in an area clearly outside her/his area of expertise, the instructor may be exempted from the reporting process, provided written consent for the exemption is given by the Head or Equivalent prior to the beginning of the course.  To have the rating results of an exempted course included in the reporting process, the instructor would again need the written consent of the Head or Equivalent.
c. In cases of serious extenuating circumstances that could demonstrably affect the quality of instruction, the instructor or someone on behalf of the Instructor may request written consent not to have the results reported.  To have the rating results of an exempted course included in the reporting process, the instructor would again need the written consent of the Head or Equivalent.
d.  To assist them in the use of the data Heads, Head Equivalents, and Deans shall be provided with a statement of purpose and expectations, a broad description of statistical terms, and a statement on the limitations concerning the use of statistics from small and difficult classes (see Appendix #6.)

  ( Top / Bottom of  Report )

PUBLICATION

Publication here means making the results available to students for purposes of course selection.  Some of the recommendations in this section flow from the previous section and are intended to put in place the needed exemptions to protect Faculty in cases where such protection is warranted.  The Task Force has been aware that the business of publication is the most sensitive area in the use of instruments for the evaluation of teaching.  At the same time the Task Force is aware that publication is required if the Universal Instrument is to be used to assist students in the business of course selection.  It is clearly not possible to devise a fail safe system to prevent every attempt at abuse or misuse of the published information.  Nonetheless, appropriate warnings about the illegitimate use of the published information need to be in place and access to the published reports must be limited to registered students.  In the view of the Task Force, the medium for publication of the results of the Universal Instrument should be electronic.  The Task Force discussed the possibility of restricting student access to the evaluations of only those courses in which a student has the right to register, but recognized the difficulty in defining appropriate limits for restrictions and implementing the restrictions technically.  The Task Force suggests that this discussion might be reopened at a time when we have more information on the technical possibilities of the system to be developed for reporting and publishing the results of the Ratings Instrument.

18.    Publication of the results of the Ratings Instrument shall be done not before the end of the term following that in which they were administered in order to allow for verification and inclusion of the instructor's comments if any.  Publication will be done electronically provided students have adequate access and that accurate security measures can be put in place.

19.   Instructors shall have access to the published reports for their courses.

20.   The system shall have built in safeguards to restrict access to students registered at the University of Calgary.

21.   If the results of the evaluation instrument are not reported they shall not be published.

22.   Student demographic background information shall not be published in cases where it could compromise the anonymity of students.

23.   All results shall be published except under the conditions noted in #21 and 22 and in the following exemptions listed below:

a.   Once the response rates are known, and before the results are made known to the instructor, the instructor may request that the results not be published in cases where the number of respondents falls below 20% of course enrollees, or fewer than eight (8) respondents, whichever number is greater.
b.    Unless requested by the instructor, the results shall not be published in cases of courses taught by Term Certain Instructors.
c.    In cases where the instructor is attempting radically new instructional approaches and/or teaching in an area clearly outside her/his area of expertise, the instructor may be exempted from the reporting publication process provided written consent is given by the Head or Equivalent prior to the beginning of the course.  To have the rating results of an exempted course included in the publication process, the instructor would again need the written consent of the Head or Equivalent.
d.    With the written consent of the Head or Equivalent, results shall not be published in cases of serious extenuating circumstances that could demonstrably affect the quality of instruction.  To have the rating results of an exempted course included in the publication process, the instructor would again need the written consent of the Head or Equivalent.
e.    Unless ratings are requested for individual instructors in team taught courses the ratings shall be published only in overall aggregate form.
f.    The published reports for exempted courses shall have only the statement, "results not available."

24.   The results from the Universal Student Ratings of Instruction made available to the students shall have the following cautionary notice:  "The reports are provided for the personal use of students for purposes of course selection.  The information reflects recent student opinions about certain aspects of course offerings and instructors.  It may not be relevant for your own personal needs and, it may not be indicative of the instructors in other settings or in future course offerings.  This information should be supplemented by information from other sources such as course information sheets, other students, student advisors, faculty advisors and instructors.  The reports are copyrighted by the University of Calgary.  Unauthorized copying, distributing or disseminating of the reports or the information within them by any individual may constitute grounds for disciplinary proceedings."

The results shall also be accompanied with advice to students on how to read the reports (see Appendix #9).
 

  ( Top / Bottom of  Report )

 GUIDELINES FOR INSTRUCTOR FEEDBACK, REPORTING AND PUBLISHING OF STUDENT EVALUATIONS OF INSTRUCTION

The following recommendations relate to the form in which the data will be reported and published.  The concern of the Task Force was to provide enough information to be useful to all interested parties.  Minimally this should mean the inclusion of appropriate demographic information, some comparative data, and the possibility of a statement about the course from the instructor with respect to publication (see Appendix #7 for sample graph).

Feedback of Universal Ratings Results to Instructor

25.   To enhance the contribution of the universal ratings instrument to the quality of instruction, the results  provided to course instructors of record shall include the following:

a.  The mean, median, standard deviation, frequency distribution, and decile course/instructor on each rating item.
b.  The number of student respondents and number of course enrollees.
c.   A comparison of course/instructor rating on each item with the corresponding mean, standard deviation and decile for Department, Faculty and University courses at the same level (i.e., 200-level, 300-level).
d.  Where it does not compromise student anonymity, mean and standard deviation for each item will be provided by gender, required/not-required course, major/non-major, student age, number of prior university/college courses taken, percentage of classes attended, rated workload of class, and student's expected grade in the course.  (See also Recommendations 10-24 on In-class Administration, Reporting, and Publication regarding guidelines for protecting student anonymity.)

Reporting of Universal Ratings Results to Heads, Head Equivalents and Deans

26.   In recognition of the dual formative and summative responsibilities of Heads, Head Equivalents and Deans, reports to them on the results of the Universal Ratings Instrument will be identical in form and content to the information provided to the instructor.  (See Recommendation #17 for exemptions relating to reporting.)
 
Publication of Student Ratings for Purposes of Course Selection

27.   The results of the Universal Ratings Instrument shall be published separately for each academic session.  The published results are the exclusive property of the University and are for the sole purpose of assisting University of Calgary students in course selection (see Recommendations 18-23 for exemptions and use of published data).  The published results shall include the following information:

a.   Mean, median, frequency distribution, and standard deviation on each rating item for the course/instructor.
b.  The number of student respondents and number of course enrollees.
c.   A comparison of the course/instructor rating on each item with the corresponding mean and standard deviation for Department, Faculty at the same level (i.e., 200-level, 300-level).
d.   Mean student rating of the course workload.
e.  Total number of times the instructor has taught this same course.
f.   An optional 60-word section on course selection information provided by the instructor.

  ( Top / Bottom of  Report )

OTHER USE OF THE DATA

There is the possibility that the information from the Ratings Instrument will be useful for purposes of institutional analysis and research.  Such use should be subject to the normal controls pertaining to research involving human subjects.

28.   With the written consent of the Vice-President (Academic) or Delegate the data from the Universal Student Ratings of Instruction Instrument may be accessed for research and institutional purposes.  Such use of the data will be subject to the code of ethics governing research on human subjects.  In cases where the data is used for these purposes it shall not be published except in group average form.

  ( Top / Bottom of  Report )

IMPLEMENTATION

29.  The use of Universal Student Ratings of Instruction Instrument shall begin in the Fall Term, 1998.

30.   The publication system shall be put in place for the Fall Term,1998. 

FUTURE TESTING OF THE RATINGS INSTRUMENT

University structures do change over time as do course delivery methods and instructional approaches.  Instruments of this nature should be re-evaluated periodically for reliability and utility in response to changing environments.

31. The Universal Student Ratings Instrument shall be evaluated for its utility and reliability after the first three years of use, and periodically thereafter.

32. The three-year evaluation shall include testing for the effects of including the ID number on the face sheet of the Instrument and the effects of the publication of the results for purposes of course selection.

  ( Top / Bottom of  Report )

ARCHIVING

*33. The original Ratings Instruments completed by the students shall be retained for a period of one (1) year following the completion of the session within which it was administered.

Aggregate (i.e., anonymous to Instructor) data may be kept in perpetuity electronically for institutional and research purposes.

Individual electronic data will be kept and destroyed according to University policy regarding personnel files.
 

* This recommendation should be revisited after the development of an electronic system for the collection of data and in light of the FOIPP legislation.
 

  ( Top / Bottom of  Report )

 
 SUMMARY OF RECOMMENDATIONS FOR ADMINISTRATION, REPORTING AND PUBLISHING
 

Condition	Assessment	Reporting	Publishing
Classes size	below 8 students request by instructor	request by instructor	request by  instructor
Sample Size	below 20% or 8 respondents   request by  instructor	below 20% or  8 respondents  request by  instructor	below 20% or   8 respondents  request by  instructor
Extenuating circumstances	advice from Head	advice from Head	advice from Head
New approaches	advice from Head	advice from Head	advice from Head
Teaching outside of expertise	advice from Head	advice from Head	advice from Head
Multi-instructor  courses (more  than 3 instructors)	yes	in aggregate	in aggregate
team-taught courses (more than 3 instructors; course segments taught sequentially)	for each instructor or course overall at discretion of Faculty	for each instructor or course overall at discretion of Faculty	for each instructor or course overall at discretion of Faculty
Term Certain (sessionals)	yes	request by instructor	request by instructor
Cells of 2 or fewer	yes	no report on demographics	no report on demographics