Appendix 6

Advice to Heads and Deans on the Use of the Data
WHAT INFORMATION DOES THE UNIVERSAL RATINGS REPORT PROVIDE?

The purpose of the University of Calgary Universal Student Ratings of Instruction Instrument is to provide a common University measure of instruction that can be used by students in course selection, by Instructors to enhance their instructional activities and by Administrators responsible for the evaluation of instruction.  To facilitate the realization of these goals, a comprehensive report on the student response data is provided to Instructors and relevant Administrators, and a more limited report "published" for University of Calgary student use.  The Instrument is not only the source of information regarding course instruction, and it is not intended to be used as the sole basis for the administrative evaluation of instruction.  It should also be recognized that the reliability of rating information is affected by class size, and that caution should be exercised in interpreting rating results from small classes.  The information that the Instrument provides, and some of its limitations are discussed in more detail below.  Instructors and Administrators are strongly encouraged to review the Ratings Instrument Policy document for an understanding of the policies and guidelines that apply to use of the Instrument.
 

1. Graphical Frequency Distribution.  This provides a visual representation of the percentage of student respondents rating the course instruction at each of the seven levels available (from 1 to 7).  Because all respondents to a given item are represented in the frequency distribution, the total of the respondents across all seven items is always 100 percent.  Careful inspection of the frequency distribution can provide valuable information regarding the overall rating, including the shape, symmetry, and consistency of the distribution, and the extent to which it is based on extreme scores.  If based on the actual number of respondents, the frequency distribution can also be used to calculate the "mean", "median", "decile" and "standard deviation" of the distribution (see sections 2. to 5. following).

2. Mean Rating.  The mean rating reported is the most commonly used measure of  "central tendency" of a distribution of scores, and is what most people mean by  the term "average".  It is calculated by adding up all the ratings from 1 to 7 on each item and then dividing that sum by the number of respondents.  By taking the actual value of the ratings into account, the mean provides a valuable estimate of the net level score distributions that are symmetrical and do not have many extreme scores in one direction or another (i.e., which are not skewed in one direction or another).  Differences between their means is the most common way to evaluate differences between distributions (e.g., between courses or different rating items in the same course).  For distributions that are skewed, asymmetrical and/or based on small classes (see Frequency Distribution above), any differences between means are less informative.

3. Median.  Another measure of "central tendency" of a distribution, the median refers to the score that separates the lower half of scores from the upper half.  Because it is based on a ranking of the scores, it does not take the actual value of the rating into account and can be affected by relatively small differences in the underlying distribution of scores.  Its major advantage as a measure of the general level of scores is that it is not much affected by extreme scores.  If a distribution were perfectly symmetrical, its mean and median would be identical.

4. Decile.  Similar to the median in that it is based on a ranking of scores, the decile indicates the level of a score within a broader distribution in 10 percent steps.  For example, a score in the lowest 10% of the distribution would be termed a "first-decile score", one in the top 10% of the distribution would be a "tenth-decile score".  Thus, the decile can provide comparative information regarding the level of one rating item relative to others in the same course, or even to those in other courses.

5. Standard Deviation (S.D.).  The standard deviation is the most widely used  measure of the variability of a set of scores, and simply represents the degree to which scores in a distribution vary from its mean.  In the "normal" or "Gaussian" or "bell-shaped" distribution that is the basis for many statistics, 68.3% of cases would be found between 1 S.D. below the mean and 1 S.D. above it (i.e., 34.15% above and 34.15% below).  The S.D. provides useful information regarding the consistency of a set of scores, and is also useful for evaluating the significance of any differences between two or more means.  The higher the  standard deviation, the greater the difference between the means must be to be meaningful.