Appendix 2

Summary of Recommendations

COE-1997  

A. Format of the Instrument

    A number of comments regarding the format of the instrument were raised throughout the questions. Although we did not specifically ask about this, the volunteered information gave us some ideas on how to make the form more "user friendly". So, although we were not asked specifically to do this, we include this section to inform the next steps in the process.

a. Have a five-point rating scale with anchor points of "strongly disagree" (1) to "strongly agree" (5), and have the numbers placed in increasing order.

b. Use the existing statement that says, "Using the following 5-point scale, indicate the degree to which you agree or disagree with the following statements: "

c. Do not include a "no basis for opinion'', as all items should be able to be answered by all students.

d. Include a statement thanking the students for their time.

B. Specific Items

    In summary, all items were revised to some degree, the global item was placed F~rst. items 4 and 9 were deleted, and two new items were added. The specific changes are noted below.

Recommendation and Rationale for Item 1

    It is recommended that Item 1 be reworded from:

The course objectives and requirements were clearly stated.
to:
The course outline or other materials provided enough detail about the course (e.g., goals, readings, topics covered, assignments/exams, due dates, grade weightings).

    The changes makes the item more specific about what is meant by "objectives and requirements" and also that these need to be provided in a written format.

Recommendation and Rationale for Item 2

    It is recommended that Item 2 be reworded from:

The course reflected stated objectives.
to:
The course as taught was consistent with the outline and other materials.

    The rewording defines more concretely what is meant by "reflected" by using the phrase "taught" and refers directly to the course outline and other written materials.

Recommendation and Rationale for Item 3

It is recommended that Item 3 be reworded from:

Instruction was conducted in a well-planned manner.
to:
The course material was presented in a well-organized manner.

    To avoid the vague terms of "conducted" and "well-planned", we substituted using words provided to us by the students and faculty. The new item also ties the item to the course material rather than to "instruction".

Recommendation and Rationale for Item 4

    This item (The instruction enabled effective learning.) seemed to be asking if students had [earned anything from the course, taking into consideration different learning styles. As written it attempts to ask two questions in one: (1) Did the instruction (instructor and support materials) aid the student to learn, and (2) Did the student actually learn anything. Both of these issues came up in other areas as well and will be addressed in the section on items recommended for addition to the instrument.

    Because the construct of student learning styles is specific to students, largely unknowable to instructors, likely unknown to the students themselves, and irrelevant in many instructional situations, we are recommending that avoidance of this construct is the most appropriate course of action. Therefore, we recommend deletion of this item.

Recommendation and Rationale for Item 5

    It is recommended that Item 5 be reworded from:

Student input was treated as part of the learning process, considering class size.
to:
Student input was treated appropriately.

    The vagueness of the phrase "learning process" was dealt with through deletion. Instead, the original intent of the item was the degree to which the instructor valued and dealt with student input which is captured by the term "appropriately".

    We deleted "considering class size" because it was vague to many of the students and not capturing what it was supposed to as reported by several faculty. Also, students seem to appreciate that in large classes there would not be much participation and in smaller ones more would be expected. We anticipate that class size will be available as part of the course demographic information in interpreting findings from all of the in Phase II. Thus, the issue of the impact on student ratings of instructors through the deletion of the phrase "considering class size" is an empirical one that will be dealt with in Phase II. The change in the item also makes it less tied to classroom participation which may be difficult to do in large classes. Finally, other items would also require this as an attaching phrase (i.e., item 8: being reasonably accessible to students, item 10: methods of evaluation). If the Task Force decides to retain such a phrase it will need to make "considering class size" more concrete (e.g., in classes with fewer than 50 students).

Recommendation and Rationale for Item 6

    It is recommended that Item 6 be reworded from:

The instruction cultivated the student's enthusiasm for the subject.
to:
The subject matter was communicated with enthusiasm.

     The new item shifts the question from how much enthusiasm the instructor has instilled in the student, which is largely uncontrollable, to how much enthusiasm the instructor has in presenting the material, which is completely under the instructor's control. In addition, the item specifies that the enthusiasm should occur in the communication of the course subject matter.

Recommendation and Rationale for Item 7

    It is recommended that item 7 be reworded from:

The instructor was reasonably accessible to students.
to:
Opportunities for course assistance were available (e.g., instructor office hours, out-of-class appointments, e-mail, telephone).

    There was an expectation that the term "instructor" would not be used in any items. The new item does avoid this in the stem. In addition, "reasonably" and "accessible" were problem terms due to their vagueness. Instead, we assimilated the primary reason for wanting to access the instructor (course assistance) into the new item, and also provided a number of examples of appropriate means by which to do so.

Recommendation and Rationale for Item 8

    It is recommended that Item 8 be reworded from:

The instructor showed respect for the students.
to:
Students were treated courteously.

    The new item shifts the primary construct being assessed from "respect", which is vague to "courteous" which is more observable and concrete. In addition, faculty members were very much opposed to the term "respect". Given the large number of students who indicated that respect was interpreted to mean courteous or polite, the new item assesses what the students think it means. Again, the term "instructor" was in need of replacement.

Recommendation and Rationale for Item 9

    It is recommended that Item 9 (The course provided a fair evaluation of student learning.) be deleted. How can "the course" provide an evaluation of learning? The vagueness of the terms "fair", "evaluation", and "student learning" all indicated that there was a problem with this item. It was often suggested that items 9 and 10 be combined in some fashion. In the recommended change to item 10, some of the aspects of item 9 are incorporated.

Recommendation and Rationale for Item 10

    It is recommended that the question reworded from:

Methods of evaluation provided opportunity to demonstrate understanding of the material.
to:
The evaluation methods used for determining the course grade are appropriate to assess student learning.

    The new phrasing for this item came from information in items 4, 9, and 10. "Evaluation methods" is specifically tied to "grades" in the new item, which is what students think it means. Most of the evaluation methods at the University are designed to assess learning not understanding and thus "understanding of material" was replaced with the more concrete "assess student learning". Finally, the term "appropriate" should reduce the concern expressed that the methods may not be under the instructor's control (e.g., in large section classes, multiple-choice exams may be the only reasonable alternative).

Recommendation and Rationale for Item 11

    It is recommended that Item 11 be reworded from:

Students' work was graded promptly, considering class size.
to:
Students' work was graded promptly.

    The only change was to drop the phrase "considering class size". We recommend this change for the same reasons as in the rationale for changes to item 5.

Recommendation and Rationale for Item 12

    It is recommended that Item 12 be reworded from:

All things considered, the instructor is an effective university teacher.
to:
The instructor is an excellent university teacher.

    For this item, it was impossible to avoid the use of" instructor" The vague phrases "All things considered", however, does not add anything to the item and detracts from it for the most part. We have suggested changing "effective" to "excellent", due to the concerns over the ambiguousness of the term "effective". There may be similar concerns expressed over "excellent" in Phase II.

    There was a consensus among the researchers that this item should be the first one. Priming effects of answering the previous questions would likely have an effect on the response to this item. While the reverse may also be true, this item is the most global and most subjective. The new wording of the items makes them more concrete and/or observable, and so less likely to be affected by answering the global question.

Recommendation and Rationale for Additional Items

    Based on the above information (and also that included in the following section) we are recommending the inclusion of two additional items:

I learned a lot in this course.

Faculty members in particular cited this issue as one on which they would like some feedback.

The course support materials (e.g., readings, audio-visual materials, speakers, field trips, software, etc.) used in this course helped me to learn.

Both students and faculty members were interested in the feedback on any support materials they used.