USE OF STUDENT OPINION SURVEY DATA In an effort to assess adherence to the seven principles approved by the Faculty Senate to guide the use of the Student Opinion Survey, the Teaching Effectiveness Committee distributed the attached questionnaire to department chairs, deans and program directors in the spring, 1988. Responses were received from all but two departments, resulting in a 97% response rate. Principle 1: That student opinion of instruction be only one of the ways to evaluate teaching. Units should seek additional ways, depending upon their particular needs and interests. ’ 100% of the respondents agreed with this principle. 24% of the units responding have generated additional evaluation instruments, and of those units which have not developed additional instruments, 56% reported an interest in doing so. A number of respondents questioned the validity of the present instrument used to solicit student opinion of teaching effectiveness. Relatively few units indicated that teaching effectiveness is evaluated in ways other than the student opinion survey; approaches reported included class visitation by chair or peers, review of course materials, and faculty self-report. Principle 2: That the new form be administered in all courses at the University. This is necessary in order to ensure completeness and reliability of data. Units would be free, of course, to develop other instruments for use in addition to the TEC form and, in accord with Appendix C, to use only data from those other instruments. 69% of the respondents indicated that the Student Opinion of Instruction Survey (SOIS) form is administered in all of their units' courses. Of the 31Z who indicated that the form is not administered in all courses, several reasons why not were offered: the form is not appropriate for some courses such as field experiences, independent study, and practicum courses; very small classes are not surveyed; and the form is not useful for team-taught courses. In most of those units (24%) reporting that they have developed additional forms, those forms are used in addition to the SOIS form rather than in place of the SOIS form. Principle 3: That the new form be administered every semester. 81% of the respondents indicated agreement with this principle. Reasons reported for disagreeing with this principle included concern that frequent administration leads to complacency on the part of students ("overkill") and that administration every semester for every course is not necessary to establish reliability in the evaluation of a faculty member's teaching effectiveness. Principle 4: That data from the new form be processed in such a way that both individual faculty and department heads know the following: a. the University mean and standard deviation for each statement on the form by the level of the course (e.g., for all 1000-level courses surveyed in the University during the past three years); and b. the unit mean and standard deviation for each statement by the level of course (e.g., for all 1000-level courses surveyed in the unit during the past three years). 81% of the respondents indicated that they use Principle 4a. for the evaluation of teaching in their units, and 87% use Principle 4b. Some respondents who reported that they do not use these principles indicated that they did not understand the meaning of standard deviation. Principle 5: That individual units request specific means and_ standard deviations for particular kinds of courses (e.g., for all freshman and sophomore op level lecture courses designed primarily for General College Students). Only 4% of respondents indicated that they have ever requested the information described in Principle 5. Most respondents indicated that they did not know such information was available or that they had been unable to obtain the information. Several respondents questioned the utility of this kind of information. Principle 6: That administrative analyses of student opinion pay attention only to data that indicates [sic] a statistically high or statistically low performance when compared to response for similar courses. This means that basic satisfactory competence must be assumed whenever the student response to a statement is less than one standard deviation below the mean for similar courses in a unit. 67% of respondents indicated agreement with this principle. Those who reported disagreement with this principle generally indicated that standard deviation is not an adequate measure of deviation from an acceptable norm, or they questioned the one-standard deviation cut-off point, or felt that evaluations below the mean should always be considered unacceptable. Fewer than half (44%) thought that faculty members understood the impact of this principle on teaching evaluations. Principle 7: That, except in the case of new faculty, administrative evaluations be based not on [sic] course-by-course or semester-by-semester data but on patterns established over the past several semesters by all courses taught by a faculty member. 80Z of the respondents reported that they apply this principle when conducting administrative evaluation of teaching by faculty members, and 83% indicated agreement with this principle. Those who reported that they do not apply this principle indicated that they believe data from individual semesters are important to consider in order to identify problems early. Most respondents (69%) indicated that they believe this principle is understood by faculty in their units. Discussion of the principle by chairs or distributing it in writing was suggested in order to make faculty more aware of the principle. CONCLUSIONS AND RECOMMENDATIONS: 1. There is a great need to evaluate teaching on the basis of a variety of methods. According to the survey results, a majority are interested in developing additional means of evaluating teaching. Assistance should be provided to unit heads, faculty, deans, etc., in order to identify and implement additional methods. Copies of the seven principles should be distributed semi-annually to the faculty and unit heads and other administrators. Included along with each copy of the principles should be an explanation of potentially confusing aspects of the student survey process (e.g., definition of standard deviation). Presently, we emphasize the evaluation of faculty teaching. More attention needs to be given to the development or improvement of teaching. The TEC should continue to consider the problems associated with the current methods of evaluating teaching. PRINCIPLE #1: "That student opinion of instruction be only one of the ways to evaluate teaching. Units should seek additional ways, depending upon their particular needs and interests." Do you agree with this principle? Yes:100%(54) No: If you answered ''No", please give your reasons: For the evaluation of teaching in your unit, have you generated additional evaluation instruments? Yes:24%(13) No:74Z(40) Both:2Z(1) If you answered "Yes" to #3 above, please attach a copy of any such evaluation instruments; these will be incorporated into the TEC's file of such documents. If you answered "No" to #3 above, would you be interested in creating such an evaluation instrument? Yes:56Z(30) No:17Z(9) Blank:28Z(15) Comments: Use the space below and on the back to record any additional observations and opinions regarding Principle #1; if none, so state. t PRINCIPLE #2: "That the new’ form be administered in all courses at the University. This is necessary in order to ensure completeness and reliability of data. Units would be free, of course, to develop other instruments for use in addition to the TEC form and, in accord with Appendix C, to use only data from those other instruments." Is the Student Opinion of Instruction Survey (SOIS) form administered in all your unit's courses? Yes:69%(37) No:31%(17) If you answered "No" to #1 above, in the space below please identify each course not being so surveyed. After each course, provide a brief explanation as to why it is not being so surveyed. (If more space is required, please use the back of this sheet or an "addendum" sheet. ) Has your unit developed any other instruments to survey student opinion of instruction? Yes:24Z(13) No:76Z(41) If you answered “Yes" to #3 above, please attach examples of each instrument and complete the following: a. In the evaluation of the teaching of faculty members, do you employ your unit's form in addition to the TEC form? Yes:77%Z(10) No:15%(2) Sometime : 8Z(1) In the evaluation of the teaching of faculty members, do you use your unit's form instead of the TEC form? Yes:8%(1) No:85%(11) Somet ime: 8Z(1) Comments: Use the space below and on the back to record any additional observations and opinions regarding Principle #2; if none, so state. PRINCIPLE #3: "That the new form be administered every semester." 5. ua sk Do you agree with this principle? Yes:81Z(44) No:19Z(10) If you answered ''No", please state your reasons. At the present time, SOIS forms are not distributed routinely in some or all of the courses that fall in the categories listed below. Please indicate which, if any, of your unit's courses have not been surveyed by means of the SOIS form since the inception of the present system in the spring semester, 1986: (if none, or not applicable, so state) Summer School Courses: Team-taught Courses: Courses in which fewer than six student are enrolled: Basic Student Teaching Courses: Courses conducted by teaching assistants who are not listed as instructor of record: Courses with no instructor of record listed in the schedule of classes (i.e., taught by "Staff"'): g. Other courses (please list and explain): Comments: Use the space below and on the back to record any additional observations and opinions regarding Principle #3; if none, so state: PRINCIPLE #4: "That data from the new form be processed in such a way that both individual faculty and department heads know the following: a. The university-mean and standard deviation for each statement on the form by the level of the course (e.g., for all 1000-level courses surveyed in the university during the past three years). The unit mean and standard deviation for each statement by the level of course (e.g., for all 1000-level courses surveyed in the unit during the past three years.)" Do you use the information reflected in Principle #4.a. for the evaluation of teaching in your unit? Yes:81%(44) No:19Z(10) Do you use the information reflected in Principle #4.b. for the evaluation of teaching in your unit? Yes:87%(47) WNo:11Z(6) Unsure:2Z%(1) If you answered "No" to #1 or #2 above, please indicate your reasons. Comments: Use the space below and on the back to record any additional observations and opinions regarding Principle #4; if none, so state. PRINCIPLE #5: "That individual units request specific means and standard deviations for particular kinds of courses (e.g., for all freshman and sophomore level lecture courses designed primarily for General College Students. )" Have you ever requested the information described in Principle #5? Yes:42(2) No:94Z%(51) Blank:22(1) If you answered "Yes" to #1 above, please explain how you used this information to evaluate teaching in your unit(s). If you answered "Yes" to #1 above, but did not use the information to evaluate teaching, please indicate your reason(s) for requesting the information. If you answered "No" to ##1 above, please explain why you did not request this information. Comments: Use the space below and on the back to record any additioanl observations and opinions regarding Principle #5; if none, so state. PRINCIPLE #6: “That administrative analyses of student opinion pay attention only to data that indicates (sic) a_ statistically high or a statistically low performance when compared _ to response for similar courses. This means that basic’ satisfactory competence must be assumed whenever the student response to a statement is less than one standard deviation below the mean for similar courses in a unit." Do you agree with this principle? Yes:672(36) No:26Z(14) Yes&No:22%(1) Don't Know:2Z(1) Blank:4%(2) If you answered "No" to #1 above, please explain your position. Do you apply this principle when conducting the administrative evaluation of teaching by the faculty members of your unit? Yes:63Z(34) No:20Z(11) , Usually:4Z(2) Blank:13Z%(7 Do you feel that the impact of this principle upon the evaluation of teaching is understood by the faculty members being evaluated in your unit? Yes:442(24) No:24%(13) Uncertain:62(3) Blank:26Z%(14) If you answered "No" to #4 above, what do you recommend be done to make faculty members aware of this principle? Comments: Use the space below and on the back to record any additional observations and opinions regarding Principle #6; if none, so state. PRINCIPLE #7: "That, except in the case of new faculty, administrative ie evaluations be based not by course-by-course or semester-by- semester data but on patterns established over the past several semesters by all the courses taught by a faculty member." Do you apply this principle when conducting the administrative evaluation of teaching by the faculty members of your unit? Yes:80Z(43) No:13Z(7) Both:4%(2) Blank:4Z(2) Do you agree with this principle? Yes:83Z(45) No:92(5) Both:4Z(2) Blank:42Z(2) If you answered "No" to #1, above, please explain your position. Do you feel that the implications of this principle to the evaluation of teaching are understood by the faculty being rated in your unit? Yes:69Z(37) No:11Z(6) Unsure:92%(5) Blank:9%(5) NA:22(1) If you answered "No" to #4 above, what are your recommendations for making faculty members aware of this principle? Comments: Use the space below and on the back to record any additional observations and opinions regarding Principle #7; if none, so state.