UCCS Assessment - CAT - Student Assessment of Learning Gains (SALG)

http://www.flaguide.org/cat/salg/salg1.htm
Elaine Seymour, Douglas Wiese, Anne-Barrie Hunter, Bureau of Sociological Research, CU-Boulder, Sue Daffinrud, LEAD Center, University of Wisconsin-Madison

Why use the SALG?

What is the SALG?

What is involved?

Instructor Preparation Time:	Time is needed to: clarify and prioritize class learning objectives and their related activities that the teacher wishes to be evaluated; check which existing questions express these and which need to be edited or added. No instructor time is needed to administer the survey, collect, and analyze the resultant data.
Preparing Your Students:	Time should be spent explaining the nature of the instrument to students, how to access and complete it.
Class Time:	Instrument can be give in or out of class. It takes 10-15 minutes to complete the sample instrument.
Disciplines:	Appropriate for all.
Class Size:	Appropriate for all.
Special Classroom/Technical Requirements:	Students need access to the web.
ndividual or Group Involvement:	Normally individual, but could also be adapted for use with small groups.
Analyzing Results:	Data analysis is performed by the program. Instructors receive summary data, averages, and standard deviations (by question or sub-question and cross-tabulations for any pair of questions).
Other Things to Consider:	To insure meaningful results, student responses must be guaranteed anonymity. The instrument may be administered as a final student classroom evaluation instrument: several chemistry departments have adopted it for this purpose. It may also be used at any point in the semester for mid-course corrections to classroom teaching methods. Demographic data may be included for correlation with gender, major, or ethnicity.

Description

The Student Assessment of their Learning Gains (SALG) instrument is an on-line instrument (http://www.wcer.wisc.edu/salgains/instructor/SALGains.asp) that provides information about the specific gains that students perceive they have made in any aspects of a course that instructors have identified as important to their learning. The sample instrument is divided into broad aspects of the class or lab, for example, students' perceptions of their learning gains from:

Students can also be asked to make estimates of their learning retention and the adequacy of preparation for future classes offered by the current class.

The sample questions in each question grouping can be edited and augmented to reflect any set of learning objectives.

After each section, the student is invited to add write-in comments. (In a forthcoming version of the program, a template will be added to allow instructors to categorize and count these additional comments by type.) Students complete the instrument on-line, and instructors to receive a summary of results in both statistical and graphic form.

**Figure 1**. Statements from the Student Assessment of Learning Gains (SALG) Sample Instrument (Focused on Broader Learning Issues of Interest to the Teacher).
Q1. How much did each of the following aspects of the class HELP YOUR LEARNING?
		NA	Was of no help	Helped a little	Helped	Helped a good deal	Helped a great deal
A.	The class's focus on answering real world questions	NA	1	2	3	4	5
B.	How the class activities, labs, reading, and assignments fitted together	NA	1	2	3	4	5
C.	The pace at which we worked	NA	1	2	3	4	5
D.	The class and lab activities	NA	1	2	3	4	5
1.	class presentations (including lectures)	NA	1	2	3	4	5
2.	discussions in class	NA	1	2	3	4	5
3.	group work in class	NA	1	2	3	4	5
4.	hands-on class activities	NA	1	2	3	4	5
5.	understanding why we were doing each activity/lab	NA	1	2	3	4	5
6.	written lab instructions	NA	1	2	3	4	5
7.	lab organization	NA	1	2	3	4	5
8.	teamwork in labs	NA	1	2	3	4	5
9.	lab reports	NA	1	2	3	4	5
*10.	specific class activities (list)	NA	1	2	3	4	5
*11.	specific labs/activities (list)	NA	1	2	3	4	5
*12.	specific lab assignments (list)	NA	1	2	3	4	5

Assessment Purposes

Instructors can discover how much each component of their course is seen by their students as contributing to their learning. This allows instructors to adjust their teaching methods to meet student learning needs more effectively. They also have a basis upon which to discuss specific types of learning difficulty with students. Use of the instrument (especially where it is followed by class discussion of the results) encourages students to reflect upon their own learning processes, and to become aware of what (in their own behavior as well as that of the teacher) enables or deters learning.

Limitations

Students must be guaranteed anonymity: student identification is assigned by the program and is used only for the purpose of checking that all members have completed the survey. Instructors may add requests for demographic information like gender, race/ethnicity, and major and look for correlation across those variables. Correlation of student responses to class scores involves additional off-line analysis. Students should be explicitly informed if this step is taken.

Instructor Goals

The authors are interested in suggestions from users as to other types of questions or information they would like to collect from students that would be consistent with the overall learning gains format. The option of including gender, ethnicity, major, year in school, and other demographic variables may be offered in a subsequent version of the instrument.

Step-by-Step Instructions

Emphasize the usefulness of the information the students offer for your teaching, and the seriousness with which their responses and additional comments are taken. (Our research finds a high degree of student cynicism about the value of their feedback to instructors.)

Analysis

The authors are considering the addition of other questions to the sample instrument, of additions to the statistical package, and a template for the classification/coding of additional typed-in student responses. User feedback on these and other issues are encouraged.

Pros and Cons

A fall 1999 faculty tester (in psychology) offered the following comment: "Overall, I think I'm getting a greater volume of analytic, honest, and potentially valuable feedback with this instrument than with any other I've used. I suspect it's partly the medium, and partly the high percentage of tailor-made questions."

Theory and Research

Research has found that effective teachers share several characteristics (Angelo & Cross, 1993; Davis, 1993; Reynolds, 1992; Murray, 1991; Shulman, 1990). Two of these characteristics are relevant with respect to this type of instrument:

There is substantial research which concludes that administering classroom instruments based on student perceptions of the efficacy of particular teaching methods can be both valid and reliable (Hinton, 1993). The SALG instrument discussed here is one method for obtaining information of direct utility to the classroom teacher about class content, teaching strategies (and the approach in which they are grounded), student activities, testing and grading procedures, materials and resources, organization, pacing, or workload. This information can be used to adjust aspect of any class so as to increase student learning. It can increase the awareness of learning processes in both teacher and students, and form the basis for discussions between teachers and their students, teaching assistants, and colleagues about methods that increase learning.

The instrument has its origins both in a need expressed by instructor classroom innovators and in the evaluation findings from a five-year multi-institution initiative to improve learning in undergraduate chemistry by the use of "modular" teaching. As with other instructors implementing classroom changes, modular chemistry instructors seek new forms of assessment that better reflect their revised learning objectives and pedagogy. These include more appropriate and accurate tests of student learning, and more precise feedback from students on the value to their learning of different aspects of the class.

The basis for a useful form of student feedback to instructors (and their departments) emerged from findings from a student interview study that formed part of the formative evaluation of the modular chemistry consortia. Three hundred and forty-four students were interviewed in a matched sample of modular and more traditionally-taught introductory chemistry classes at eight participating institutions. The sample was chosen so as to represent the range of different institutions across the two consortia. They were: two research universities, three liberal arts colleges, one community college, one state comprehensive university, and one Historically Black college. (Two more community colleges and one research university were added to the sample later).

The focus group interviews were tape-recorded, transcribed verbatim, and the text files entered into a computer program to assist with the analysis. Student observations were of three types: answers to interviewers' questions, spontaneous observations, and agreements with observations made by other focus group members. There were 12,993 discrete comments of all three types. We analyzed these data in two ways-in terms of student assessment of (1) instructor performance as teachers and (2) their own learning gains. In these analyses, we discovered that although students gave positive or negative ratings to specific aspects of the class or of their teacher's classroom performance (e.g., the quality of the teacher's lectures and demonstrations, or the fairness of their tests), the grand totals for all students' observations on how well instructors performed their teaching role were (for both the modular and the comparative classes) broadly 50 percent positive, and 50 percent negative. Thus neither group of instructors got a clear picture of the overall utility of their classroom work when students offered judgments of their performance as professional teachers. This is, arguably, because students lack the knowledge or experience to make such judgments. This finding reflects the common instructor experience that asking students what they "liked" or "valued" about their classes, or how they evaluated their teacher's work (often without offering any criteria for these judgements), tells the teacher little about what students gained from their class.

By contrast, in both the modular and comparative classes, students gave clear indications about what they themselves had "gained" from specific aspects of their classes. When all specifically gain-related student observations were totaled and divided into three types-positive (things gained), negative (things not gained), and mixed reviews (qualified assessments of gains), 55 percent of the observations were positive (for both types of class), 33 percent (modular) and 32 percent (comparative) were negative, and 11 percent (modular) and 13 percent (comparative) were "mixed." The strong similarity between the student learning gains evaluation totals for the modular and comparative classes (though not for particular items) is likely to reflect the early stage of development of the modules and the teachers' limited experience in using them at the time of these interviews. The issue here, however, is not the relative merits of modular or more traditional chemistry teaching, but the hypothesis suggested both by our data on reasons for instructor dissatisfaction with traditional course evaluation instruments, and by these student interview data: that it is more relevant and productive to ask students about what they have gained from specific aspects of the class than about what they liked or disliked.

Student Assessment of Learning Gains (SALG)