Home > Blogs + Podcasts > The Teagle Liblog

The Teagle Liblog

June 30, 2008

Assessment Markers
By Richard J. Light, Harvard University

When undertaking an assessment project that has as its goal the improvement of student learning:

Write out as precisely as possible what “treatments” or “interventions” you intend to make. Ideally the description should be specific enough so that any other campus could—if you get good results from your intervention—implement a similar intervention on their campus.

State specifically to all concerned what you would consider a good outcome. Is the process itself the outcome (“good discussions”) or do you want demonstrable results in student learning? There are many outcomes that we all might agree would be desirable, even when they are quite different. For example, suppose a campus initiates a new introductory course in the physical sciences with the hope of stimulating new and growing student interest. One possible outcome might be improving test scores on a final exam relative to past years in similar courses. A quite different outcome would be to simply ask, “How many of the students in this new introductory course choose to take a second level course in the sciences, compared to past years? How many choose to take even a third level course, compared to past years?” It might well be that a campus decides the most important thing is getting more students to take more advanced courses in the sciences, rather than simply attaining slightly higher final exam scores.

Comparison helps, but what to compare to? Last year’s results? Results at a comparable institution? Places using the same rubric or instrument? Define the comparison early. Every campus knows it is important with any innovation to have a “comparison group.” Yet there are so many possible comparison groups. It is helpful for any campus to be as specific as possible when they choose a comparison group. For example, suppose a campus wants to improve students’ general writing. Many campuses state this as a major goal. Is the comparison group each freshman’s writing the day they enter college compared with how well they write after an intervention? Or is the comparison group a sample of students from other similar colleges? Or might a comparison group consist of students from the same college from prior years? There is no answer that is in the abstract “always correct”—the big question and challenge is to specify a compelling comparison group, and to do it in advance of the intervention.

Measure at an intermediate stage so mid-course corrections can be made. This is a big point and it is done so rarely. Staying with the writing intervention example, many campuses that try new ways to improve student writing only gather information at the beginning and then again at the end of the entire intervention. It should be hard to gather information once or twice or even more frequently during the actual intervention. For example, in a writing improvement effort, gathering data even as frequently as two or three times each semester can offer valuable insights. Plus, if after three rounds of gathering information it is hard to see any improvements, that might suggest a tweaking of the “treatment” because it would suggest that the intervention doesn’t seem to be as effective as had been hoped for, at least so far.

Nobody succeeds 100%. That’s OK. Suppose fifteen campuses each try a new method for teaching biology to undergraduates, in the spirit of stoking up both knowledge and student interest in sciences. Should we expect, as we take an overview of all fifteen campuses that are trying different ways to stoke up this interest, that all fifteen will succeed? It probably makes sense to agree that some campuses will hit home runs, some will hit doubles, some will hit singles, and some won’t succeed at all. Should we then be dismayed at those campuses that aren’t succeeding? Probably not. A far more constructive view is that when trying new ideas, some ideas will end up working out well, some not so well, and some not at all. This is not a surprise. This is the way bench science often works, and it is the way social science often works. So for campuses, the reward should come from trying new ideas and innovations, with the full knowledge that some won’t be successful.

Learn from failures. Iterate. The advantage of having a number of campuses working on broadly similar innovations is that each can learn from the successes and also the failures of others. If then campuses each try to improve their students’ writing by working alone and never talking to one another, each campus might or might not succeed, and that is fine. Yet by talking to one another, and sharing progress notes (or lack of progress notes), each campus can iterate and constantly work to improve its efforts based on what it is hearing from, and learning from, other campuses. This is close in spirit to what defines a “learning organization.”

Email This Article | Subscribe to E-Updates

Post Your Comments