Teagle Essays

Using Assessment Evidence for Improvement in Complex Institutional Environments

Charles Blaich

Center of Inquiry in the Liberal Arts at Wabash College

Four years ago, I started the Wabash National Study of Liberal Arts Education, a longitudinal study that examines the factors that influence the outcomes of a liberal arts education (e.g. critical thinking, need for cognition, interest in and attitudes about diversity, leadership, moral reasoning, well-being). When I talk about the Study and its impact on institutions, I often describe it as a research project that morphed into an assessment project. But as I've thought about that description, I've come to realize that it is inaccurate. That description makes it sound as though the original intention of the project was "pure" research and that our goal of helping institutions came later.

But that's not true as our goal was always to help institutions. What's changed profoundly for me over the last four years is not the goal of the study, but my understanding of the importance of evidence in promoting improvements in student learning. I started the Wabash Study believing that evidence was the single most import element necessary to promote the improvement of student learning. I now see it as an essential piece—the first and perhaps easiest piece of the improvement puzzle. Evidence about student learning enters into a swirl with other factors such as institutional culture, values, recent history, economic factors, and governance structures, as well as the narrative(s) that all influence what and how we teach. It turns out that using evidence to advance student learning in this complex environment is complicated and calls on many skills beyond our ability to analyze and make sense of assessment data.

So how does one go about using assessment evidence for improvement in our complex institutional environments? In working with a wide range of institutions in the Wabash National Study, I've learned a few lessons and identified a couple of hazards, that is, things to "watch out for" because they sound perfectly reasonable, but can also mark the end of an effective response to assessment evidence.

First, the lessons.

Using assessment evidence to improve student learning is an "evidence informed" rather than "evidence driven" process. It's important to remember that for people at your institution—the "consumers" of assessment evidence—the programs, departments, etc. that you plan to assess have all existed prior to your current plans to assess them. They were all carefully designed and revised by smart people making choices about what do to. New assessment data may provide information that these people will find useful, but it may also contradict basic assumptions that have been guiding their actions in the past. Don't carry forward the results of your data analysis expecting to be welcomed with shouts of "at last someone can help us truly improve these programs!" Expect instead to be welcomed with skepticism and perhaps outright suspicion. After all, regardless of what you learn from the assessment evidence, the current state of affairs isn't random, it was carefully constructed.

For those who have the task of analyzing assessment data:

• Data analyses are the first 10% of the project.

• There is no such thing as the "killer" presentation or report that on its own sparks community action. These reports and presentations are at best the starting point for a series of conversations. Create a structure to host these conversations.

• To be effective, assessment data must be interpreted in community—this is a messy process in which students, faculty, and staff, work together to shape the "what did we learn" narrative that becomes the basis for action.

• No matter how fancy your quantitative analyses, I believe that institutional survey and assessment data cannot be fairly interpreted without having conversations with students.

• Don't rely on combined data sets creating generalizability. There are institutional differences on measures of engagement and student learning, even for institutions that look like peers by many of our institutional metrics. Another way of saying this is that the characteristics that faculty and administrators use to determine institutional peers may have little to do with campus learning environments.

• Sometimes merging data from institutions together means wiping out “effects.”

• Even if you get similar effects, it may still mean different things. We have seen a number of instances in which similar survey results pointed to very different underlying issues.

• Conference papers and research publications on your assessment data do not improve student learning. There's nothing wrong with using assessment data for scholarly research, but it is important to be honest with yourself that allocating time for doing analyses and writing up papers will take you away from the complicated social processes you'll need to engage in to bring your data into your communities.

Be patient, using assessment evidence takes time. Jillian Kinzie (NSSE Institute at Indiana University) has estimated how long it takes to use assessment data once the campus has the data in hand:

• Increasing campus dialog on assessment - 1 year

• Creating a rubric - 1.5 years

• Increasing active learning - 3 years

• Changing first year retention - 5 years

• Changing critical thinking - 7 years

• Changing graduation rates - 10 years

Respect the power of the institutional narrative to overcome anything that evidence can throw at it. In my experience, all institutions have a narrative that they use to describe their work, values, and the way people there do things. These narratives are incredibly powerful and useful. They remind community members what they should be doing. I've begun to think of this narrative as a "treaty" of sorts that has been developed regarding faculty and staff workload, priorities, goals, admissions, etc. However, this treaty was crafted without some of the information that may emerge from your assessment project. If that information cuts against it, the narrative will protect itself. It is important to find the time and space for this information to persist long enough so that people at your institution give it serious consideration.

Now the hazards.

Sometimes reasonable statements are nothing more than pleasant sound ways of calling things to a halt. Here are a few common comments that may be pointing out perfectly reasonable things, or are an institution's way saying "not so fast there assessment cowboys!"

• “The most important outcome of this assessment project has been to create conversations.”

Positive implication: Rich, formative, inviting conversations are a necessary part of good assessment projects. Good conversations build community and trust, and they can develop support for implementing responses to assessment data in challenging political and resource-limited environments.

Negative implication: Sometimes satisfaction with good conversations is really an excuse that keeps institutions from moving on to responding to their data. When it comes to making changes to our institutions, good conversations are a means, not an end. We set the bar too low when we congratulate ourselves for merely talking about student learning.

• “Our campus/student body/curriculum/department is unique.”

Positive implication: Developing a thicker understanding of the culture of an institution is important both for developing better interpretations of assessment evidence and identifying politically and culturally plausible responses. Taking the special qualities of our institutions into account is a good assessment practice.

Negative implication: The less-favorable take on “uniqueness” sounds like this, “The unique qualities of our campus/students/curriculum/department means that fifty years of research on the practices and conditions that promote student learning don’t apply to our institution. Student confusion about assignments or projects is a sign that our program is appropriately challenging; syllabi are restrictive pedagogical tools and student confusion about the structure of their classes is formative; our students learn from lectures because our faculty are master lecturers; data indicating that 40% of our students don’t write drafts of their papers is meaningless because they actually use continuous revision; etc.”

• “We need more assessment data before we can act.”

Positive implication: Student learning is complicated. It is often helpful to dig into data to get a better sense of its implications for an institution. For example, it is always important to use focus groups to follow up on survey data to build a better understanding of the specific institutional and classroom practices behind students’ survey responses.

Negative implication: Assessment becomes the ongoing process of gathering and analyzing data without any action toward the improvement of student learning. There are two reasons this can happen.

First, assessment evidence often challenges practices and structures that faculty, staff, and administrators value by suggesting that these practices and structures do not promote student learning. Sometimes the easiest way to keep doing assessment without threatening those cherished structures is to continually gather more data. In essence, doing more assessment becomes a defense mechanism. When this defense mechanism kicks in, gathering and filing data safely away without using it becomes a ritual institutional act.

Second, it is important to remember that scholarship is built around the process of gathering and reflecting on evidence and information for the sake of gathering and reflecting on more evidence. It will seem both natural and satisfying to academics for assessment to evolve into scholarly research. But scholarly research, especially in the liberal arts, eschews application. Its success is based on the fact that it generates more questions and more publications, not on whether it creates practical changes outside of the world of scholarship.

• “We need a better data report.”

Positive implication: The 75 page statistical masterpiece in which you describe findings on 50 logistic regression analyses, your thoughts on the sphericity assumption, your bootstrapping procedures for error terms, and the significant 6-way interaction between gender, SES, mac versus pc preference, performance on the vuvuzela-appreciation scale (VAS), and student view on which Beatle was most influential on NSSE Academic Challenge scores was not entirely clear to your humanities or—even—your social science faculty. It may be important to write a shorter report with simple descriptive statistics and cross-tabs.

Negative implication: You're on a snipe hunt for the perfect report. Any report you write is either: a) too long or too short; b) too simple or too complicated; or c) leaves open the question of what the assessment data mean or doesn't allow any room for faculty and staff input on the interpretation. The bottom line is that people may be reluctant to read and think through anything that is complicated, but recoil at the thought of being spoon-fed assessment data. Assessment evidence is complicated, and so is making sense of it.

With all these lessons and hazards in view, what should be done?

• Gather, review, and make preliminary sense of evidence prior to the start of any improving student learning project. Projects should focus on working with the community to make sense of the evidence, disseminating the interpretation of the community, and developing and providing resources for actions.

• Have a clear plan for disseminating assessment evidence on campus. Consider which key faculty, staff, and governance structures need to be involved in considering and/or responding to the assessment evidence.

• Create crisp outcomes for the project. What exactly do you want to happen with this project?

• Develop plans for involving students in the process of interpreting the evidence and evaluating possible responses to the evidence. What do you want to be different two or three years from now?

• Double loop learning: Assess the assessment. Create moments every 1-2 months where you step back, evaluate how the process is going, and make revisions to help you move towards your goals.

These remarks have been modified from a talk given by Charles Blaich at a Teagle meeting on August 17-18, 2010 in New York City.

Email This Article