Data Driven Decisions?

I don’t have a particular recommendation other than that we base decisions on as much hard data as possible. We need to carefully look at all the options and all their ramifications in making our decisions. ~ Dorothy Denning

Accountability in education has always been a political talking point. But in the late 1990s it began to pick up speed and in 2001 President Bush enacted the No Child Left Behind policy. Ever since then, accountability in education seems to increase every year. Every year we have another data set we need to collect and report on, something new to assess. Along with this increase in accountability reporting and learning outcomes assessment is an increased focus on data and data collection. Which lead to a phrase that has now become quite common in education, data driven decision making.

Around the year 2000 the phrase started making its way into interviews and opening day speeches. We as educators were going to revolutionize what we do by making data informed decisions. What this seemed to spur more than anything in the early days was more and more data collection. Eventually leading into the big push for the development of course level, program level and finally institution level learning outcomes. We wrapped these assessments into our comprehensive program review work and we talked about it incessantly.

We really love two things in education, we love forming committees to talk things to death and we love using jargon. We love talking about making decisions and even the process of developing the process to make decisions, but making decisions, not so much. So we have really become enamored with talking about the process of making data driven decisions. We love collecting the data and even love listening to our research offices tell us all about the data they’ve collected. We get extra tingly when they, “disaggregate” the data. But unfortunately what I’ve seen in the last twenty years is a lot of descriptive data about who are students are and our success and retention data. More recently that data is normally disaggregated by race, gender and age. But that seems to be as far as we go, we don’t do enough to analyze that data and use that analysis to drive positive change.

On every campus we have student support programs for a variety of groups, they may be by gender, ethnicity, or major. And what I don’t see happening is actually using a data driven process to analyze these programs. And of course, you can’t assess outcomes without established goals and measurable and quantifiable metrics. Let’s use a made up example.

Let’s say that our research office in their annual research dump shows us that our Martian students are not succeeding, or being retained at the same level as our other students. So we decide, based on our data, to start the Martian Student Support Program, the MSSP because the only thing we love nearly as much as jargon are acronyms. The step that is often overlooked at this point is the establishment of specific and measurable goals. So when we start the program, we should set goals like the program will lead to an 10% increase in success rates over the next three years. That way we can assess to see if the program is actually accomplishing the goals (reasons) that it was created for in the first place.

Now, I’m not picking on student support programs, this should be done for academic programs as well. And honestly, we have this data for most academic programs. Our comprehensive program review data packets often contain exactly this data. We are often given success, retention, transfer and completion data. For our workforce programs, at least in California, we can even access employment and salary information albeit on a two to three year delay. So we have all of this data and that’s great, but where’s the decision making?

We rarely look at programs, see their data flagging and make significant decisions. If a program is generating few transfers, certificates or degrees, how often to we see decisions being made to change or discontinue a program. Far too often these decisions are more related to the likability and personality of the faculty in that program. We seem afraid to even assess student support programs much less make decisions to change or discontinue them and honestly I think it’s fear that the administration will be perceived as biased against the specific student group the program is supposed to be helping.

Where is the type of data analysis and data visualization that shows us where are true programming and support gaps really are? How are we using data to discover new pathways and new ways of doing things? Where is the decision making based upon these new or even older analyses?

Now I’m not saying it’s never done, I’ve seen small examples from time to time. But in general I haven’t seen the operational change to a data driven decision making framework we’ve been talking about for twenty years. My academic training is in science, so this is especially sad to me.

While not exhaustive or definitive by any means, a Google search looking for examples of data driven decision making in community colleges leads to lots of links to models. There is absolutely no shortage of papers, sites and consultants to help you create a data driven decision making culture at your college. What I didn’t find were specific examples. Now I’m sure they exist, however the difficulty in finding them with a Google search means they are likely not as common as we would like, or frankly as they should be. I would love to see examples of where this is working in the comments.