So Much Data, So Little Information
Data, data, data, is the word of the movement in education policy these days. If some is good, more must be better, and we’ll take all that we can get. Throw in some references to Moneyball, an eye-popping statistic or two (More open education data could generate $1 trillion in worldwide value each year!), shake well, and you’ve got an education plan!
Reality, of course, begs to differ. Dig in a bit, and the caveats and nuance are quickly revealed. All tests scores are data, but not all data are test scores. Overemphasize one source of data and it stops measuring what you think it’s measuring. It’s not how much data you have, it’s how you use it. And, running through the whole debate, the oft-unsaid, oft-ignored, yet always important dictum: “Data” is not the same thing as “information.”
This is a critical distinction. I have a friend who’s a bodybuilder, but if you just feed his height and weight into a BMI calculator, you’ll be told he’s obese. Other data—say, his percentage of body fat—clarifies the picture, but ultimately it’s a human mind that assembles the data into usable information. Information is something you can act on. In my friend’s case, if we were asking a doctor to give him advice, that doctor needs information, not just height and weight statistics. And depending on what my friend needs, his height and weight may be irrelevant.
In education, test scores are less helpful versions of the height and weight statistics in the above example. There are obvious differences. “Height” and “weight” are physical characteristics for which we have valid, generally agreed upon measuring tools. “Learning” and “proficiency” are abstract, and any one measurement will at some level of detail wind up being a little arbitrary (unless you really believe that one wrong answer can constitute a bright line between “proficient” and “somewhat proficient”).
There are also similarities. Height and weight are some of the easier things to measure these days, and test scores provide the most accessible, large scale sources of data in education. They’re the “easy” thing to measure (relatively speaking), and can be used to draw some basic conclusions.
They can also be misused or misunderstood. In the case of the not-actually-obese bodybuilder, it’s the assumption that the Body-Mass Index (BMI) is an accurate calculation for identifying appropriate weight classes. And sometimes it is! There are people identified by the BMI as underweight who are, in fact, underweight to a degree that it materially affects their health. Similarly, there are people with the same height and weight as my friend who are actually obese. This is more than just a-stopped-clock-is-right-twice-a-day-ism. The BMI was intended to categorize most people in a particular way, and often isn't off by more than a category.
What it doesn’t do, and what test-score-based data systems are even farther from doing, is convert data into standalone, human-ready information. The BMI ends up being a slightly more complicated form of data. When combined with other data (like body fat percentage), it helps humans construct better information.
Unfortunately, the data trend in education is largely about more data, not better information. Take the obsession with labeling and ranking schools, often relying on a couple different test-score-based calculations and maybe some graduation data. The rush to proliferate these systems as absolute arbiters of quality led to a lot of shoddy systems being rolled out. It’s fair to speculate that this was a case of the political opportunistis -- those who wanted to find reasons to label public schools as failures in order to push their own agendas -- getting the better of the technocrats, social justice advocates, and frustrated citizens. Those who would have preferred more useful information actually got a disappointing sea of new, questionable data points.
There are people working hard to find ways to convert data into information and use it for good. The National Education Policy Center at UC-Boulder, for example, released a report in late October investigating ways data can be used for improvement and accountability. They concluded that, “DDIA [data-driven improvement and accountability] in the U.S. has come to exert increasingly adverse effects on public education,” but they go on to identify several principles and recommendations for bringing our system more in line with more successful models.
Students, families, teachers, schools, districts, states, and the country all need good information about education. Many times, that information starts with data, but we need to do more to build better sources of data and use it in a way that helps inform improvements. The algorithms should work for us, not the other way around.