Cut Score Shenanigans

Periodically (as in now, incidentally) controversy arises over the setting of “cut scores,” that point that demarcates pass from fail or several points denoting A from B, B from C, and so on. As any student can tell you, one point can mean the difference between success and failure. Where that point sits on the continuum of possible points, or scores, on a given test is critical—and, in fact, almost wholly arbitrary.

There is no such thing as an objective test. All tests are subjective, from the choice of test topics to the construction of individual test items, from the determination of correctness to the number of correct responses that constitutes success or proficiency or mastery or what have you. These are human decisions up and down the line and therefore subject to human idiosyncrasies.

Let’s say a test is composed of 100 items, thus 100 possible right answers. One group might decide that a score of 50 or more correct answers constitutes success. This group bases its cut score on one set of factors. However, because the factors that go into determining success are so numerous, it is impossible to consider all of them. Another group, basing its decision on a different but equally reasonable set of factors, might set 70 or more correct answers as the cutoff for success on the same test.

The point is that because cut scores are arbitrary—based on whatever criteria the decision makers choose to use or to ignore—they can be infinitely manipulated. Want to make a test look rigorous? Raise the cut score—in other words, make it harder to score enough points to pass. High failure rates often are referenced by our snake oil politicians and education “reformers” to say, “Look, we have high standards—and most student can’t meet them because schools are lousy.” But set the cut score lower and those same oily folk are likely to lament, ”Look, too many students are skating by, so our standards must be too low because our schools are lousy.”

If the same snake oil sellers want, they can (and do) change cut scores from year to year for essentially the same test, and so can make it appear as though students are scoring better or worse over time, depending on which result resonates with their ideology and political ambitions. Beating the drum of rigor to an accompanying wail about the lack of student “progress” on standardized achievement tests is the theme song of the ongoing manufactured crisis, to use Berliner and Biddle’s term, in public education—that “crisis” in which the public schools are the scapegoat for all of America’s ills.

Cut score shenanigans victimize test takers—students—and, through the misuse of standardized tests to judge teachers, rate schools, and characterize communities, a host of other stakeholders, all of whom are held hostage to ideologically driven policies regarding the importance of testing. Faith in the power of testing and the accuracy of cut scores is ill placed. It’s not going to cure any supposed ills of our public schools. It’s just snake oil, folks.

Note: For the uninitiated I recommend David Berliner and Bruce Biddle’s The Manufactured Crisis: Myths, Fraud, and the Attack on America’s Public Schools. It was published in 1996—twenty years ago next year—but it remains spot-on.