As May begins, I feel confident in declaring the end of major combat operations in the war against Spring 2021 State Testing. States are doing what they have decided to do. Parents and students will do what parents and students do. Of course, skirmishes in remote outposts or densely populated strongholds will continue to claim casualties – à la John Laurens in Hamilton (the show, but not the cast recording). And zealots who haven’t heard or don’t believe that the war is over will remain huddled in their man caves, continuing to pop up on Zoom calls fighting a long-lost battle.
For most of us, however, attention has now turned to the critical question of what happens next after tests have been administered. Lines in the sand are being redrawn and after a tactical retreat a new battle cry has emerged among those opposed to Spring 2021 testing:
“Test if you dare, but you must not compare!”
You can administer state tests, but you cannot make comparisons as you attempt to interpret and use the results from those tests. Interpreting state tests results without making comparisons – now that would be a neat trick. In reality, however, it’s a classic example of a Catch-22 – “A paradoxical situation from which an individual cannot escape because of contradictory rules or limitations.”
“That’s some catch, that Catch-22,” he observed.
― Joseph Heller, Catch-22
Go ahead, try to interpret a state test score without making a comparison to something. I’ll wait.
“What about …” – Nope. Still waiting.
“But these tests are criterion-referenced, we abandoned norms decades ago.” Did you really, though? Anyway, it doesn’t matter. Still waiting.
Fact is, a state test score is useless without comparisons. An individual scale score has no inherent meaning, and you can double that lack of meaning for a vertical scale score. At a minimum, you need some knowledge of the score distribution, and for all practical purposes, some external reference point is also required.
Achievement levels fare just slightly better without comparisons. Convention stipulates that we regard an achievement level, or performance standard, as a criterion, albeit a criterion heavily influenced by norms during standard setting. Even so, interpreting an achievement level classification without making comparisons is pretty much impossible.
At the most basic level, of course, an achievement level classification is the comparison of a student’s scale score to the achievement level standard. I can hear the arguments now about why that is not an appropriate comparison to make this year.
Even if we accept the student classification (i.e., interpretation of the student’s test performance) as valid, reliable, and fair, so what? We don’t interpret student performance without comparison to something: expectations based on past performance, the performance of other students in the class or school, past performance of the student’s teacher with similar students.
The bone that they offer you to gnaw on is that you can use the spring 2021 test score to identify strengths and weaknesses of an individual student. Unfortunately, that’s the one piece of information that state tests are not designed to provide – as those pushing the no comparisons argument know all too well.
“Insanity is contagious.”
― Joseph Heller, Catch-22
The argument against making comparisons started innocuously with the recommendation that spring 2021 test results shouldn’t be used for school accountability. Sure, no problem.
Next on the chopping block were aggregate school scores – “Remember opt outs!”
Then there were grave concerns about individual test scores due to remote test administration, something apparently much trickier than a year of remote instruction.
And oh my, “What about the students?” – students, who apparently are going to be unmotivated to take the test seriously and simultaneously scarred for life by being asked to take a test with no consequences.
And of course, there is the trump card that need not be named, but effectively ends all arguments in 2021.
I don’t think that I can endure another weekend of tweets saying that state tests would be fine if, you know, we just didn’t use them for anything and didn’t report results for districts, schools, and students – and that’s not just for spring 2021. [Actually, I could get behind the idea of not reporting results for students.]
“[They] agreed that it was neither possible nor necessary to educate people who never questioned anything.”
― Joseph Heller, Catch-22
Equally true, is that it is neither possible nor necessary to educate people who always question everything. Sadly, we appear to be willing to accept that people in the United States fall into one of those two categories – dependent upon who is attempting to educate them at the time.
It was a very different world just 7-10 years ago, in the years leading up to the implementation of the new state assessments based on the Common Core State Standards. At that time, vast resources (e.g., money, time, materials, outreach efforts) were devoted to preparing people to interpret and use the results of the new assessments. Making appropriate comparisons (or avoiding comparisons) to old assessments and old performance standards was a major focus of that effort. [I still have the wristband flash drive that CCSSO provided full of helpful materials for states to use working with local educators, media, and the public.]
This time around, however, the mantra seems to be that we cannot report test results because we cannot trust people to interpret and use them correctly. The list of “people” we cannot trust apparently includes local educators and policymakers, the media, and the general public. State policymakers are included, too, of course, but why bite the hand that feeds you when you can achieve your desired end simply by saying, states will do the right thing, of course, but everybody else …”
Making the argument that people are incapable of understanding how to interpret and use state test results appropriately seems to me to be a risky proposition for the large-scale assessment community – one with the potential for significant, long-term, negative unintended consequences.
Add to that the unsettling concept of a professional community involved in education arguing against their ability to educate the people.
And while we’re at it, the notion that it’s too difficult to educate people to understand something as simple as test scores is just a little bit scary for the future of our democracy.
“You know, that might be the answer – to act boastfully about something we ought to be ashamed of. That’s a trick that never seems to fail.”
― Joseph Heller, Catch-22
The fact that we are willing and able to make arguments against testing or reporting scores in spring 2021 based on an appeal to expert opinion (i.e., our technical expertise and superior understanding of test scores) is itself an indictment of our field and the broader state assessment community – supporters and critics of state assessment alike.
Making these arguments in spring 2021, decades into annual state testing, is a clear admission that we have failed miserably at our single most important job – to ensure that stakeholders are equipped to appropriately interpret and use the test results that we provide to them – which includes providing test results that are useful to them.
Yes, that job was even more important than ensuring the technical quality of those test scores, for the simple reasons that it was more challenging and failing to succeed had far-reaching consequences.
Nobody said it was going to be easy. They never promised us a rose garden or The Rose Garden. In 2015, states, particularly Smarter Balanced states, ran from state comparisons faster than a Trump Republican offered a COVID-19 vaccine. In 2021, the mere mention of comparisons throws well-intentioned progressives into a panic over inadvertently perpetuating a deficit mindset. Deal with it.
The bottom line is that our field has a job to do, so let’s roll up our sleeves (get vaccinated while they’re rolled up) and just do it.
Image by PETE CHACALOS from Pixabay
Embrace the Absurd 