It’s Time, for a Change

As we approach the one-year anniversary of watching Americans struggle to read graphs, interpret data, and cope with the harsh reality that even the tiniest percentage of a very large number is still too many individual stories, there may have never been a time when there was such universal agreement that it’s time for a change – not even last November, apparently.

For me, however, March has always been a month bursting with the promise of change. In just a couple of weeks, it will be light outside at 7:00 pm even here in Maine. By the end of the month, scientists and groundhogs will agree that it’s spring. And each March, we have the madness of the NCAA basketball tournaments to cleanse our palate as we transition from celebrating another Tom Brady ring to anticipating the start of the baseball season.

Professionally, too, March was the month when the balance of our attention tipped from cleaning up after last year’s test to preparing for this year’s test. TAC agendas were chock-full of equating plans, standard setting plans, security plans, validity analysis plans (just kidding) – in short, a whole lot of plans. It feels right, therefore, that my thoughts turn to “change” as I ponder my next batch of blog posts.

Almost from the start of my career in large-scale testing, I have had a complex relationship with the concept of change. In 1990, during my first year in the field, Al Beaton offered his oft-quoted and still-quoted observation, “If you want to measure change, don’t change the measure.”  Wise words. But just a few years and one innovative assessment program in Kentucky later, I crossed paths and swords with Dan Koretz.  Following that meeting, Dan (ahead of his time in working remotely, by the way) went on to build a 30-year career spreading the word that if you want to measure change, you damn well better change the measure early and often.  Wise words.

There you have it, the challenge of large-scale testing in the age of state assessments in a nutshell: change the measure without changing the measure.

There is nothing so stable as change

So, we learned as much as we could as quickly as possible equating – about trying to build alternate test forms that didn’t need to be equated (I’ll stop short of calling them parallel forms) while simultaneously learning how to equate those test forms. Because, as any rookie psychometrician will tell you, equating works best when it isn’t necessary. As I will discuss later this month, tackling that challenge test after test, year after year, might have been enough to make for a satisfying career.

Alternate forms, however, were just the tip of the iceberg when it came to change and state assessment programs.  Over the past three decades, state policymakers have come up with just about every change scenario that one could dream up for their state assessment programs, and we would lay awake at night wishing it were just a dream. Among my favorites:

• If you want change, change the measure!
• If you want to change who you are measuring, be more flexible without changing the measure!
• If you want to change what you are measuring, but don’t want to change your testing constraints …
• If you want to change assessment contractors, but want to keep your test and don’t want to change the measure …
• If you want to change assessment programs, but want don’t want to change what you are measuring …
• If you want to make changes to the measure, but don’t’ want those changes to change what you are measuring …

As a field, for the past three decades we tried to rise to each of those challenges and come up with a solution that changed what was supposed to change and didn’t change the rest. Some of those solutions were more creative than others, some more successful than others. Through it all, we relied on a special combination of a belief in test theory and the “science of psychometrics,” trust that everyone involved was trying to do the right thing to improve student learning, and just enough luck to make it all work out well enough.

One Shining Moment

Those experiences of the past three decades are what has made the past year leading up to this spring so incredible. For the first time ever, many of us are saying with conviction, if you want to measure change in spring 2021, please, please, please don’t change the measure. Administer a previously used test form from 2019, if possible. Pre-equate, if possible. There simply are not enough rabbits in our psychometric hats to do what we normally do on the back end of state assessment programs.

That is not to suggest, however, that there is nothing that can be done to produce technically sound test results in spring 2021. It may take just a little more time to produce those results, and it will certainly take more effort to communicate test results effectively to stakeholders so that spring 2021 test scores are interpreted and used appropriately.  But, if state policymakers decide that they want spring 2021 test results, I am confident that the field can give them spring 2021 test results.

We can give policymakers the option to administer Spring 2021 state tests because the various change scenarios that we have dealt with over the past three decades have prepared the field for this moment.  There will be profound changes to large-scale testing in the years ahead, and I hope that the field and K-12 education will be better because of them. Spring 2021, however, is the moment we have been preparing for all our lives, and we must embrace the task in front of us.

“To each there comes in their lifetime a special moment when they are figuratively tapped on the shoulder and offered the chance to do a very special thing, unique to them and fitted to their talents. What a tragedy if that moment finds them unprepared or unqualified for that which could have been their finest hour.”

Winston Churchill

Image by Gerd Altmann from Pixabay