Spoiler and content warning: This post includes details from the film Tenet and a discussion of the future of large-scale testing which may upset the reader or spoil the movie if you have not seen it. The Armageddon and the Apocalypse are not for the faint of heart.
I am not a movie guy by any stretch of the imagination. The last movie that I went to a theater to see was Whip It, and that was only because I arrived in downtown Minneapolis for a Taylor Swift concert too early to check into my hotel. You can imagine, therefore, that it takes a lot to convince me to watch a particular film. In the past few years, however, there have been two movies that friends convinced me I just had to see. The first was Avengers: Infinity War. The second, which I watched this weekend, was Tenet. Both movies offer a take on trying to stave off the destruction of the world, or at least the destruction of all life as we know it. Their appeal to my colleagues toiling in the current climate of large-scale testing was immediately obvious.
This Part is, A Little Dramatic
Our field is fully engaged on two fronts – in battles for its soul and for its very existence. The weakness of our position in both battles may have been exacerbated by the events of 2020, but make no mistake, our positions were already weak as 2019 came to a close.
There may be 10-12 years left to act before tripping critical tipping points for climate change, but we are already at the final tipping point with regard to large-scale testing. What we do in the next 12-24 months will determine the fate of all of those young psychometricians being churned out of graduate programs across the country.
In hopes of reaching another 31 people, I repeat here what I wrote last summer:
I can appreciate, and even respect, healthy doses of self-doubt, guilt, and penitence accompanied by a satisfying amount of self-flagellation. There is, however a fine, but important, line between healthy self-flagellation and self-immolation.
A key point in Tenet is “What’s Happened, Happened,” but as the character Neil explains , “What’s happened, happened, it’s an expression of faith in the mechanics of the world, it’s not an excuse for doing nothing.” We can simply give up on large-scale testing and take actions (including doing nothing) that will burn it to the ground, where, believe me, it will not rise again from the ashes like a Phoenix. Or we can continue to keep the faith and fight the good fight to improve tests and testing, a fight that so many of us have been engaged in for the past 30 years, 50 years, or 5 years, depending on where you may be in your career or when you saw the light.
If the goal is to move testing forward, the key is not to take actions, or allow actions to be taken, that will destroy the field before we have a chance to improve it – a tall order for a bunch of measurement nerds.
The Protagonists
The protagonists in the Avengers movies are superheroes. In Tenet, the self-labelled Protagonist is a highly trained, CIA, James Bond type.
In contrast, we are collection of socially awkward individuals who self-selected into a profession that is on the outermost fringe of education. Large-scale testing, the area within education where we would least likely be required to interact with people, in general, educators, in particular, and heaven forbid, children.
It is this group of social outcasts who now find themselves thrust into the midst of the most complex social issues that our country, or perhaps any country, has ever faced. Psychometricians who thrive in a world of probabilities and confidence intervals, along with academics who by their very existence as academics know too much to ever arrive at a definitive yes/no conclusion, are entering a debate which has become increasingly binary, where there is a Black/white answer to every question.
Truth is something that is theorized, even though it is neither observable nor attainable in measurement. In the current battles, the single acceptable Truth is that there are no universal truths.
We Live in a Twilight World. There Are No Friends at Dusk.
In any problem-solving situation, it is critical to understand the problem that you are trying to solve. It is equally critical to understand the problems that others are trying to solve. Those who have lost faith in large-scale testing, or never had faith in large-scale testing, are not going to contribute to finding a productive solution to improving large-scale testing.
As we have seen in the debate over spring 2021 state testing, it can be tempting to align oneself with such people when their views temporarily align with yours on a specific issue. Any victory gained through such alliances, however, is destined to by pyrrhic.
This is Where our Worlds Collide
More dangerous than external threats, are the internal disagreements, fueled by misunderstandings, misconceptions, and miscommunication regarding tests and testing (e.g., use, utility, and consequences) that have clouded the field. Misunderstanding, misconception, and miscommunication are likely only to increase as fairness, which the joint Standards concede “has no single technical meaning and is used in many different ways in public discourse” is set to become the predominant validity concern in the impending debate.
These disagreements regarding tests and testing are more than merely semantic and academic. At their very core, arguments about tests have a measurement solution and arguments about testing do not.
Consider these two questions about the tests, testing, and the performance of Black students (or substitute any other subgroup of students):
- Does a test, or test score, support the intended (or even likely) claims or inferences about the performance of Black students?
- Should testing not be used for a specified purpose, even if the claims or inferences about the performance of Black students are fully supported?
I am not saying that one question is more important. I am not even arguing in this post that individuals or certain organizations will not, or should not, have an interest in both types of questions. I am arguing that they are distinct questions that address different problems with different set of assumptions and are approached using different techniques.
Recent history has demonstrated how counterproductive it is when the distinctions between questions such as these become blurred.
You Have to Start Looking at the World in a New Way
One of the thornier time-twisting paradoxes of Tenet is the “temporal pincer” where teams wage their fight simultaneously, one moving forward from the past and one moving backward (literally) from the future, each team learning from the experiences of the other.
We may not be able to learn from the future in the same way the Protagonist does and it isn’t even clear we can learn from the past. The future, or at least the future consequences of our decisions, however, is not as much of a black box as we might like to think. Although we have often been caught off guard or unprepared, most of the consequences of our decisions regarding large-scale tests and testing over the past three decades have been eminently predictable – even to socially awkward psychometricians, certainly to more general educational measurement specialists.
As measurement professionals, however, we have not been a group inclined to take the long view. To a degree, this has served us well in our work with policymakers, who all too often also are focused on putting out the fire in front of them or on finding the short-term solution that can be implemented within their ever-decreasing tenure.
The way that we look at the present and the future has to change.
We also have to be very careful about the way that we look at our past.
Tenet cites the Grandfather Paradox, but any story involving time travel warns of the unknown, unpredictable, and potentially far-reaching effects of “erasing” the past (see Back to the Future, Harry Potter, Quantum Leap). Nihilistically tearing asunder the foundation of our field, no matter how sandy the ground on which that foundation has been laid, is an action that will have consequences – again eminently predictable consequences.
There will be many who are fine with those consequences, but be sure that you know what problem they are trying to solve. It may not be the same problem that you are trying to solve.
Bold I’m Fine With, I Was Afraid You Were Going to Say Nuts
Early in my career, I signed on to be part of KIRIS and the Vermont Portfolio Project, so you may make a solid case that I am fine with both bold and nuts. For the sake of argument, however, let’s agree to say bold. I know that we are on the cusp of a new era of boldness in large-scale testing. Well-respected psychometricians and measurement specialists are ready to boldly lead large-scale testing to places where no woman has gone before (and where few men have thought of going).
That future can only happen, however, if we take actions now to ensure that there is a future for large-scale testing. This can be the end of large-scale testing or a new beginning.
My choice: “I’ll See You in The Beginning, Friend.”
Embrace the Absurd 