Unless you’re lucky enough to totally cut yourself off from the outside world relaxing with a good book on a warm, sandy beach or hiking a beautiful mountain vista it’s impossible to go a day without encountering a discussion of which jobs or professions are likely to be replaced by AI, if they haven’t been already. Paralegals apparently are obsolete as are graduate students functioning as research assistants. The intriguing part there being that although workers in both roles perform similar functions, one is viewed as a distinct career path while the other more as a training ground or apprenticeship.
Speaking in generalities, which as a good psychometrician I have been trained to do, I view the landscape thusly. Some professions and professionals are adopting the appeasement approach, welcoming in the AI invader that will eventually replace them. Other fields, a la the MLB with steroids, are willing to look the other way as long as possible while reaping the rewards of AI use. Still others, like professional writers, seem to be circling the wagons, pushing back like the old-time NCAA or IOC, declaring persona non grata and imposing lifetime bans on anyone whose writing is even remotely associated with AI. Each of which is a predictable and somewhat reasonable response to an unpredictable and often cruelly unreasonable situation.
Whither psychometricians on the role and use of AI in our work?
Ah, one cannot answer that question without first considering the questions of who a psychometrician is and what a psychometrician does.
A Psychometrician By Every Other Name…
As someone who has been labelled a psychometrician since the turn of the century, I feel supremely qualified to state that the criteria and qualifications for placing psychometrician on your business card or LinkedIn profile are as clear as mud. We know that a psychometrician’s work is quantitative, has something to do with tests, and that those tests generally involve making inferences from observable behaviors about mental abilities, attitudes, and/or processes. And among those three, quantitative must be the most important criterion given that I can think of many people who call themselves psychometricians who have nothing to do with the inferences drawn from test scores and I know a good number of psychometricians who have gone years without seeing a single actual test item.
Ironically, and perhaps prophetically, AI seems to have a particularly difficult time defining describing the role of a psychometrician. No doubt connecting with psychometrics with psychology some AI descriptions stress human interactions and the personality traits necessary to make those types of interactions successful. Now there’s a classic AI hallucination if I’ve ever seen one.
One AI description in the same vein, apparently confused psychometricians and psychometrists, and classified psychometricians as health-care support professionals, a label which might fit some of my colleagues at NBME, but not one that applies broadly to psychometricians in PK-12 or higher education. In fact, I can think of several educators who might view psychometricians as more of a health threat than a health-care support.
So, who is a psychometrician and what do we/they do?
For the sake of containing this discussion somewhat, I am going to go out on a limb and then chop off the low-hanging fruit. To whit, I’m picking up my cleaver and lopping off all of those psychometricians who have been trained merely to apply psychometric procedures to data. Whether they have a master’s degree, an EdD, or a PhD, if they function as technicians routinely and rotely applying preselected IRT models and statistical procedures then although they fulfill a vital role in the process, there is something lacking there and it’s a given that their positions are eminently replaceable by AI.
I am also going to ignore those at the other end of the spectrum, those psychometric gods who develop new models. I couldn’t begin to describe what they do and how they do it.
For now, let’s focus on those of us in the messy middle.
I want to turn my attention to those who have devoted a large portion of their careers to taking human decision-making, and in many cases humans, out of psychometrics.
To Err Is Human
So much of psychometrics is devoted to minimizing error not only in what we have measured, but also in how we measure.
In one camp, we have those seek to minimize human error in the psychometric process. There are some who do this by establishing rules, Standards, or guidelines, if you will, for engaging in psychometric activities. There are others who strive to automate routine psychometric procedures to increase efficiency and minimize human error. These people are well-intentioned and we can say that their efforts and the fruits of their labors are good, at least until they aren’t.
You want rules to guide you and automated procedures to follow. You don’t want to have to reinvent the wheel, at least not until you’re skidding out of control on the ice or stuck in the mud. At that point, however, you want a psychometrician who can analyze the situation and decide whether simply adding some studs or chains to your wheel is sufficient or whether it’s necessary design the ski, pontoon, or caterpillar track you need in place of a wheel. You want a psychometrician who understands enough to know which Standards and conventional ways of doing things are lode-bearing and which can be torn down without collapsing our house of cards.
In another camp, you have those who have sought to remove humans from the process as much as possible, if not altogether. Most often this pursuit is undertaken to increase efficiency without sacrificing accuracy and precision. Increases in accuracy and precision are a bonus. (I’m intentionally avoiding validity and reliability.)
Item writing has always been the most human-heavy component of the test development process, often involving some of the most human of all humans, teachers, hundreds of them gathered together for a week at a time. Although the psychometrician generally only views the herd from the safety of a podium, it should not come as a surprise that psychometric labs across the country have been dedicated to creating computer-generated clones from the DNA of human-developed items.
We successfully kept humans out of the scoring process for decades, but when they crept back in during the 1990s, the psychometric industrial complex set its sights on automating scoring. A campaign that has been quite successful.
Of course, when the tests have been developed more humans are introduced into the process in the form of teachers to administer the tests and students to take them. Getting computers to administer our tests was easy; we just had to wait for technology to catch up. But eliminating students, the subjects of our measures, our raison d’etre? Surely, you jest. As a young teacher, each year (usually in dead of winter) I heard veterans lament, this job would be perfect if it weren’t for the kids. Then they would get back to work.
As a budding psychometrician, I heard the same lament as students did one silly or another when presented with our carefully constructed tests. Psychometricians, in contrast to my teacher colleagues, responded with a hearty, “Challenge accepted!”
We employed simulated data wherever we could – as a wise man one said, never underestimate the value of simulated data or overestimate the value of real data. We somehow figured out a way to apply Bayesian statistics before we had longitudinal or historical data, quite the impressive feat. And now articles appear as fast as peer reviewers can recommend them for acceptance demonstrating new ways to remove students from psychometric modeling. (I’m sure that work is well underway to replace peer reviewers with AI to speed up the process.)
So, perhaps it is feasible to eliminate humans and humanness from every part of the psychometric process.
It’s a Brave New World!
Pump The Brakes
Strike one in our simulated logic is that virtually all of our efforts to minimize the role of humans in psychometrics have focused on doing what we already do more efficiently. But the problem with that approach is that what we already do with testing has a low ceiling for utility and truth be told is kind of boring. Note, guardrails on the side of highways have limited utility and are kind of boring, but it’s important that they are there.
Strike two and a more fundamental flaw is that from its beginnings our psychometrics have been built around the concept of stability and educational measurement, at its core, is about change. The goal of education, in general, and instruction, in particular, is change. Not just movement along a fixed scale, which we can estimate, but real change in underlying abilities and attitudes as one moves across grade levels and learning progressions. The psychometricians we need are people who can understand, interpret, and then design tools to measure and describe that change.
Which brings us to strike three and our fatal flaw. The manner in which we have siloed psychometric processes so that a working psychometrician never has to see a test item or be actively involved in interpreting a test score rips the heart out of a profession that at its core, when done well, represents the pinnacle of being human.
The Ultimate Human
While the future of psychometricians was bouncing around the back of my mind, I found myself reading Addy Baird’s 2026 book, The Magical Game, a tale of “the spirit and history of baseball’s superstitions, rituals, and curses.”
Early in the book, Baird describes one of the built-in features of baseball with its 162-game season:
This great litany of games and at-bats offers almost endless opportunity to look for patterns – something our human brains are perfectly calibrated to do. Our superior pattern processing, in fact, is one of the central features of our advanced species, to the extent that it’s one of the main things we look for in interstellar communication in our attempts to find intelligent life in the universe. (pp 22-23)
She goes on to say,
“I think that’s what really makes us human, our need and our propensity and our ability to create meaning out of thin air, basically. To find meaning in things that seem to be intrinsically useless.” (p. 24)
And that’s when it hit me, Baird is describing psychometricians and the joy of psychometrics, the siren that drew so many of us into the field in the first place: the opportunity to look for patterns; the ability to create meaning out of thin air, basically; and to find meaning in things that seem to be intrinsically useless.
Does anybody do those things better or hold those things more dear than a psychometrician?
Which means that if those are the things that make us human then arguably, psychometricians, with all of their human awkwardness, are the pinnacle of our species, the ultimate human being.
I was only able to sit with that comforting thought (or truly frightening thought) for a few minutes until I realized that those are the very skills and dispositions that we are building into AI. Artificial intelligence, with the emphasis on intelligence, is an attempt to build a more perfect human.
And that realization largely makes moot the question of what psychometric tasks we can and should hand over to AI and which require a human. The distributions of human psychometricians and AI platforms performing psychometric functions can be plotted together on one of our ubiquitous unidimensional scales: Psychometricianness.
With that question out of the way, we are free to tackle the larger question: How can we rethink psychometrics and educational measurement to better model change, to reflect the realities of and meet the needs of education?
Image by Gerd Altmann from Pixabay
Embrace the Absurd 