Confessions of a Psychometrician
By OJ Simpsons Paradox with Charlie DePascale
Charlie – As we waited six long months for the release of the 2017 NAEP results, some wondered whether we would ever know the whole story; what really happened that February when NAEP reading and math went digital. Now that those results have been released and the NAEP trend line preserved, what do we really know?
This week, we are pleased to welcome, OJ Simpsons Paradox, a statistician and part-time psychometrician, usually locked deep within the bowels of the government where he has the ear of top education policy makers. Today, he is here to offer his hypothetical account of how a broken trend line could be and should be “fixed” without anyone suspecting a thing.
OJ: It all starts with NAEP. The one constant through all the years, Ray, has been NAEP. America has rolled by like an army of steamrollers. It’s been erased like a blackboard, rebuilt, and erased again. But NAEP has marked the time. This assessment, this trend line, is a part of our past, Ray. It reminds us of all that once was good, and that could be great again. People want the trend line, Ray. People definitely want the trend line.
Charlie: OK. You can call me Ray. But aren’t people skeptical?
OJ: Ray Ray, you just tell them what they want to hear hear hear hear hear. You need to tell em tell em tell em What they wanna hear.
Charlie: Sure, people will hear what they want to hear, believe what they want to believe; but this is psychometrics, measurement, facts…
OJ: It’s statistics, son. Facts are stubborn things, but statistics are pliable.
Charlie: Pliable, yes. But, if the trend line were broken, how could you fix it? You tell us that in the national sample students taking the test on paper performed 4 percentage points better on each item than those taking the test on computer. That sounds like a big difference.
How does that compare to the p-value difference normally found between a top-performing state like Massachusetts and the national average or with states near the bottom of the list?
OJ: Right, in State A there is a 5-point scale score difference …
Charlie: Wait. Sorry to interrupt. No, I am asking about the national p-value difference.
OJ: Mindset. You start with the mindset that the trend has been preserved and that you need incontrovertible evidence to prove that it has been broken. The rest is just statistics.
You tell me that there is a 5 point difference between a state’s performance on paper and computer. You think, “Damn, five points on NAEP is huge!” NAEP can go 30 years without changing by five points.
But, could a difference that large happen by chance? Maybe not too often, but 5/100 times, 1/100 times, 1/1000 times – you see where I am going with this?
Charlie: But what about Power? With a paper sample of only 500 students…
OJ: Power! We can take a year to report results and nobody bats an eye. We can post cute little Twitter surveys while people are waiting and people ‘like’ them.
We can take the time we need to prepare the message. When I worked for a state we were taken to court and lost when we wanted to take two days to prepare a memo before releasing results.
We can bury you with videos, charts, graphs, data tools when we release the results.
That’s the only power I need.
Charlie: People will want to know what happened to the trend line.
OJ: We are reporting that nothing happened to the trend line, Don. Reports that something hasn’t happened are always interesting to me, because as we know there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know.
Charlie: What does that mean?
OJ: Exactly!
Charlie: The trend line. Was it broken?
OJ: Son, we live in a world that has trend lines; and those trend lines need to be maintained. Who’s gonna do it? You? You have the luxury of not knowing what I know – that misrepresenting performance of an individual state, while tragic, probably saved lives; and my existence, while grotesque and incomprehensible to you, saves lives.
You want me maintaining that trend line!
YOU NEED ME MAINTAINING THAT TREND LINE!
Charlie: Well, thank you OJ. That’s all the time we have today. We are all looking forward to the release of the 2018 NAEP results later this fall.
OJ: We’ll see.
Embrace the Absurd 