When the headline of an article on a respected website by a respected author uses a word like “dumber” as in “Test Results Show Americans Are Getting Dumber” it gives me pause. My first thought was that perhaps the word choice was just the product of an eager headline writer trying to get clicks. But no, there it was boldly proclaimed in the first line of the Chad Aldeman article:
“There’s no way to sugarcoat it: Americans have been getting dumber.”
If the goal was to get attention, it certainly got mine.
In this day and age we don’t throw around words like “dumber” loosely – at least outside of presidential campaigns. In polite society and academic debate seldom is heard a discouraging word as we euphemize our way through meetings and presentations in a way that euthanizes spirit and shuts down the type of critical give and take that makes ideas stronger and made America great.
As a child of the 60s and man in my 60s, I’ll confess that it took me a while to accept the idea that there were no dumb people, only dumb ideas. And just as I was getting used to that, we moved onto there are no dumb ideas and its corollary, there are no stupid questions. I was on board with the spirit of that concept until far too often I found myself in the company of people who took that idea literally. Now, it seems, we have come full circle, as we are wont to do; or perhaps, not quite full circle. I haven’t read yet whether ideas are also getting dumber again as Americans have been getting dumber, but one would think that the two are positively correlated.
So, what are we to make of this argument that over the past decade or so, Americans have been getting dumber?
For fun, let’s not stop at just the last decade when NAEP scores started to level off and fall after their initial rise following the introduction of state NAEP and the subsequent NCLB bump. The decline of the US on international assessments goes back further than that; and we know all about state achievement standards and the race to the bottom. On top of that, we’ve lived through decades of declines in college admissions test scores that resulted in recentering, renaming, revising, and ultimately, reinventing the SAT as a high school state assessment weakly to moderately aligned to the CCSS.
That’s a whole lot of getting dumber to process; so, let’s dig in, shall we.
Dumber is Relative
By definition, the word dumber and the concept of getting dumber are relative, or at least comparative, in that they require a point of comparison. Like virtually everything else that we do with educational assessment, the type of comparison that we make might be norm-referenced, criterion-referenced, or some bastardized combination of the two.
At first glance, the comparison in this case seems decidedly criterion-referenced. Almost across the board, Americans are scoring lower on scales than they did previously, with the possible exception of scales measuring weight.
A bit of a cognitive leap is still necessary, however, to go from a lower NAEP, PISA, or TIMSS score to a conclusion that Americans are getting dumber. Those tests, after all, are not designed to measure how dumb Americans are and none report results on a dumbness scale. If they did, a lower score might be desirable.
As I have written previously, it’s difficult to describe exactly what is being measured by NAEP or state tests and we have never given a name to the construct being reported on any of their scales. Personally, I think of the concept being measured as Proficiency in the particular content area being assessed.
Perhaps then, we can say with confidence that a lower score indicates that, on average, Americans are getting less Proficient in Reading and Mathematics – as defined by NAEP and other tests – acknowledging that the concept of Proficiency is also not very well defined.
But is less Proficient necessarily the same as dumber?
Let’s quickly review the other common comparisons involving Proficiency which might lead one to conclude that Americans are getting dumber.
- Americans could be perceived as getting dumber if we maintain our level of Proficiency while people in other countries increase theirs. For a while that was the case with things like the percentage of people attending college and earning degrees; and it has also occurred to some extent on international tests.
- Americans could be perceived as getting dumber even if we increase our level of Proficiency, but the level of Proficiency needed to function or to be considered college-and-career-ready increases. It’s like inflation. That $5 bill just isn’t worth the same as it used to be last year or when we were kids.
- Americans could be perceived as getting dumber if the skills in which we are Proficient simply are not as relevant as they used to be. What skills are needed to survive or thrive in society today, tomorrow, ten years from now?
Less Proficient In What Exactly
When NAEP scores are released in a few hours, undoubtedly there will be some changes from 2022; some scores will be higher and some will be lower. Inevitably, the changes will be modest in terms of the number of points moved on a 500-point scale, although all but the smallest changes for the smallest subgroups in the smallest states will be deemed statistically significant.
But what does a 3-, 5-, or even 10-point difference, on average, mean in terms of what kids know and can do?
What exactly is it that kids, on average, could do better in 2019, 2010, or 1990 than kids today? That’s a difficult question to answer, much too difficult to answer based on test scores alone. Yes, achievement level descriptions and exemplar items mapped to scale score points provide a bit of information, but not in enough detail and to a level of precision that helps us interpret change in scores over the past decade.
What about the difference between what kids performing at the 90th and 10th percentile can do? How well can we describe the performance difference between students at those two levels beyond points on a scale, achievement levels, or heaven forbid, effect sizes?
Even if we can pull that off, however, how relevant is the performance of kids at the 90th and 10th percentiles. We know that the kids at the 90th percentile are Proficient or above, even in relation to the ever aspirational NAEP Proficient benchmark. And I’m going out on a limb here and predicting that information about what kids at the 10th percentile on the 4th and 8th grade tests can do is actually much less important than information about who those students are and the quality of the educational opportunity they have received.
What if we narrow our comparison to kids at the Proficient cut score and kids at the first quartile (i.e., 25thpercentile)? What does that gap look like in terms of skills, what the kids can and cannot do.
And what of that cohort of kids, who on average, scored 2, 3, or 5 points higher or lower in 2024 than in 2022, 2019, or 1990?
What can they do and what are they unable to do proficiently; and how worried should we be about that?
Is It The Same Boat?
NAEP prides itself on maintaining its precious trend line, and there are clear benefits to doing so, but in this case that trend line might just be a double-edged sword.
One of the downsides of maintaining the same scale since the mid-1990s is that it is the same scale that was used in the mid-1990s. A lot has changed since then. Seriously, how concerned should I be that kids today are not Proficient on things that we thought were important in 1994? Are they not able to get the time to stop flashing 12:00 on a VCR?
Of course, we all understand that such a literal interpretation of maintaining the same scale and preserving the trend line is more theoretical (or theatrical) than a reflection of reality. There have been many small changes in what has been measured and in how it has been measured since the 1990s – each individual change carefully studied and deemed not significant enough to require a new scale or break in the trend line.
But those of us who received a well-round liberal arts education are by now thinking of Theseus and his famous ship.
Even if the scale is the same, does a test score, particularly a mean scaled score, really allow us to compare the performance of students in 2024, 2019, 2010, and 1995 in a meaningful way?
My answer to that question is an unequivocal, NO!
So, let’s not waste time with such questions. As interesting as they may be, comparisons to the past are not all that important or relevant. We need to continue to look forward to the future.
There will be plenty to chew on even if we just focus on the 2024 and 2022 scores and what together they tell us about what kids in various states, subgroups, and levels of achievement can and cannot do. NAEP can also provide us with a lot of additional information to help us put that content/skill information in context. NAEP is good like that.
Tell us what kids can do and how well they can do it.
Tell us what you tested that they cannot do well.
Tell us what you know about their opportunity to learn.
With that information in hand we can begin to determine just how worried we should be and to figure out what we need to do next, what our highest priorities should be.
Image by Rob van Ruiten from Pixabay
Embrace the Absurd 