It’s been a little over two years since Midnights (NAEP Edition) became my most popular blog post of all time – and by an amount well beyond any margin of error. The post celebrated the rare conjunction of major events in two of my worlds; the releases of Taylor Swift’s Midnights and the 2022 NAEP results. A lot has taken place since that glorious weekend in October 2022.
Well, at least for Taylor Swift.
Since October 2022, Taylor won the Grammy Award for Album of the Year (her 4th), released a new album with 31 songs, re-released two of her older albums with a ton of additional songs, launched the Tour to End All Tours performing close to 150 shows at sold out stadiums across 5 continents, released a movie of that tour that was a hit in theaters and then on Disney+, was named Time magazine Person of the Year, and on top of all of that , fell in love and won the Super Bowl in her rookie season.
NAEP results, meanwhile, are still very much feeling ’22.
Since October 2022 we’ve been staring at one data point. And although we are more than Ready for It, all that we can do is to Tolerate It until the powers that be deem that the time is right to release the NAEP 2024 results at some point later this fall.
When those ‘24 results are finally released we will have another data point. A second data point in which NAEP results will
- best-case scenario, confirm what state tests and interim assessments have been telling us about the recovery from the pandemic for the past two years or
- worst-case scenario, paint a completely different picture leaving everyone confused.
To be fair, there is a likely in-between scenario in which that second data point will do what NAEP results have been doing since state NAEP was introduced in the 1990s; that is, confirm the results in many states (give or take a constant) and call into question state test results and achievement standards in a few others. And that type of monitoring is a primary reason that state NAEP became mandatory under NCLB.
When that scenario has played out in the past, often it has been a net positive, either by serving as a wake-up call to the state and/or by bolstering the case of reform-minded folks in the state trying to convince their legislature that something needed to change. There was also that time that the discrepancy between NAEP and state test results led to the development of the Common Core State Standards. I’ll leave you to your own opinion on how that turned out.
Before moving on, let me circle back to my best-case scenario. Policymakers and psychometricians have a really hard time accepting (or even wrapping their head around the idea) that the best-case scenario for a biennial assessment like NAEP or annual tests like those administered by the states is that
they tell you nothing that you didn’t already know before – even when what you knew was that performance was not very good.
We don’t want to see cases where there are discrepancies, where NAEP says that the state is going to hell in a handbasket and the state thinks that everything is hunky-dory and is conveying that message to its parents, students, and teachers. It’s never easy to receive poor test results; but the situation is not nearly as bad when everyone agrees that that performance is bad and is already working toward a solution.
Unfortunately, there will always be discrepancies and so NAEP will continue to serve a purpose, as will annual state tests, and even interim assessments when they are used for progress monitoring as nature intended. These tests serve a purpose, limited though it may be. They are what they are, what they were, and what they will be. Nothing more. Nothing less.
I lump all of these tests together because in the big scheme of things, NAEP’s lone data point since 2022 isn’t all that different from the 3 data points produced by state tests administered in 2022, 2023, and 2024 or even the 9 data points generated from interim assessments administered three times per year over the same time period. True, 9 is more than 3 which is more than 1, but it’s still only 9.
And 9 data points is nowhere close to the approximately 150 instructional days in the school year. I have my doubts that even 9 data points are enough to support the inspirational claim at the end of the NAGB 2020 video on The Evolution of NAEP
As we look to the future, NAEP will continue to serve as the national yardstick that informs education policy, practice, and measurement.
My purpose here is not to pick on NAEP because all of us involved in testing at the state or even district level have made similar statements about our own assessment programs. And we all suffer from the same lack of data. The question, therefore, is where we are going to find the additional data that we need. The answer, more testing. No, of course, more testing is not the answer. It’s pretty much never the answer.
Taylor, please show us the way
For the answer, and a peek at our assessment future, let’s look to the past and present of Taylor Swift, and to the approximately 150 life-changing concert experiences that she delivered over the past two years via the Eras Tour
Prior to the current Eras Tour, there were only a limited number of ways to collect data from a Taylor Swift concert:
Option A. You could attend 30+ concerts across North America between 2008 and 2018, listen to the music, take lots of pictures, make new friends, collect way too much merch, and experience everything firsthand.
Option B. You could frantically search the internet hoping to find some poorly shot videos with snippets of music from the concert before they were taken down.
Option C. You could purchase the Tour Book online and listen to the music (CD or streaming) while flipping through the snapshots of tour moments.
Option D. You could wait for the concert video/documentary to be released on DVD for the early tours then on streaming services.
None of these options are ideal. Option A might get you a nice sample of data, but who has the time to attend all of those concerts while working a full-time job in educational assessment. Option B has promise, but it’s an idea that needed to be fleshed out and needed technology to support it – much like performance testing in the 1990s. Option C is probably the equivalent of current state and NAEP tests in terms of the scope of data collected and the quality of the experience. Option D gives us the kind of repackaging of data that we’ve come to expect from Sean Reardon and friends. It’s slick, entertaining, and informative, but it takes a while to produce and is just not quite enough to satisfy.
But all of that changed with the Eras Tour.
From the very first night and the very first sparkling bodysuit there were high-quality, professional photos posted on Twitter (X) and Instagram.
Then there were fan videos of snippets and songs, sets, and special effects that weren’t being taken down. Does she really dive into a hole in the stage?
Then there were videos of whole songs.
Next there were high-quality videos of long sections of songs posted to official sites.
And then came the live streams!
You could go online and find one or more influencers live streaming the entire concert! Any concert!! Each and every concert!!!
It was a little glitchy at times which could be frustrating, but it was real-time, and it was as if you were there (almost).
But that was just the beginning.
Next came the people who would scour the internet for you and live stream the live streams on a popular video sharing site. When one streamer sat down, paused to go to the restroom, got too jumpy or had somebody stand in front of the camera, the facilitator would quickly switch to another one. These sites became online concert communities. It wasn’t THE concert experience, but it was a really cool concert experience.
Was the experience perfect? Of course not. The videos were often videos of the video board, but that’s just like being there in person. Taylor’s voice was sometimes drowned out by people near the phone singing loudly and badly. And the behavior one or more of the facilitators may or may not have drifted from cringy to creepy to criminal, but the reaction was swift, and others filled the void. The show beyond the show went on.
So, how does this apply to us as we sit and wait for the 2024 NAEP results?
The data are there in the schools. Every day individual students and groups of students are receiving instruction, learning, and creating products to demonstrate their knowledge, skills, and competencies. It’s our job to get creative in working with schools and students to access what’s already being produced. In real-time. Unobtrusively. Efficiently.
We will have to relax and refresh some our thinking about security. We will have to recalibrate our thinking about technical quality; but that’s OK as long as there is enough signal to cut through the noise. For the first time, we’ll have to think, like educators, about quantity – about interpreting and drawing inferences from multiple data points collected over time. It will be messy and there will be unintended consequences; but such is life, and we will deal with it.
Most importantly, we will have to recognize that large-scale assessment has always been first and foremost a data collection activity. The data are there. The technology is there. We need to devote as much effort and energy to figuring out how to collect the data we need as we have been devoting over the past two decades to figuring out what data we need.
Image by jjcwhicker from Pixabay
Embrace the Absurd 