There was a time I was certain that I was going to be an astronomer when I grew up. That period fell between my fireman/baseball player phase and the musician/educator phase that lasted into my late-20s. It was the end of the Sixties and most everyone’s attention was focused on the heavens because there was this race to the moon and, frankly, the news here on earth was pretty depressing. Becoming an astronaut seemed too much of a stretch; but an astronomer, studying and understanding the moon, the stars, the universe, that was fascinating and within reach.
I have fond memories of those evenings, after dinner, sitting on my back steps with the girl “next door,” mapping the phases and position of the moon; or lying in that small patch of grass in her yard and pondering the stars and our futures while staring up at the equally small patch of sky that was visible between the houses and the trees.
I put next door in quotation marks because technically she lived one street over and two houses up, but our backyards were adjacent in that marvel of urban design that provided us kids (and there were lots of us in this Irish Catholic neighborhood) with connected dirt- and grass-covered backyards as far as a child’s eye could see, separated only by chain-link fences, eminently hoppable by even the most nonathletic among us.
I remember that Christmas morning when I received my first real telescope. It was a roller-coaster of emotions as the sheet-covered object sitting prominently in the middle of the room turned out to be a new party dress for my sister hanging on my mother’s ironing rack. But then I turned, and there tucked under the tree, was the box containing my Tasco telescope. Santa had delivered!
Now those evenings in the back yard were spent discovering the moons of Jupiter, the rings of Saturn, and exploring the craters of the moon. Seeing the polar cap on Mars and then almost being blinded by a streetlight when the front leg of the tripod slipped a bit. That summer afternoon, setting up the telescope on my grandmother’s porch on the second floor of our triple-decker; finding just the right spot between her tables of herbs and flowers to watch the solar eclipse play out on a bright white sun projection screen.
Looking back, I would say that my dreams of a career in astronomy peaked with the 8th-grade science fair in which I paired some grainy black-and-white photos of craters on the moon taken with a 35mm camera attached to my telescope with the traditional baking soda and vinegar volcano. I definitely had different career plans in mind long before I took my first physics class in tenth grade; nevertheless, I did sign up for the 3-year physics sequence, culminating in AP Physics.
Although my career went in a different direction, there are pleasant residuals from those early years of wonder. In August 2017, my sister and I paused in a parking lot to make a pinhole in a sheet of paper and project the solar eclipse onto the side of our mom’s car. In 2019, I celebrated vicariously through my daughter’s texts as she attended events in DC commemorating the 50th anniversary of Apollo 11. And I was still excited last month when I was able to zoom in on a photo of Venus and the Moon I took with my little Sony point-and-shoot camera and get a view of the Moon’s craters not all that different than the one I saw through my first telescope.
Yes, I do still use an actual camera on special occasions.
Aim for the Moon, If You Miss…
I majored in music, shifted to educational research and evaluation, and ultimately pivoted to large-scale assessment after calling a toll-free number I saw on the back of a score report. A circuitous and unplanned path, for sure.
But just how far from my interest in astronomy did I stray? How far afield is a career in state testing from staring off into the night sky and asking why? For the purposes of a midsummer blog post, not very far at all.
First, the astronomer and assessment specialist are far removed (one might say, light years removed) from the objects and phenomena they’re observing. It is only a small step to compare the distance between most assessment specialists and schools to the distance separating astronomers from the planets and stars they study.
Second, there is the unsettling fact that although experts in both fields know and understand a lot about the objects that we study, there is so much more that we don’t know. In astronomy, despite all of the advances made in our understanding of the universe, how it works, its structure and its origins since we believed, not so long ago, that the Earth was flat and the center of the universe, there is so much that we don’t yet know. Experts estimate, “the matter we know and that makes up all stars and galaxies only accounts for 5% of the content of the universe!” The remaining unknown 95% appears to be distributed between dark matter (27%) and dark energy (68%).
I think that 5% is also a good estimate of what we know and understand about schools, districts, and even state education systems based on test scores and other data that we collect and process. To be fair, 5% is not a bad ROI when you consider the percentage of state and federal education funding devoted to assessment.
Third, both the astronomer and large-scale assessment specialist make inferences based on observations of things, not as they are now, but as they were at a particular point in time in the past. We like to think that the gap between testing and our reporting and interpretation of test scores is similar to the second it takes for light to travel from the Moon to the Earth, or at worst, the 8 minutes or so it takes sunlight to reach us, even though with g-force building daily, it can feel like the 4-month gap between state testing and reporting (6 months for NAEP) is more akin to the 4 years it takes light to reach us from the nearest star.
For our primary purposes as large-scale assessment specialists, however, that time lag is really inconsequential. In the big picture scheme of things in which we operate, barring some cataclysmic event (e.g., a global pandemic), not a whole helluva lot happens to change schools, districts, and certainly not states at the macroscopic level in the time between testing and reporting.
Real change in schools and districts is much too small and occurs much too slowly to be captured by our assessment instruments within a single accountability cycle. For better or worse, that’s the way things are.
And truth be told, it really is not only better, but best, that way. Despite the very real sense of urgency fueling education reform, it would not be very good for anyone involved if schools, districts, and states were changing dramatically and in unpredictable ways over relatively short periods of time. Although we don’t want schools and districts to remain as unchanged as the Moon decade after decade, sitting stuck at the starting line like New York’s infamous racing pineapple, neither do we want them to behave like shooting stars that shine brightly for a moment before vaporizing in the atmosphere.
Our conundrum is that while schools and districts are stable and slow changing at the macroscopic level at which we in large-scale assessment study them, changes are occurring much more quickly and much more often within schools and classrooms, and to individual students. And we have been sucked into the black hole of trying to apply our traditional methods to understanding that change.
Which brings me to the fourth similarity between astronomy and large-scale assessment, or educational measurement, in general: the contrast between Newtonian physics and quantum mechanics. In the oft-cited metaphor, educational measurement and large-scale testing is analogous to Newtonian physics, with our measurements leading to inferences regarding the position and movement of large groups of students being the equivalent of classical mechanics and laws regarding the motion of macroscopic objects. (I guess if its analogous, this might be an analogy instead of a metaphor.) Either way, quantum mechanics, in contrast, with its focus on things that are very small, is analogous to the interactions that take place in schools and classrooms between and among students, teachers, administrators, etc.
As the metaphor/analogy continues, from our external perspective, large-scale assessment and educational measurement is quite good at explaining and describing the predictable performance of states, districts, schools, and large subgroups of students, and there is definite value in that endeavor. As a scientific field, however, we are lacking in our quest to be able to do the same for the performance of individuals resulting from interactions in the classroom. As a summary of our efforts in that area, this description of quantum mechanics seems particularly apropos:
Developed during the first half of the 20th century, the results of quantum mechanics are often extremely strange and counterintuitive.
We are very much in the same boat when we try to apply psychometric principles and statistical processes to understanding the performance of individual students and the learning that takes place during instruction. The more we focus on and devote our resources to developing tests to precisely and accurately measure where a student’s performance is at a given point in time, the less likely we will be to understand learning, instruction, and all of the forces that affect students and teachers in schools. Of that I am certain.
We need to apply different, new sciences to questions related to understanding the changes to individual students that are taking place within schools. At this point, I’m not sure what that science will look like, where it will come from, and who will lead the way. Data science, by its nature, is focused on big data. Computational psychometrics might be headed in that direction, but I’m not sure.
Formative assessment is but a critical first step in developing principled processes that appear to have the desired impact on student learning when implemented with fidelity under certain conditions. It will still require a giant leap to progress from effective formative assessment to a “measurement” science that effectively models effective instruction and learning within schools and doesn’t produce results that are often extremely strange and counterproductive.
Perhaps it will take scientists like my second cousin, who make the jump from large to small. A few years older than me, his interest in astronomy lasted a bit longer than mine. After bachelor’s and master’s degrees in Physics, he earned a doctorate in Physiology and Biophysics, and a distinguished career in academia and as a research scientist with the National Institute of Standards and Technology eventually led him to the emerging field of nanobiotechnology.
In the meantime, the rest of us can strive to do a better job applying the knowledge and tools at our disposal, namely the combination of educational measurement, assessment, and program evaluation. We can begin by recognizing and acknowledging that scale scores, achievement levels, and growth scores tell us little on their own. They need context. Ideally, one of two scenarios should be in place whenever we use large-scale assessment:
- Test scores (pre and post) are connected to a specific theory-driven, evidence-based educational intervention that has been implemented and is being evaluated.
- Quantitative test scores (ideally pre and post) lead to further quantitative and qualitative studies to identify potentially promising practices, processes, and policies that eventually lead us back to the first scenario.
If we can land a man on the moon, we can do that. It’s not rocket science.
Image by Okan Caliskan from Pixabay
Embrace the Absurd 
You must be logged in to post a comment.