It’s hot, humid, and the middle of summer, so I’ll keep this brief.
The first and most important question that you should ask when presenting someone with state test results is:
Do you agree with these results?
That question applies, in particular, to administrators receiving school- and district-level results, but also to
- state education leaders and policymakers receiving state-level results from their state assessment or from NAEP,
- teachers receiving class-level results,
- to a somewhat lesser extent to a parent or student receiving individual results, and
- it even applies to psychometricians as they are producing the results, but that is a bit of a tangent to today’s discussion.
The reason that question is so important is that the most important function that state testing plays in education, above all else, is to be confirmatory. In an ideal, even if highly dysfunctional education world, results from state tests should corroborate what state leaders, school administrators, teachers, students, and parents already know about school performance and/or student achievement.
That statement applies whether test results (and student achievement) are consistently low, consistently high, increasing over time, or in the midst of a decade-long freefall.
Yes, this take on agreement is related to my long-held and oft-stated position that state tests (and NAEP by extension) should not be telling participants in the process anything that they didn’t already know. However, it also gets at a couple of principles that are even more fundamental with regard to the purpose of state tests and future of state assessment:
- You cannot solve a problem until you accept that there is a problem.
- “Every system is perfectly designed to get the results it gets.”
Opening Gambit: Agreement Before Explanation
There are any number of reasons why a policymaker or educator might dismiss test results, but one is always at the top of the list. As the majority of kids taking a NAEP Geography test might say, “denial isn’t just a river in Ohio.”
We need to break through simple denial and get to the root of any real or perceived disagreements.
We begin, therefore, by simply asking, given all of the other relevant information available to them, whether they believe that these test results accurately reflect student achievement. Note, that asking for agreement on student achievement being low is not the same as asking whether they expected test performance to be low.
Simply put, we have a different problem, or set of problems, to solve if there is disagreement over the accuracy of the results.
Having reached agreement on student achievement, we can entertain productive arguments over whether test results are low because the tests aren’t measuring the right things, there is inadequate or unequal opportunity to learn (which has its own set of issues to debate), the population is changing, the students are not engaged or in school.
The bottom line, however, is that we don’t even get to those arguments without that initial confirmation that the test results accurately reflect student achievement.
Endgame: Why Confirmation Is Key
Looking toward the future, the reason that it’s critical that we understand that the fundamental purpose of state assessment is confirmation is captured perfectly in the famous Edwards Deming quote:
“Every system is perfectly designed to get the results it gets.”
As long as we continue to believe that the function of state assessment is to measure student achievement then we will devote precious resources to build systems designed to produce better tests.
Since the days of Horace Mann, however, the purpose of state assessment was not to build better tests but to ensure that schools had a common understanding of content and achievement standards (for lack of better terms). Only when we understand that will we begin to focus our resources on systems designed to produce that shared understanding.
What would the framework for such a system look like? As a starting point, there are three critical steps:
1. State standards
The first step was state content and achievement standards. All states established those standards. A subset of states devoted adequate resources to disseminating and explaining the standards to key stakeholders. Only a handful of states, however, made the ongoing commitment to ensure that the standards are the focus of curriculum, instruction, and assessment in the schools.
2. State-supported assessment in schools
The second step is state support of assessment in the schools. That support includes both devoting resources to improving and sustaining formative assessment practices and to providing high-quality assessment instruments and services that can be used by schools. One often sees modeling or signaling high-quality assessment practices for schools as a purpose of state tests, but modeling is a far cry from providing access to tests, scoring, reporting, and data management services.
3. Reducing barriers to honest assessment and reporting of student achievement
If there is an “honesty gap” problem to be solved, the solution was never going to be to double-down on the test-based accountability policies and practices that were, at a minimum, already contributing mightily to the problem. Never mind adding educator evaluation to the mix. Commitment to steps 1 and 2 above is a starting point to supporting open and honest communication, but there are a lot of burned bridges that need to be rebuilt.
Is there still a role for an external state test in such a system? Sure, at least at the outset; But if we are doing things right, its footprint and significance will inevitably diminish over time. Which should have been the goal from the beginning. More on that in a future post when it’s a bit less hot and humid.
Embrace the Absurd 