Work the Problem

With the Olympics coming to an end, I began browsing through the rest of the Peacock app and came across the recently added 1995 film, Apollo 13.  Since first renting the film at Blockbuster as soon as it went to VHS, I must have watched it two dozen times. For some reason, I feel a close connection between my career in large-scale testing and Apollo 13, specifically the scene below where the spaceship explodes and all hell breaks loose.

[Aside: Yes, I know it sounds like there should be therapy involved here, but instead I have a blog. After Apollo 13, Titanic is the film I most closely associate my large-scale testing career, but I am saving that discussion for Volume 2 of my memoirs.]

For some reason, that pivotal scene in the movie always reminds me of the day in late summer or early fall (or sometimes later fall) when state test results are released.

People frantic at the DOE, state house, and testing company as they look at the pages (now screens) of test results in front of them. This is showing a “quadruple failure” that can’t happen – it has to be an “instrumentation” problem (i.e., a problem with the assessment). I’ll double-check the equating and get back to you.

Meanwhile, at the local level, I see district administrators, principals, and teachers scrambling to deal with very real situations in front of them.

We have third graders who can’t read and eighth graders with little fluency in mathematics who are wandering aimlessly on the journey to algebraic thinking. Our dropout rates have stopped rising at the high school, but chronic absenteeism increases every year. Our mobility rate is exceeded only by our churn rate for new teachers. If I can’t get these three grants renewed, we’ll have to cut the after school program.

Each time I watch the movie, I dream of the day when our educational policy or assessment counterpart to flight director Gene Kranz will stand up and deliver lines like:

“…those guys are talking about bangs and shimmies up there, it doesn’t sound like instrumentation to me.”

“Let’s work the problem people. Let’s not make things worse by guessing.”

“Let’s look at this thing from a… um, from a standpoint of status. What do we got on the spacecraft that’s good?”

Defining The Problem First, Then Work the Problem

The first and most important step to solving a problem is to correctly identify and define the problem that must be solved. I have long argued that when designing large-scale assessment programs we never successfully complete this very first step, and most often ignore it completely – perhaps subconsciously, perhaps by design, perhaps out of fear of being forced to acknowledge the enormity of the situation.

The problem that we are trying to solve is more complex (i.e., less well-defined and constrained) than the safe return of three astronauts in a severely compromised spacecraft. Ours is a wicked problem – “a problem that is difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often difficult to recognize”; and where “the effort to solve one aspect [of the problem] may reveal or create other problems.”

Let that sink in.

Is there any phrase that describes the recent history of K-12 large-scale assessment and accountability as a tool of equity and education reform better than “the effort to solve one aspect of the problem may reveal or create other problems.”

Which makes it all the more critical that we at least attempt to define and understand

• the large problem to be solved (the mission of public education in a democratic republic),
• the specific aspects of the large problem that each of the public education systems and subsystems is designed to address (those systems and subsystems include everything and everyone associated with fulfilling the mission of public education) and
• the support role that assessment can play facilitating the functioning of each of those systems and subsystems.

Daunting?

We know that.

No aspect of public education, including assessment, is for the faint of heart.

Unleashing the Power of Assessment – A Single Drop of Water

Assessment cannot exist on its own. We stare with awe at a well-crafted test item and admire the beauty in the simplicity of a test characteristic curve, but the fact remains that there is no such thing as assessment for the sake of assessment. Assessment owes its existence to that which is being assessed.

As assessment specialists, it is only when we acknowledge and accept the infinitesimal space that assessment occupies within the infinite void that is the mission of public education that we can unlock the true power of educational assessment.

Assessment is a small, but critical, component of every subsystem within the public education system (academic, fiscal, management, administrative, etc.). Assessment provides information necessary to monitor and evaluate the functioning of each subsystem. Some data may be collected and evaluated continuously. Other data collected continuously and evaluated periodically. Other data may be collected periodically. All of the information derived from assessment serves the single purpose of ensuring the optimal functioning of the subsystem to which it is connected.

When each of the subsystems functions well there is an increased likelihood that the system will function as designed. When the system functions as designed there is at least a chance that we are moving forward in the effort to solve the wicked problem.

At the very least, when the system functions as designed we are in a better position to evaluate the efficacy of the design of the system (e.g., it’s more difficult to evaluate the effectiveness of a vaccine if people refuse to take it.)

The Nature of Assessment and the System – Finding Balance

The relationship between assessment and the subsystem to which it is attached can be mutualistic or parasitic. We know that instruction can flourish when it includes the appropriate use of formative assessment processes. We also know that the inappropriate use of assessment can crush the soul of a school system. And we cannot ignore the existence of other parasites that attach themselves to assessment, ultimately destroying both the assessment and its partner system.

How then do we ensure a mutualistic relationship between assessment and the subsystem that it supports? The answer is balance. Within each subsystem, the amount of time, effort, resources, responsibility, etc. attached to assessment and the rest of the subsystem must be in balance.

Similarly, the larger system functions optimally when there is balance throughout the system – when each subsystem is functioning as designed, fulfilling its assigned role – no more, no less.

There is a difference, however, between the mutualistic relationship between assessment and the subsystem to which it is connected and the relationship between the various subsystems within public education and the overall education system. Whereas, the parties in a mutualistic relationship cannot thrive without each other, the various subsystems within the overall system of public education may be quite independent from each other – even though all contribute to the overall mission of the system.

Understanding this difference is critical to prevent us from falling into the trap of trying to view the larger system from an assessment perspective and talk about solutions based on a “comprehensive assessment system” or a  “balanced assessment system.” Attaining a state of balance within and across subsystems, as described above, is a desirable goal, but there is, in fact, no single or unified “assessment system” that can be identified and declared balanced or unbalanced across the entire system. Searching for a “balanced assessment system” or attempting to design a solution around a “balanced assessment system” will prove to be a fruitless and frustrating exercise.

Success Is Not Possible – Failure Is Not An Option

The Apollo 13 mission was described as a “successful failure” in that the original goal of landing on the moon was not achieved, but in the wake of the emergency the astronauts were returned safely to earth. We are not in a field that offers such well-defined opportunities for success.

We have entered a field and taken on a Sisyphean task where ultimate success is probably not possible. There will always be more problems to overcome in public education. At the same time, we know that we must operate with the understanding and constant reminders that failure is not an option – making it more difficult to celebrate or even find satisfaction in partial or short-term successes, although we must be sure to celebrate and find satisfaction.

Additionally, borrowing (liberally and probably inappropriately) from Nietzsche’s discussion of art there is the reality that because assessment “makes apparent much that is ugly, hard, and questionable in life; does it not thereby spoil life for us?” And our answer must be no. We don’t shrink “before a problem that arouses dread.” We work the problem because that is who we are and what we have chosen to do. We work the problem. We evaluate. We reset.

As I said, assessment and public education is not for the faint of heart.

Image by Gerd Altmann from Pixabay