My reengagement with the world of educational conferences last month had many upsides. It was great to reconnect with old friends and old ideas, seeing how some of each were relatively unchanged since our last encounter, but others had grown significantly in the intervening years. There was also the joy of being exposed to new people with new ideas, fodder for countless new blog posts.

But as Robert Kraft noted during The Greatest Roast of All Time: Tom Brady, at every family reunion there’s that one person you’re trying to avoid. The same principle held true for me as I attended sessions and receptions at NCME and NEERO last month. I knew that at any moment, I could come face-to-face with my nemesis – the Joker to my Batman, the Lex Luther to my Supergirl, the truth to my president. Now I know that several of you are so vain, you probably think this post is about you, but fear not (yet), I’m saving those stories for the podcast.

No, this specter didn’t assume human form. It was amorphous, ill-defined, and nebulous, but omnipresent. And while at the conference, I knew that at any moment, with the next sentence out of someone’s mouth, or each time a button was pushed to advance the slideshow, I might hear or see the phrase:

THEORY OF ACTION

Three little words that for decades haunted my every waking and sleeping moment – coincidentally, several of both occurring during multi-day staff meetings. The sleepiness exacerbated, perhaps, by my TOA drinking game – take a sip from my bottle of “Diet Coke” every time a colleague said theory of action.

It’s not that there is something inherently wrong with the concept of theory of action per se. It’s not even that TOA has become a meaningless buzzword like innovative, comprehensive, balanced, or aligned. But much like interim assessment, growth, and understanding how properly to consider use and consequences in validation, in the context of  educational assessment and accountability the problem with TOA is in the poor application and execution of a good idea.

In many ways, I view the current state of Theory of Action in educational assessment like TikTok accounts. Everyone is supposed to have one, but it’s not quite clear what they’re supposed to do with it. TikTok accounts can be entertaining, sure, but the really good ones are few and far between. I’ll stop short of saying that TOA pose a national security risk, but like TikTok accounts, in practice, it’s often not clear what real purpose they serve – except perhaps for making a lot of money for a small percentage of users.

For those unfamiliar with the concept, a TOA “describes how an organization intends to work to create the outcomes it wants.” (Nordengren & Javurek Andres, 2022). My colleagues at the Center for Assessment offered this definition in a 2016 paper:

A theory of action outlines the components of the system, while clearly specifying the connections among these components. Most importantly, a theory of action must specify the hypothesized mechanisms or processes for bringing about intended goals…the theory of action should describe how the particular clear goals will be achieved as a result of the proposed…system(s). (Marion, Lyons, & D’Brot 2016))

As implied above, it’s hard to argue against anything in either of those statements. So, what’s my beef. Why does the mere mention of TOA drive me to drink?  In the remainder of this post, I’ll lay out my four main concerns.

• The Magical Number
• An Air of Detachment
• The Theory is Mostly Theoretical
• The Wrong Question

The Magical Number

 A TOA invariably includes a diagram showing the components, connections, and mechanisms mentioned above and the relationships among them. Sometimes, that diagram might be the main artifact of the process of developing a TOA.

Also almost invariably, these complex graphic representations of TOA violate my “rule of 10” which states that a diagram must contain fewer than 10 arrows.  

There are exceptions, of course, but they are exceedingly rare. I did see one nice TOA in Philadelphia with only five boxes and four arrows, which is probably bare minimum, even for a birds-eye view TOA. Especially in our field. I wouldn’t want to come out of a weeklong meeting with this TOA:

Some might argue that diagram is a fair representation of the implied TOA behind NCLB in 2001. That’s debatable, but the simplistic diagram most definitely reflects the prevailing TOA for test-based educator evaluation systems in the 2010s that tended to classify 99% of teachers as Effective or Highly Effective.

So, more than two arrows, but fewer than 10.

Why fewer than 10?

For that, I go back to George Miller’s 1956 article, “The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information,” described on Wikipedia as “one of the most highly cited papers in psychology.”

I get it, education, in general, and improving education, in particular, is complex, with lots of moving parts and interconnected players. And I understand that when you’ve hired an expert TOA facilitator – paid by the arrow – and convened a large group of stakeholders – pulling them away from their jobs for a day or more – you’re not going to walk out of that meeting with a diagram containing a handful of boxes with a single unidirectional arrow between each.

Still, when I see a diagram with 10, 15, or 20+ arrows on it, I know that the likelihood of getting from the beginning to the end is nil.

An Air of Detachment

My second concern is with the name: Theory of Action. For me, it exudes an academic air of detachment. It’s not a plan of action, a course of action, a call to action, or even an overconfident declaration that I’m gonna get me some action.

A TOA says that I’m not going to get my hands dirty.

It’s simply a theory of action.

But wait, where exactly is the theory?

The Theory is Mostly Theoretical

My third concern is that when I look at most TOA, even ones that are highly regarded, I don’t see a lot of theory behind the components and connections and mechanisms. On a good day, I may see a statement proclaiming that the actions in the TOA are grounded in learning theory – an understanding of how students develop and learn.

Less common is to see a clear statement of the Theory of Change or Theory of Improvement that is supposed to be the catalyst for the Theory of Action.

Even less common is to see a theory of action reflected in a theory of action. What does that mean? Most TOA contain a list of actions but little evidence of an understanding of the psychological, philosophical, or social theory dealing with the processes that make action(s) likely to occur at either at the individual or the system level.

If it’s starting to feel like we are out of our depth, in over our heads, that’s good. Remember that feeling.

Where does a state assessment program or state accountability system fit in?

The Wrong Question

Which brings me to my fourth concern. Framing the issue in terms of what is the theory of action for implementing a state assessment program is asking the wrong question. It’s fundamentally and fatally missing the point.

A state assessment program is not a component, a connection, or even a hypothesized mechanism or process for bringing about change.

At its best, a state assessment program is a tool to provide information called for by one of the other boxes in a theory of action; for example, obtain credible, comparable information about student achievement. (Let’s say if, hypothetically, one of the components in your TOA was to fund a massive program called Title I for the purpose of improving the achievement of disadvantaged students then you might implement a state assessment program to collect information about student achievement to evaluate the success of that program.)

If a statement like the one above is not a component of the TOA supporting your Theory of Change or Theory of Improvement, then you don’t even need to consider a traditional summative state assessment program.

If your TOA includes a component related to providing tools to support districts in generating credible, timely, and/or actionable information about student achievement then we can talk about a different kind of state assessment program.

Basically, the same holds true for state accountability systems. I’m 99.44% of the way to rejecting the viewpoint that an accountability system is a mechanism for change and improvement. It’s a measurement instrument that at its worst can be a significant barrier to change.

Pulling it All Together

These are wicked problems that we are trying to solve, decades (perhaps centuries) of deeply engrained policies and practices that we are trying to change.

It’s been 60 years since The Coleman Report on the equality of educational opportunity was authorized by the Civil Rights Act of 1964. It’s been nearly 40 years since Coleman wrote about the theory of social action and the dangers of trying to isolate individual actions from societal actions or the systems within which they are embedded.

One of the reasons that a good TOA has so many arrows is BECAUSE the problems we are trying to solve are wicked problems and the actions that are needed are so complex.

One of the reasons that a complex TOA leads to a sense of detachment from responsibility BECAUSE the problems are so big.

I still firmly believe that we will solve these problems when we (the big we) decide to do so.

And a well-constructed TOA may be a useful weapon in guiding the conversations needed to influence that decision. A picture’s worth a thousand words.

But it won’t start with a state assessment program or state accountability system.

Header image by Kevin Sanderson from Pixabay