All Systems Go!

Seriously, systems, that is comprehensive and balanced assessment systems: It’s time for you to go.

After much deliberation and a bit of soul-searching, I have come to the conclusion that it is time to retire the use of the word systems when discussing or describing educational tests or assessment programs.

I’ll admit that letting go of the notion of assessment systems or systems of assessment won’t be easy for me.

I was introduced to psychometrics and IRT in the mid-1980s at the University of Minnesota by the fine folks who founded Assessment Systems Corporation.

My first job in large-scale testing was with Advanced Systems in Measurement and Evaluation (ASME).

Prior to NECAP, the two assessment programs that defined my career were the Kentucky Instructional Results Information System (KIRIS) and the Massachusetts Comprehensive Assessment System (MCAS).

In the first half of my career, I wrote extensively about Comprehensive Assessment Systems.

Throughout the second half of my career, I watched as my colleagues at the Center attempted relentlessly to promote and explain the concept of Balanced Assessment Systems.

Recently, four people whose work and thinking I have respected throughout my career published a white paperand discussed the urgent need for Comprehensive and Balanced Assessment Systems.

Assessment systems are as much a part of me as the major body systems that keep me functioning from day to day.

Why, then, have I reached the conclusion that we need to step away from assessment systems?

Every system is perfectly designed to get the results it gets.

 We have all heard some version of the adage above. In whatever context it is used, it is generally not uttered to indicate that everything is fine or to compliment a job well done. Rather, it’s usually serves as the opening statement before the speaker, or writer, makes a compelling argument that major changes are needed to the way things are currently being done.

Such has been the case the many times over the years that I have heard a keynote speaker use the phrase to describe the current state of assessment in PK-12 schools. Formative assessment, assessment for learning, or assessment for instructional purposes is shortchanged because the system is designed to promote the use of summative assessment for evaluative purposes.

OK, fine. We need to change the way that we privilege large-scale assessment through our educational policies to ensure that the process of educational assessment can be more comprehensive and our use of educational assessment more balanced.

How do we do that? What does a comprehensive and balanced assessment look like?

Easier said (& said & said & said) than done

 As I mentioned, balanced assessment systems have been a priority of my former colleagues at the National Center for the Improvement of Educational Assessment for more than a decade. And they are not alone. Over the years, I sat in the conference room in the Center listening to presentations from individuals and earnest companies who stopped by to describe their latest innovative project or product, with its appropriate developmental codename, that would finally provide schools with a comprehensive and balanced assessment system.

 Despite the use of nothing but the best and most up-to-date academic jargon to describe the critical characteristics of balanced assessment systems, it’s still not really clear what a balanced and comprehensive assessment system would look like and how it would be used in an elementary, middle, or high school. Despite the promising portraits of how the latest advances in technology would platform and power these systems, the task of actually finding a fully functioning balanced assessment system in the wild has been daunting. Even designing and operating a prototype like Buckminster Fuller’s Dymaxion House or the successive versions of the General Electric Home of the Future unveiled over the last century has proven to be beyond our ken.  

Systems aren’t what they used to be

On top of the apparent difficulty in actually implementing a comprehensive and balanced assessment system, there is the unfortunate reality that time has not been kind to the concept of systems.

Back in the day, systems were cool. Systems engineering put people on the moon. Interstate highway systemsand public transportation systems would make all of our lives easier. And has anything better ever come out of North Carolina than the SAS system?  (Well, maybe Andy Griffith.)

Then systems started to lose some of their shine. I can’t be sure, but it probably started with the ill-fated attempt to switch to the metric system. The two-party system certainly hasn’t helped. Calling something or someone systematic was once a compliment, but now it has negative overtones – boring, methodical, not spontaneous or innovative. And I don’t have to explain systemic.

Do you want to be part of the system or part of the solution?

An assessment system by any other name

Why have we had such a hard time designing a comprehensive and balanced assessment system?

If it were just a matter of finding a more appropriate name to describe comprehensive and balanced assessment systems then I would say soldier on.

The real problem, however, runs deeper than the label. 

In a January 2021 post, Assessment by Any Other Name, Please, I argued that we were doing a disservice to ourselves and our field through the overuse of the word assessment to describe each of the components that might be part of a comprehensive and balanced assessment system: formative assessment, interim assessment, large-scale summative assessment. In that post, I recommended names that better reflected the purpose and use of each component, reserving the term assessment for the process of formative assessment. 

In this post, I make the same argument with regard to the term system. I think that through our use of the term assessment system, and our talk of comprehensive and balanced assessment systems, we are overplaying the interconnections between and the interdependence of the various types of assessment and testing conducted in schools for distinct and specific purposes, with the deleterious effect of making them codependent when they should be largely independent.

If we want to stick with the concept of systems, I would argue that formative assessment to inform instruction, interim assessment for progress monitoring, and large-scale summative assessment to collect data on instructional results for evaluative purposes, are analogous to the major body systems that I mentioned previously.

Our existence and day-to-day well-being are dependent on systems of systems, starting with the Big 5 that we learned about in elementary school: circulatory system, respiratory system, muscular system, digestive system, and nervous system.

Is there interplay among these body systems? Of course, there is.

Is there a benefit to thinking of the entire body holistically? Without a doubt.

 Would there be any benefit in attempting to describe the circulatory, muscular, nervous systems, etc.  as a single comprehensive and balanced system? I don’t think so.

Formative assessment is a complex system of processes, tools, and strategies used during instruction.  

Interim assessment for progress monitoring, likewise, is a complex system in its own right, built around regular testing for a particular purpose.

Large-scale summative assessment for evaluative purposes, when done well, is also a complex system. 

 We need comprehensive and balanced assessment. That is, …

• We need each of these types of assessment to be more comprehensive.

• We need better balance in how these types of assessment are supported and used in PK-12 education.

We don’t need to continue trying to force these three systems (as well as other types of assessment systems) to fit under a single umbrella called a comprehensive and balanced assessment system.

Image by Gerd Altmann from Pixabay