What Has Stopped Us from Getting to Go on Balanced Assessment Systems
K-12 large-scale assessment generates a great deal of discord, but there are two statements regarding large-scale assessment on which there is near universal agreement:
- The importance and impact of large-scale assessment on K-12 education is disproportionate to its utility in improving student learning.
- The best way to rightsize large-scale assessment is the development and implementation of high-quality balanced assessment systems.
There we have it. The problem and solution stated in two succinct sentences, short enough to be a tweet with a few characters left over for an appropriate hashtag and emoji.
If we can put a man on the moon…well, we could put a man on the moon 50 years ago…and we might be trying to put some women and men on the moon again in a few years … wait, what … sorry.
If we can put a man on the moon, why can’t we build a high-quality balanced assessment system?
Surprisingly, the answer is not the overuse of ambiguous terms and the lack of a clearly defined construct (the usual culprit in educational assessment). The definitions and descriptions of balanced assessment systems have remained remarkably consistent over the past three decades, and state standards provide as good a stand-in for a construct as we are going to find.
Historically, and even as recently as the turn of the century, the answer might have been the vast assessment void below the level of standardized, large-scale tests. At this moment, however, there is clearly no shortage of formal assessment instruments and procedures of all kinds below the large-scale level.
The lack of quality of one or more of the component parts is also not the primary concern. Yes, large-scale tests remain under constant attack for their inherent shortcomings and fundamentally flawed foundations. True, there are some who will always decry the quality of interim assessments and question the motivations and parentage of those who promote them. And formative assessment will always seem just a little too squishy to those of us trained in assessment and educational measurement, but that’s our problem.
Over the past twenty years, there have been tremendous improvements in the design, development, and use of the processes and instruments commonly referred to as formative, interim, and large-scale summative assessment. There has also been progress in ancillary forms of assessment such as portfolios, exhibitions, performance tasks, and assessments of non-cognitive skills. Advances in technology and theory have made assessment options that were unwieldy and impractical only a few years ago commonplace; and I have no doubt that trend will continue.
I will even betray my better judgment and suggest that the lack of assessment literacy among stakeholders at all levels (i.e., teachers, administrators, assessment vendors, policy makers) is not the primary barrier to balanced assessment systems. There is no question that increased assessment literacy is a prerequisite for the successful implementation and use of balanced assessment systems. Like a drivers’ license, which is most useful when you have access to a car, however, assessment literacy will be most useful when stakeholders have access to balanced assessment systems.
Why then has it been so difficult to implement balanced assessment systems?
I propose that the biggest obstacle has relatively little to do with our knowledge, skills, and abilities related to assessment, but rather is almost wholly attributable to our total lack of attention to, understanding of, or perhaps even awareness of systems and systems engineering.
In short, we continually fail to realize that a balanced assessment system is an entity unto itself – separate and distinct from any formative, interim, and large-scale assessment programs or instruments it might comprise.
A Path Forward for Building and Managing Balanced Assessment Systems
That we have missed this critical mark on balanced assessment systems should come as no surprise. As David Conley (2014) noted in discussing A New Era for Educational Assessment, we have “a historical tendency to focus on bits and pieces.” That is true of the history of assessment development that Conley described, but is also true of the structure of education in the United States, in general, and is certainly true of our training in higher education. We are quite good at breaking things down into their component parts, but not quite as good as putting them back together again. (Wow, I just had a childhood flashback there that really explains a lot.)
Consequently, most efforts at building balanced assessment systems to date can best be described as DIY projects. We look to individual school districts in search of effective practices that can serve as exemplars for other districts. But where do I turn for expertise on designing and implementing balanced assessment systems? I know whom to call (or read) for advice and support on large-scale assessment systems, formative assessment, and even interim assessments. I am at a loss, however, when I try to find similar expertise on balanced assessment systems.
There are some companies specializing in large-scale or interim assessment who are now directing some of their efforts toward building top-to-bottom solutions that might be regarded as self-contained balanced assessment systems. There are also states, like Louisiana, who have devoted a great deal of resources to vetting instructional and assessment materials in an effort to help schools build coherent systems of their own. Such efforts should be applauded. However, just as it is said that we cannot improve teaching effectiveness by focusing on improving one teacher at a time, neither can we hope to develop balanced assessment systems one school district or one assessment system at a time.
If we want to produce balanced assessment systems then we need to begin to focus on what it takes to develop balanced assessment systems; that is, how to effectively manage the systems of formative, interim, and large-scale assessment which reside within systems of schools and districts and so on. We need projects devoted to building balanced assessment systems from the ground up.
Returning to landing a man on the moon, systems management or systems engineering (i.e., effectively controlling and managing the concurrent design and development of independent systems that had to function together flawlessly) has been called The Secret of Apollo. In a 2011 EdWeek commentary, I suggested that achieving the same level of success in educational assessment would be just a bit more challenging than those trips to the moon and back. If we are serious about undertaking the challenge of developing balanced assessment systems, however, we must do it the right way by actually funding projects focused on building balanced assessment systems.
At the same time, we must also rethink graduate programs in educational measurement and how they can help produce a generation of professionals who have a better understanding of assessment systems. Last summer, an NCME session, Teaching and Learning “Educational Measurement”: Defining the Discipline?, addressed the question what is needed to be an expert in educational measurement? One of my takeaways from the session was that there is barely enough time and definitely not enough resources available in graduate programs to cover the discrete knowledge, skills, and abilities needed to produce graduates who might reasonably be considered experts in educational measurement. Perhaps the development of assessment systems engineers will have to be a separate specialization within the field.
It is said that every system is perfectly designed to get the results it gets. Our lack of progress on balanced assessment systems certainly bears that out.
Embrace the Absurd 