DIY DEI

As I observe our field’s nascent do-it-yourself (DIY) attempts to embrace, understand, and enact the concept of diversity, equity, and inclusion (DEI) what I have seen can best be described as akin to the enthusiasm and curiosity of a baby discovering its feet for the first time.  And like the baby, I am sure that the process will almost certainly involve the field (and individuals within it) repeatedly putting its feet in its mouth as it learns how to deal with this newly discovered part of its being.

At the risk of belaboring and mixing metaphors, as the field attempts to fit the whole DEI foot into its mouth, I also think that in our haste to address this important issue a) we may be biting off more than we can chew,  and b) we risk missing both the big differences between parts of the foot such as the heel and the toes as well as the finer distinctions among the little piggy that went to market, the one who had roast beef, and the one who cried “wee wee wee” all the way home.

Finally, all of this is playing out in an environment in which there is no solid ground under our newly discovered feet. The historical and conceptual foundations (to borrow a phrase) not only of our field (i.e., educational measurement and assessment), but also of the P-12 public education system in which we function, and the society that created it are not simply being subjected to rigorous questioning (which is always a good thing) but are being rejected outright by many with little critical thought (which is never a good thing).

So how does a field that is known for constantly running in circles (After careful thought, I think multiple-choice items are your best option.) move forward on DEI – begin to crawl, take baby steps, and ultimately make giant strides?

Well, one step at a time, of course, but it will not be easy.

In particular, I believe that there are seven potential pitfalls that might cause us to stumble as the field grapples with DEI.

The first four are fairly high-level general observations about DEI and educational measurement and assessment.

Validity – It’s Always Validity

Regardless of where you stand in the never-ending debate on the interpretation of validity, there should be universal agreement that DEI, ultimately, is a validity issue. That is to say that some aspect of DEI is related directly to the inferences and claims that can be supported about student achievement based on a test score.

There are other aspects of DEI, such as the use of test scores for high-stakes admissions and accountability purposes, in which the relationship to validity might be more debatable, but for the love of Messick and all that is good, let’s restrain ourselves from engaging in that debate and pledge to keep the focus on issues related to DEI.

Rest assured, the next version of the joint Standards, will still begin with the word Validity.

Unidimensional, Multidimensional, Unitary, Hierarchical, Conditional

 One thing that DEI has in common with most of the constructs that we measure is that we don’t seem to have a handle on what it is even at the most basic level.

Should we view DEI as a unitary concept like validity?  Does Equity correspond to Construct Validity?

Or are diversity, equity, and inclusion three separate, but somehow interrelated, concepts or constructs?

If they are separate, but interrelated, in what way(s) does that play out? Is there a hierarchical structure? Can you not have equity without diversity and inclusion?

Is DEI, as a whole, a concept that is greater than the sum of its component parts?

To what extent is the concept of DEI or the constructs of diversity, equity, and inclusion (if they are constructs) unidimensional, norm-referenced, dependent upon some other factors.

And how does any of this relate to our measurement precepts and principles of validity, reliability, and fairness?

Tests and Testing

 With no intention of poking the validity bear (see above), there are clearly aspects of DEI which relate to testing and the use of test scores that exist above, beyond, and separate from concerns about the technical qualities of a test, its fairness, reliability, or the accuracy of inferences drawn about student achievement based on test scores.

Those DEI concerns may apply to the use of test scores even if we were able to reach complete agreement on the ability of those scores to support inferences and claims about student achievement, competencies, readiness for, or even likelihood to succeed at, the next level.

Why is it important that we are aware of and acknowledge this possibility? Because even if we do everything possible to improve our test instruments, administration, scoring, accessibility, etc. it is very likely that there will still be DEI concerns related to testing that do not have a measurement or assessment solution. We need to be prepared to deal with that.

Our Tests, Ourselves

DEI differs from other recent efforts to improve tests and testing in that it explicitly applies to the composition of the field and the processes of test construction as well as to the tests and test scores produced by that field and those processes.

Achieving DEI within the field can be viewed as a worthy end in itself, separate and distinct from the extent to which that state is a prerequisite for achieving DEI in tests and testing. Conflating the two will likely be a disservice to both because the criteria and methods by which they should be evaluated are clearly different.

My guess is that achieving DEI in the test construction process will be a much less well-defined, and consequently a much more challenging, task than achieving it either in the composition of the profession or in the use of tests and test scores.

The Other D-E-I: Decontextualization, Education, Individuals

My final three points in this post represent more specific concerns related to three different “D-E-I” concepts that must be understood if efforts to improve diversity, equity, and inclusion are to be successful. Each concern reflects an aspect of the push-and-pull, the tug of war that is inherent in large-scale testing.  I will cover Decontextualization in some detail.  The issues associated with Education and Individuals should be self-evident.

Decontextualization

An initial focus of DEI efforts within assessment has been on the decontextualization of items. I’ll admit that when I skim released MCAS Mathematics items from 1998-2000 and 2018-2021, the latter set of items appears to contain much less “engaging” context.  It would be a grave error, however, to assume that this outcome is the result of the deliberate application of a revered measurement maxim passed down from on high and etched in stone: Thou Shall Not Have Context In Items. Nearly a half-century ago when a younger Jim Popham sounded the alarm, the field eliminated items about yachts, backyard tennis courts, and suburban lawn care. It didn’t, however, eliminate context intended to engage students.

Increasing student engagement through test items and reading passages is not a new idea. Efforts to make tests, particularly large-scale tests, more engaging, however, are fraught with dangers.

• At Advanced Systems in the 1990s, we were criticized in some states because those liberal Yankees from New Hampshire were filling the test with propaganda about the environment.
• In Maine, heightened concerns about teen suicide led to the mobilization of the state police and national guard to recover and then redistribute grade 11 test booklets after the poem Richard Cory by Maine poet Edward Arlington Robinson was removed.
• The attempt by Massachusetts to include a passage from the Pulitzer Prize winning novel The Underground Railroad by author Colson Whitehead blew up in its face and we all became much more familiar with stereotype threat.
• In advance of Spring 2021 testing, states scrubbed their English language arts and mathematics tests of passages and items related to viruses, hospitalizations, and lost family members.
• Each year there are concerns expressed about the reading load of test items for students with disabilities and emergent bilingual students.

The list could go on.

To a large extent, decontextualization is a defense mechanism.

In my opinion, decontextualization is also the unfortunate outcome of a bad game of measurement telephone (you go talk to your friends, talk to my friends, talk to me) or perhaps just one more example of the inadequacies of the “train the trainer” model that we love so much. The source of the problem is construct irrelevant variance (CIV).

Personally, I view CIV as similar to Critical Race Theory (CRT) in this regard. It is a theory, concept, a set of ideas, discussed, but not fully understood, in graduate-level educational measurement courses and thoroughly bastardized by the time it is communicated to test developers and K-12 educators. The complex concept reduced to one word: decontextualize.

Education

As critical as it is that we improve DEI in educational measurement and assessment, we should be well aware in 2022 that improving assessment alone will not be sufficient to change the system.

It might conflict with our self-important sense of  illusory superiority, but it is critical that we acknowledge that educational measurement, assessment, and its purveyors and practitioners are mere cogs in the education-industrial complex. Although we play a crucial role in the education ecosystem, we are close to the bottom of the food chain. Alternatively, our role in that ecosystem might best be described as scavenger or decomposer rather than producer or consumer.

At times we have tried to position assessment as the catalyst. If we test it, they will learn it – which is a very different proposition than if it’s not on the test, it won’t be taught. The truth, however, is that where education and society lead, we will follow anywhere that they tell us to. We will follow where they lead.

Individuals

DEI, by its very nature, is focused on the individual student.

Educational measurement and assessment, in contrast, have long been stymied by the individual student.

Our personal version of Heisenberg’s Uncertainty Principle involves any attempt to describe accurately and precisely individual student performance on the basis of a test score.

We are constantly pushed to provide more information about individual student performance than can possibly be supported by our tests. Until there is a unified theory of assessment (that’s on my to do list for the summer by the way), we have to proceed with caution and humility.

A field’s got to know its limitations.

On The Shoulders of Giants

As we get our wobbly feet under us on DEI, it is a wise decision not to follow in the footsteps of forebears. However, I believe it would be a mistake not to stand on their shoulders to get a clearer view of the path they have trod, the distance we have yet to travel, and the obstacles in the road ahead. As a starting point, I recommend that we modify our perception of Angoff as simply a standard setting method and read his 1987 essay as APA Division 5 President, Philosophical Issues of Current Interest to Measurement Theorists.

Image by Darby Browning from Pixabay