A Changing Nation, A Changing Framework

The US Census and NAEP, two of my favorite topics to write about, have been in the news recently because of changes. Last week, data from the 2020 Census was released summarizing changes to the racial/ethnic makeup of the United States across the last several Census counts. Earlier this month, NAEP also made news, albeit with much less fanfare, with the NAGB approval of a new Reading Framework for the first time since 2004. The extent to which the changing framework is/isn’t (should be/shouldn’t be) a direct response to the changing nation kept NAGB occupied much of this past year and could be fodder for its own blog post, the topic for a good book, or the theme of a conference.

In this post, however, I won’t stray too far from my usual pastimes:

  • Discussing the flow of data and the information provided by large-scale, data collection efforts like the Census (and state assessment programs)
  • Raising thoughtful questions about (aka, trolling) the choices made by NAGB

“It’s black, brown, it’s yellow,
It’s red, it is white.
Every man’s the same in the good Lord’s sight.”

Much like the Census itself, the opportunity to pull out some good Up With People lyrics only comes along every ten years. Although perhaps these lyrics are a bit dated. “Every man’s” has been updated to “Everyone’s” and although the Lord may view everyone as the same, in our assessment context, about the only time the perspective that everyone is the same might be acceptable is when you are debating Charles Murray.

What the 2020 Census revealed about the country is that “it’s black, brown, its’ yellow, it’s red” and it is a little less white than it was in 2010 (i.e., “white alone”) both in terms of raw number of white people and their proportion of the population.  The headlines proclaimed the country is more urban, more diverse, and less white.

The decennial Census serves an important purpose. It provides accurate aggregate data at the national level and enables comparisons to be made at lower levels such as state, county, city, etc. And the Census folks, like our friends with NAEP, provide some neat data manipulation and visualization tools. That’s all great.

My main point today, as always, is that we must not forget that the flow of Census data is up from individual households to the nation as a whole.

  • My wife and I didn’t need the 2020 Census to tell us that our daughter had graduated high school and college and moved out on her own since the last Census in 2010. (even though she was in our house on April 1, 2020, but that’s just noise, right)
  • We have also seen the turnover in families on our street, in our neighborhood, and in the town since 2010.
  • We know that so many people have moved into Portland, ME and Portsmouth, NH over the last decade that there are virtually no rentals available.
  • I have seen companies moving their headquarters back into Boston from the suburbs because that’s where the young people want to live and work (until they are burned out and tossed aside for the next crop of young workers, oops, sorry).
  • I have watched my high school classmates moving back into, and young people from the MA DESE for the first time moving into, the same Boston neighborhoods we left in the 1970s and 1980s.

And I didn’t need to wait for the 2020 Census to understand the impact that the urban migration is having on the people already living in the city when Zillow.com is there to tell me that the triple-decker house in Boston (three 6-room, 3-bedroom apartments) that my uncle was barely able to sell for $1,000 in the late 1970s, sold for $300,000 in 1999 and now has a Zestimate ® of $972,800.

And then there’s NAEP.

`Who are you?’ said the Caterpillar.

Alice replied, rather shyly, `I–I hardly know, sir, just at present– at least I know who I was when I got up this morning, but I think I must have been changed several times since then.’

As we have discussed, no institution takes change more seriously than NAGB. Remember the Trend Line!

As someone comparable to the incomparable, but very relieved, Harvard professor Andrew Ho might have said (in an alternate universe) as he recently completed his term on NAGB:

“Nobody wants the first statistically-resistant variant of NAEP to be developed on their watch – or should I say (in)variant of NAEP [slight chuckle]. Raise a glass to the trend line!”

It is understandable, therefore, that the long-awaited update to the NAEP Reading Framework, coming in the midst of increased sociocultural awareness and a racial reckoning, might cause an Advanced Level of angst.   As reported in The 74, however, NAGB conducted themselves in true NAGB-like fashion and approved a revised Reading Framework that I am confident should allow them to preserve the trend line, if they so choose to do so, a few years from now when the first NAEP Reading tests are administered under the new Framework.

My thoughts today are focused on the implications, and perhaps unintended consequences (ah, I love that phrase), of two seemingly small “updates” to the Framework, with differences from the current framework noted in bold.

Under purposes:

Specific purposes for each question communicated to students on all assessment tasks

Under Universal Design Elements available for all students:

Samples of student writing as examples

Nobody reading this blog post had to be in the room where it happened to know the equity and Fairness arguments that were made and the specific sections of the joint Standards that were cited to support those changes. My concern today is not with those elements being included in the specifications for any assessment, including NAEP.

Rather, my concern is that the addition of those elements to the NAEP Reading Framework, their inclusion ultimately on the NAEP assessments, and the way that they are communicated will only hasten the ongoing slide down the slippery slope of placing too much attention on individual student performance on NAEP.

There is such a fine and blurry line between providing the appropriate information and supports students need to understand and respond to a test item and opening Pandora’s Box – causing teachers and schools to focus on preparing students to perform well on NAEP, thereby unleashing a multi-million-dollar NAEP test prep industry.

As the Nation’s Thermometer (before the data are processed and it becomes the Nation’s Report Card), we don’t want students and teachers preparing explicitly to do well on NAEP.

And we certainly don’t want to do anything that will alter the flow of information and allow NAEP to inappropriately influence curriculum and instruction. We don’t want samples of student writing in response to NAEP items to become exemplars for instruction in schools across the country. That’s not the way this is supposed to work.

As The 74 reported, NAGB member and Stanford economist Eric Hanushek was actually quoted as saying

He said the board has gotten itself in “an undesirable position” and has conflated the assessment of reading with the goal of improving reading. “The latter is not in our charge,” he said.

NAEP, the US Census, and large-scale state summative assessments serve an important purpose. It’s critical, therefore, that we remember what that purpose is and don’t do anything to make it more difficult for them to function as needed.

Image by CANDICE CANDICE from Pixabay

Published by Charlie DePascale

Charlie DePascale is an educational consultant specializing in the area of large-scale educational assessment. When absolutely necessary, he is a psychometrician. The ideas expressed in these posts are his (at least at the time they were written), and are not intended to reflect the views of any organizations with which he is affiliated personally or professionally..

%d bloggers like this: