20 Questions – Student Proficiency Edition

Some of my fondest childhood memories are of the times spent on family road trips. The time in the car between destinations could have seemed interminable (much like the debate about spring 2021 state testing), but instead was filled with sing-a-longs and games. The sing-a-longs featured an eclectic mix of George M Cohan (Give My Regards to Broadway, Over There), folks songs (This Land is Your Land, If I Had a Hammer), and camp songs – “amster amster dam dam dam” – even though we never went to camp, and of course, a spirited rendition of the chorus of Cross Over The Bridge whenever we crossed over a bridge. There was much less variety in our choice of games. Our favorite game by far – 20 questions.

So, today as the country opens up and families prepare to hit the road again, I thought it might be fun to play a round of 20 questions. The mystery we are trying to answer today:

Is [insert name here] Proficient in Grade 8 Mathematics?

For the sake of simplicity, let’s call our fictional 8th grade student Charlie (now the most common name used across genders).

Also for the sake of simplicity, we will imagine that we are living in the distant future, let’s say 2025 – a dystopian utopia where there is either a universal definition of proficiency applicable across all students or each student has their own unique definition of proficiency based on their individual interests, skills, personal needs, and other factors they deem relevant – whichever you prefer. In either case, we simply accept that proficient is proficient.

We have 20 questions to help determine whether Charlie is proficient in grade 8 mathematics.  When you think about it, that’s not really all that different from the number of questions on some state tests, particularly the shortened versions being administered this spring.

Adhering to best practices in assessment design, we will adjust the rules to allow short answer questions in addition to the traditional Yes/No questions.

Where do I begin?

  1. What country is Charlie from? The United States of America

Hmm. OK. If the answer had been Singapore I would have known that there was a 79% chance that Charlie was proficient. On the flip side, if Charlie were from Morocco, only a 2% chance. The United States, unfortunately, is somewhere in the middle at about a 40% chance that Charlie is proficient, absent any other information.

  1. In what state does Charlie live? [Insert any one of about 40 states not Massachusetts or New Mexico]

In Massachusetts, the chances that Charlie is proficient would have jumped to about 50%; in New Mexico, they would have dropped to around 20% – still about 10x better than Morocco. So, let’s say Charlie lives in a state with about 25%-30% of eighth grade students proficient in mathematics.

  1. Does Charlie identify as Female?  Yes

Actually, this was a wasted question because in Charlie’s state there is not a significant difference in grade 8 mathematics proficiency based on gender.

  1. Is Charlie an English learner?  No

In contrast to gender, this question could have been much more informative if Charlie were an English learner. Statewide, only 2% of English learners are proficient in grade 8 mathematics.

  1. Does Charlie have an IEP?  No

Again, could have been informative as only 3% of students in the state with an IEP are proficient in grade 8 mathematics.

  1. Does Charlie attend school in an urban, suburban, or rural area? Suburban

Approximately one-third of the students in Charlie’s state attend suburban schools and 45% of those students are proficient.

  1. What is Charlie’s race/ethnicity? White

44% of the white students attending schools in suburban areas are proficient in grade 8 mathematics.

If Charlie were Asian, the percentage of proficient students jumps to nearly 70%. In contrast, if Charlie were Black, Hispanic, or multi-racial, the percentage of proficient students falls to 28%.

  1. Is Charlie economically disadvantaged? No

For non-economically disadvantaged white students in suburban areas, the percentage of students proficient in grade 8 mathematics jumps to 51%.

Maybe you see where I am going with this. Now that I know Charlie is female, white, not economically disadvantaged, and attends school in a suburban area, my guess about Charlie’s proficiency is still pretty much a coin flip.

However, if Charlie were Asian and not economically disadvantaged, the chances that Charlie is proficient in grade 8 mathematics would be close to 75% and I would likely guess that Charlie is proficient.

On the other hand, if Charlie attended school in an urban area, my answer would be clear at this point. The chances that Charlie is proficient in grade 8 mathematics would be slim. That wouldn’t change even if I asked a few more questions. As it turns out race/ethnicity, economic status, or even the specific urban district attended don’t have much of an impact on the chances that Charlie is proficient.

As shown in the graph below, although there is significant overlap in the distribution of test scores for students attending urban and suburban schools, most of that overlap occurs below the proficient cut score.

20Q

I’m guessing that many of you might be thinking that something feels wrong about this game. I haven’t asked anything about Charlie’s actual performance in grade 8 mathematics.

Isn’t this just dealing in stereotypes?

Isn’t this perpetuating a self-fulfilling prophecy? a deficit mindset? Or the tyranny of low expectations?

Isn’t this racist? [expressed either in a hushed whisper or shouted loudly and indignantly]

No, it’s just statistics.  And when used appropriately, statistics can be a powerful tool in describing and predicting group performance.

It’s true that these particular questions didn’t help me much in trying to determine whether Charlie – a female, white, not economically disadvantaged student attending school in a suburban area – was proficient in grade 8 mathematics.

So, let’s look at what happens if I had used my 20 questions (or even 30-40 questions) to design and develop a grade 8 mathematics state test.

Charlie takes the test and earns a score somewhere in the vicinity of the Partially Proficient/Proficient cut score. Based on Charlie’s performance on that one test (and it doesn’t really matter on which side of the Proficient bar her score falls), the chances that Charlie is proficient in grade 8 mathematics are still just about 50%.

We end up in the same place as we did with the original set of questions – a coin flip.

Winning the Game

With regard to determining the proficiency of an individual student to any degree of certainty, building a better test is not the answer. First, two decades of test-based accountability have demonstrated clearly that with testing we can have either Truth or Consequences, but not both. Second, I don’t see an end any time soon to the pressure on states to Beat the Clock in the design of their state tests – 120 minutes, 90 minutes, 45 minutes, … As I have written in previous posts, my preference is to Name That Achievement Level in one question; that is, by asking the teacher whether the student is proficient.  

Most importantly, however, one test score will rarely allow us to make a confident prediction about whether Charlie (or any student) is proficient in 8th grade mathematics. Charlie would have to score extremely low or extremely high on any single test for us to make us confident about guess about whether Charlie is proficient in grade 8 mathematics. Like many measurement instruments in the social sciences, a state test is most accurate when you don’t need it. [Important Note:  You never want to confuse confidence in the accuracy of a test score with confidence about the ability to generalize from that test score.]

When it comes to determining the proficiency of an individual student, a state test score will always be just one piece of information. We can supplement that information with other test-related information (e.g., how has Charlie performed on previous tests, what is Charlie’s growth score). We can also supplement the test score with information about Charlie’s performance in class on a variety of measures. All of that data and information is the evidence that we need to make an informed judgment about whether Charlie is proficient in 8th grade mathematics.

None of this, however, is reason to not administer state tests. A single state test was never intended to be a definitive measure of the proficiency of an individual student (even when test results used for high-stakes purposes such as promotion or graduation). The primary focus of state tests has been and should continue to be on school performance and ensuring equal opportunity for all students.  In upcoming posts, I will discuss whether continuing to administer state tests or not administering them places education reform and equity in education in greater Jeopardy!

Image by Gerd Altmann from Pixabay 

 

 

Published by Charlie DePascale

Charlie DePascale is an educational consultant specializing in the area of large-scale educational assessment. When absolutely necessary, he is a psychometrician. The ideas expressed in these posts are his (at least at the time they were written), and are not intended to reflect the views of any organizations with which he is affiliated personally or professionally..

%d bloggers like this: