Getting Back to Basics (and Proficient) With Accountability

My school has a rating of 3 stars.
What percentage of fourth-grade students know how to read (and do math)?
I have no freakin’ clue.
But that’s a key question?
Is it though?
I think so????
Percent Proficient is not a valid measure of school effectiveness!
Um, OK.
And it’s correlated with student demographics and SES!!
I know.  That’s the whole point behind Title I.
So, we agree.
[sighs] I just want to know whether kids know how to read.

I’m feeling a bit nostalgic these days.  Pining for the good old days back in 2000-2001 when as far as the feds were concerned school accountability was simply going to be the percentage of students who were proficient in reading and mathematics – and ultimately graduated from high school.

Think what you will about old George W. Bush, but The Decider knew what he wanted when it came to leaving no child behind, state assessment, and accountability. He just wanted us to tell him whether kids know how to read (and do math).

Was that too much to ask? Actually, yes, we told him, that kind of is a bit much.

Doing our best impression of Bill Clinton, but with an additional measure of condescension, we explained, you see Mr. President, it depends on what the meaning of “knows how to read” is?

Then we stepped back and watched chaos ensue for the next decade or so.

Each state had their own definition of “knows how to read” which was different than NAEP’s definition. So the states, with the help of David Coleman and a few unnamed co-conspirators, spent the next decade coming to a national, not federal, consensus on what it means to be able to read (and do math).

And by 2015 – only one year after the original deadline for all students being Proficient – we were ready to administer the first tests to determine whether kids know how to read.  Oh, we administered millions and millions and millions of tests in the interim (not even counting interim assessments), but in 2015, we were ready. A pinnacle, but not crowning, moment for our field.

Spoiler alert: Those test results showed that a majority of kids didn’t know how to read (or do math), at least not well enough to be considered college-and-career-ready – a “standard” which was slipped into the mix somewhere along the way.

Double spoiler alert: A few years later, it was discovered, somewhere deep in the heart of Mississippi, I think, that for years we had been teaching reading the wrong way. See Dick, Jane, and Spot look very confused. [Aside: We simply accept without question that nobody in the U.S. really knows how to teach mathematics; it’s a core part of our culture.]

But I digress.

While the content people were redefining what it means to know how to read (and do math), we were engaged in our own mostly well-intentioned machinations with poor little percent Proficient.

It WAS all so simple then

We started with the supposition, or proposition, that all students assigned to a grade level should be Proficient by the end of the school year.

But what about kids who don’t speak English? Or kids with significant cognitive disabilities? Or kids starting out 2,3, or more years below grade level?

Good questions. We could enter into a huge philosophical discussion about grade-level assignments and whether all means all, but let’s keep this simple:

• English learners can be exempt in ELA for a year. Remove them from the denominator. And they can take the mathematics test in their native language to the extent practicable (to the extent practicable, a lovely phrase that appears over and over again in NCLB). EL problem solved
• Kids with significant cognitive disabilities can be held to different standards and take different tests. We’ll start states off with an estimate that group includes about 10% of students with disabilities, or about 1% of all students. Throw them in the numerator. Problem solved.
• Kids below grade level can be counted as “Proficient” if they are making progress toward actually being Proficient within a reasonable amount of time. Throw them in the numerator. Problem solved.

We’re good. Percent proficient is still intact. Problems solved.

But then reality got in the way

Turns out it takes more than a year for kids to learn English. Yo no lo sabia.  And states found that it wasn’t practicable to test kids in other languages (except perhaps for Spanish), and some leading states banned teaching/testing kids in languages other than English.

Some states developed alternate assessments on which 80%-90% of kids with significant cognitive disabilities were still not Proficient, so nobody was happy there.

And there are also kids who are “on grade level” making real progress during the year, but not reaching Proficient…

Just Give A Little

Can’t we give schools credit for kids who move up a performance level, or most of a performance level, year-to-year but don’t quite reach Proficient?

Thus the accountability index was born. Full credit still was reserved for kids who were Proficient (or on track). The index ranged from 0 – 100. It looked like percent Proficient, but was not quite the same as percent Proficient, but both would be 100 when all kids were Proficient. Close enough for government work.

Give a little more.

But schools have no incentive to focus on high-achieving students.

We’ll add an indicator for the percentage of kids who are Advanced.

Give a little more.

You know, growth is really separate from status (i.e., achievement (i.e., percent Proficient)). And it’s an equity thing.

Growth should be a separate indicator.

Give a little more.

It happened incrementally over 10-15 years.

• Let’s assign grades to schools.
• Scaled scores are better than proficiency for accountability – especially with no firm deadline in place. Andrew Ho told us so.
• These indicators based solely on test scores don’t give a full picture of school performance. Along comes ESSA and the magical, mythical, so-called fifth indicator.

It’s a tale as old as time. You start out holding firm. There are bright lines you will not cross. Then you give just a little, and their argument is so convincing, and you want to keep them happy, so you give a little more, then just a little bit more, and before you know it you’re screwed.

Oh, the humanity

Horror is the only legitimate reaction when looking back on what we did to poor little percent Proficient and W’s desire to know whether kids know how to read (and do math).

Sure, percent Proficient has some shortcomings as a statistic. And it really only works with a goal of 100% and a deadline.

But percent Proficient was never the problem.

AMOs and AYP were doomed from the start.

And even Alice would be amazed at what she encountered down the rabbit holes that are alternate assessments and expectations for English language proficiency.

And, of course, percent Proficient (or any test score) is not a measure of school effectiveness and doesn’t provide a complete picture of school quality.

Whoot, There It Is

Effectiveness (and efficiency) were always part of the equation with regard to state testing, percent Proficient and Title I.

As the outcome of a program evaluation, we could have said that percent Proficient was an indicator of how the effectiveness of Title 1 programs (at the state, district, and/or school level).

We should have viewed percent Proficient as an indicator of the effectiveness of the general concept of Title I – its theory of action, if you prefer. Perhaps it would take more than extra money to solve the underlying problem.

But no, effectiveness was tied to schools.

And to make a bad situation even worse, some blowhard politician looking for a sound bite or perhaps a PR hack working behind the scenes, or perhaps simply a naïve staffer who didn’t know any better decided that schools that weren’t effective, were failing to make AYP, must be failing schools.

Failing schools.

That label was the death knell for NCLB and for our friend, percent Proficient.

Rather than doing everything in our power to disassociate ourselves from the failing schools label and school effectiveness, we started tinkering with the accountability index in hopes of making it better.

It was never going to get better. It never will get better.

States know how to evaluate school quality and it’s not with a test and an index (although both can be a part of the evaluation). They know how to include stakeholders in the process. They know about school improvement. It’s a complex, usually multi-year process.

But I still want to know if kids know how to read (and do math).

I still want to know the percentage of kids in a school who are Proficient or Meeting Expectations or Competent or College-and-career-ready or whatever you choose to call it.

It’s not enough to ask, but I don’t think that it’s too much to ask.