The Ernie Scale

Based on a true story

In the late 1980s and early 1990s, Rich Hill and Stuart Kahl led their little company that could, Advanced Systems, in reimagining, reinventing, and reshaping large-scale state testing. We were a ragtag band of young idealists who didn’t know any better. Our successes and failures were spectacular.

Little that we did at Advanced Systems during that time span went unnoticed. Even before social media, it seemed as if every constructed-response item that we scored, every test form that we equated, every scale score that we reported, or chose not to report, was documented, dissected, and debated.

Well, not quite everything we did made it into a RAND report or onto the front page of EdWeek.

This is the story of one of those things that was not like the others – the development of the Ernie Scale.

Yes, that Ernie.

Ernie, of Bert and Ernie fame, had become our office mascot. Every office needs a mascot. We did not set out to adopt Ernie as our mascot. We made no psychic or psychometric connection – latent or otherwise – between Bert and Ernie and Rich and Stu.

You know how it goes. You need a way to indicate which documents have been reviewed and approved, so for fun you pick up an Ernie ink stamp. People see the smiling Ernie on documents and soon other Ernie’s start appearing in your office – a plush doll, plastic figures, a music box.

Then one sunny day, everything’s A-OK, and …

The Set Up: A is for Air, B is for Brian, C is for Conditioner

My officemate, Brian, had been assigned a special task. When the outside temperature rose to a certain point, the AC system, housed in the closet next to our office, would start to ice up. If you didn’t stay on top of it, well, just make sure you stay on top of it.

The company’s depending on you, son.

The Questions: 1 2 3 4 5 6 7 8 9 10 ….

It began shortly after lunch, as the sun rose high in the sky. A colleague walking down the hall (probably a content person) stopped in the doorway, smiled, and asked, “What’s the temperature?”

 That afternoon everyone had a reason to walk by our office, stick their head in, and ask, “What’s the temperature?”  

The mercury in the thermometer had remained steady, but with each passing colleague, Brian’s temperature began to rise.

After checking the thermometer and responding politely a dozen times or so, the next time the question was asked, he simply looked up and responded “4”

“4?”, they asked.

“4”, he repeated, “It’s always 4.”

“What scale?”, they asked.

A quick look around the office yielded the obvious name for the scale: The Ernie Scale.

The questioner left bemused. We, it goes without saying, were quite amused.

 A Scale is Born

 Now we had a single point, ‘4’, and a name for the scale.

When you think about it, that’s not very different than the way the rest of our test score scales were formed back then. You had a mean and standard deviation, you came up with a name, and bingo, you had a scale.

Being naturally inquisitive, our colleagues at Advanced Systems demanded more details about the Ernie Scale.

We were happy to retrofit meaning to ‘4’ and the otherwise meaningless “scale” we had concocted on the spot out of thin (but warm) air. Still a novice at the large-scale testing game, little did I realize how important that skill, and the confidence to use it, would be as I progressed in my career.

It’s incredible where you can go with your imagination.

Each one in its home, Each one in its place

What could the scale score ‘4’ represent? Why would it never change?

It should have something to do with temperature.

The sun has something to do with temperature. It’s hotter when we get closer to the sun (Harvard, 1989).

Right. Right. The planets get colder the farther they are from the sun.  ‘4’ could represent the earth’s distance from the sun.

[excitement building]

But the earth is the third planet from the sun. That’s common knowledge. A couple of years from now Joe Diffie will have the song of the summer with, “Third Rock From the Sun,” and a couple of years after that there will be a hit sitcom, “3rd Rock From the Sun.”

[excitement fades]

What if we label the Sun ‘1’ instead of ‘0’ and go from there? It’s an ordinal scale, we can use cardinal numbers. We don’t need a zero point.

The planets will be 2 through 10. [Pluto was still a planet in the early 1990s.]

Earth will be ‘4’.

[Euphoria! Sing out loud, sing out strong!]

Validation – Everyone makes mistakes, so why can’t you?

Does the temperature of the planets really work that way? Let’s take a look.

Drat! Drat! Venus is hotter than Mercury by about 500 degrees. Something to do with its atmosphere. But our scale is perfect for the other eight planets.

That’s only one reversal. Mark Wilson would give his left, um…, arm, yeah arm, for a scale in which only one data point didn’t fit his model.

We can drop Mercury. It only comes up in conversation every decade or so when it transits across the sun.

No, we need Mercury for Earth to be ‘4’ on the scale. We can just ignore the reversal, Mercury and Venus are so close to each other anyway. Nobody will notice. If they do, measurement error. Besides, you can’t just drop a planet!  [Again, pre-Pluto.]

“I’m happy to be me.”

 And now, Elmo has a question FOR you.

We have a scale. It has meaning. People have stopped asking us about the temperature.

But the Ernie Scale doesn’t really help us with the air conditioning and the ice buildup?

No, of course not.

And it’s not even useful for something big like global warming. Right?

Wait. Wait. Wait. You’re getting way ahead of yourself. Al Gore isn’t even vice-president yet. People won’t start worrying about global warming for another 10 years, assuming we survive Y2K.

And they won’t expect test score scales to provide useful information for another 10 years after that.

That’s good, because as you know, our inconvenient truth is that large-scale tests just aren’t designed to provide detailed, diagnostic information about individual students.

True, but I’m sure that we’ll have much better approaches to scaling, reporting, and large-scale testing, in general, by then. Look at how far we’ve come in just the past three years.

If you keep practicing, you can do anything.  So, keep calm and eat cookies!

You’re right. Lets’ get back to work.

Now, how are we going to equate these writing portfolios?

Header Image by Alexas_Fotos from Pixabay