As the final states release results from the spring 2015 Smarter Balanced and PARCC assessments, this post is a look back at a June 2010 presentation to CCSSO Technical Isssues in Large-Scale Assessment (TILSA) members on lessons learned from my involvement with the New England Common Assessment Program (NECAP). The presentation was titled, Lessons Learned from a Multi-State Assessment Consortium or Everything I Need to Know About a Consortium, I Learned from a Tom Hanks Movie.
This presentation was made two days after the June 23, 2010 deadline for consortia to submit proposals for the Race to the Top Assessment Program and a little more than two months before grants were awarded to PARCC and Smarter Balanced in September 2010.
It may be worthwhile to look back now at this lighthearted list of NECAP lessons learned, and not simply because we should soon begin to see presentations, articles, dissertations, behind the scenes tell-all books, and even memoirs on lessons learned from the PARCC and Smarter Balanced consortia. But also because, now more than ever states, districts, and even individual schools are considering ways to come together to build capacity and share resources related to curriculum, instruction, and assessment.
Lenny: “Are you crazy? A man in a really nice camper wants to put our song on the radio! Gimme a pen, I’m signin’! You’re signin’! We’re all signin’!” (That Thing You Do!)
Deciding to join a consortium is easy. It’s pretty easy. Well, it’s reliatively easy. In the beginning, it’s easy to see all of the advantages of joining a consortium: increased capacity, shared resources, and the experience of working together to accomplish something no one could accomplish alone. It’s an exciting time. It becomes even easier when the federal government is willing to put up $135 million to fund the consortium.
The problems can begin when the decision to join is too easy.
One thing about trains: It doesn’t matter where they’re going. What matters is deciding to get on. (The Polar Express)
The first step is for states to make the decision to join a consortium. As they make that decision, each state should be able to answer the following questions and their partners in the consortium should want to know the answers.
- Why did they decide to join?
- What do they hope to gain from being a member of a consortium?
- What are they giving up by joining a consortium?
- What commitment are they making to the consortium?
I have written previously that a binding force among the states in the NECAP consortium was desperation. Each of the states had to meet the increased assessment requirements under NCLB. They determined that the only way that they would be able to do that and maintain the type of assessment that they wanted (i.e., not settle for a totally multiple-choice test) was to form a consortium, pool resources, and share costs.
It is not clear to me whether desperation or a similar binding force is there with all of the states joining the PARCC and Smarter Balanced consortia at this point in time. Some states are joining both consortia (even though the two consortia are quite different in virtually every important way), and it appears fairly easy and painless to leave a consortium at any point up to and including 2015. It will be interesting to see how this plays out. [Remember, this was presented in 2010. Yes, it was and still is interesting]
Phil Horace: I’ve found that a hit record is like a stew. All the ingredients have to come together just right. Otherwise, it’s just soup. (That Thing You Do!)
Operating a consortium requires more than simply bringing a group of states together to share an assessment. The task of managing the consortium is equally important, or perhaps even more important, than the task of building the assessment. There are layers upon layers of people working directly on the project or with a vested interest in the project working within each of the states and for the assessment contractor(s). Coordinating communication and facilitating interactions among all of these people and organizations to keep everyone on the same page and moving in the same direction is a complex task that deserves and requires attention.
Del Paxton: Ain’t no way to keep a band together. Bands come and go. You gotta keep on playing, no matter with who.… Sooner or later something makes you crazy. Money, women, the road. Hell man, just time. (That Thing You Do!)
There are going to be changes to the consortium. Those may be big changes like a state leaving the consortium or announcing that they no longer want to give one of the assessments. More often, however, they will be smaller everyday changes that nevertheless can have a huge impact on the functioning of the consortium.
- State content leads leave the department of education to return to the K-12 school districts or to take faculty positions in higher education.
- There are staff changes within the assessment contractor’s project team.
- Commissioners of Education retire, resign, or are replaced. (Governors, legislators, and board of education members also change over time.)
- The original contract expires and you have to prepare for the reality that there may be a new assessment contractor.
- A key state assessment director receives a promotion within her department and can no longer participate in consortium management meetings.
- New national common standards are adopted and states move on to a larger consortium.
Stop it, you zealots! (Toy Story)
To some extent, every large-scale assessment has a set of higher-level purposes associated with it. The assessment program is going to improve instruction, close the achievement gap, and of course, ensure that all students leave high school prepared for postsecondary success. When states form a consortium it seems that those goals and purposes become more central to the assessment program and the rhetoric becomes even loftier –
- We are all working to improve student learning!
- Save the assessment, save the world! [A Heroes reference that was slightly less dated in June 2010]
At some point, however, it is critical that everyone realizes that they are there to build an assessment and that there are actual tasks that must be accomplished successfully to build that assessment. Items must be developed, student responses must be scored, forms must be produced, manuals must be written, performance standards must be set, and reports must be distributed.
“It’s supposed to be hard. If it wasn’t hard everyone would do it. The hard is what makes it great.” (A League of Their Own)
In the beginning, a consortium has a “Hey gang, let’s put on a show!” feeling to it. There is a sense that all of these states can come together, pool their resources, and build a great assessment. Then reality hits.
Building a high-quality, custom large-scale assessment is hard work and requires people and organizations with experience and specialized expertise in a variety of areas. It requires commitment from all of the state partners, but also an awareness that they cannot do this alone. The members of the consortium must choose an assessment contractor(s) and technical advisors whom they can trust and with whom they can work as partners in building the assessment.
Consortium members must also find a way to maintain a proper balance between political concerns and technical concerns when the states’ policy makers vastly outnumber the psychometricians and assessment specialists assigned to the project.
Look what I’ve created. I have made FIRE. (Cast Away)
Yes, a consortium involves harnessing the capacity of groups of people to create new and wonderful things. However, a consortium also requires a certain degree of relearning how to do old and pedestrian things. Routine decisions that you made easily on your own, now require a process to be approved by the group. Processes and procedures that worked fine within one state must be adjusted (and fully documented) to work seamlessly across all of the states in the consortium. Things that were common practice in your state (e.g., allowing a read aloud accommodation on the reading test or allowing two days for students to complete a writing test) may no longer be acceptable.
Don’t worry Wilson, I’ll do all the paddling. You just hang on. (Cast Away)
There will be times when members of the consortium are not able to pull their own weight. An individual may be dealing with personal or professional issues that require his or her attention. A state may have to deal with a pressing political issue that detracts its attention from the tasks of the consortium. At those times, the other consortium partners have to be willing step up and fill the void.
There will also be times when the assessment or the consortium is coming under attack or scrutiny within an individual state. At those times, consortium members must also be willing to band together to support each other.
This could work! This could work… (Cast Away)
This quote applies on a number of levels.
It is important to take the time to acknowledge and celebrate small successes along the way to the final goal. Although the entire project is huge and daunting, it is composed of series of smaller tasks, each of which must be accomplished successfully. There will likely be lots of things that do not work as expected the first time around. When something does go right, allow people to have a sense of accomplishment and support the feeling that hey, this might just all come together in the end.
On another level, throughout the project try to maintain the attitude that “This could work!” despite common sense (and some evidence) that tells you that it won’t. Because …
At some point a milestone will be reached or some task will be accomplished and you will really believe “This could work!” It might be making it through the first item review meeting with educators from each of the states, or the completion of field testing, or perhaps it might be surviving the first day of operational testing; and it may not be the same point for everybody. However, it will occur. At some point you will take a breath and think, yes, this could work.
“From now on, we live in a world where man has walked on the moon. And it’s not a miracle, we just decided to go.” (Apollo 13)
Building an assessment and operating a successful consortium does not require a miracle. It does require careful planning, hard work, and a commitment to see the task through to completion.
Summary
Those are the ten quotes and lessons about operating a successful multi-state consortium that seemed most appropriate to share in this presentation. I will admit that there were a few other quotes from the Polar Express that were tempting:
- Hero Boy: Well, I… I want to believe. But… Hobo: But you don’t wanna be bamboozled. You don’t wanna be led down the primrose path. You don’t wanna be conned or duped, have the wool pulled over your eyes. Hoodwinked. You don’t wanna be taken for a ride, railroaded. Seeing is believing. Am I right?
- Considering the fact that we have lost communications with the engineer, we are standing totally exposed on the front of the locomotive, the train appears to be accelerating uncontrollably, and we are rapidly approaching Glacier Gulch, which happens to be the steepest downhill grade in the world, I suggest, we all hold on…TIGHTLY!!!
- Could all this be nothing but a dream?
Even though each of those quotes was probably appropriate at one or more points during the last eight years, they seemed just a tad too pessimistic for a presentation focused on lessons learned from a consortium that its member states view as highly successful.
However, the task that the NECAP states were trying to accomplish was pretty well-defined. In essence ,they formed a consortium as a way to keep the assessments that they already had. That is not the case with these new consortia. In thinking about the Smarter Balanced and PARCC consortia and the task that they face in building a next generation assessment aligned to the Common Core, I will end with a bonus quote that seems appropriate. This is not from a Tom Hanks movie, but from the HBO mini-series From the Earth to the Moon, for which he served as Executive Producer. The quote is presented of course, with full knowledge that ultimately the lunar modules being discussed were completed and the project to send a man to the moon and return him safely to earth was d successful.
- Perhaps we were behind schedule and over budget because budgets and schedules are based on previous experience with similar projects. We didn’t know how much it’d cost to build the LEMs or how long it would take. All we really knew was how much time we’d been given and that was running out. LEM 3 was schedule to be launched in the fall of 1968. To make the launch, NASA needed delivery some time that spring. We were working as fast as we could, 24 hours a day, seven days a week and it wasn’t enough.
Embrace the Absurd 