This month, we celebrate golden anniversary of the CCSSO National Conference on Student Assessment (NCSA) and with it, 50 years of innovation in assessment. The theme for the 2021 virtual conference is Re-Imagining Assessment. That theme is particularly well-suited for the annual assessment conference that I remember fondly as large-scale (short for the National Conference on Large-Scale Assessment or Large-Scale Assessment Conference, as the conference was known through 2007). Some of my older colleagues still refer to it simply as the “Boulder conference” after its original home.
Whatever you call it, the CCSSO conference was where we went each June to re-imagine assessment. Just a few of the ideas we first shared at the conference during my tenure at Advanced Systems, the Massachusetts Department of Education, and at the Center for Assessment that eventually became innovations in scoring and standard setting are listed below:
Scoring
- Offering practical approaches and technical solutions for including up to 100% constructed-response items on large-scale tests.
- Brainstorming and demonstrating computer-based approaches to the scoring of student’s written responses. Not automated scoring. Simply figuring out how to incorporate technology into the scoring of paper-based essays and written responses
- Providing comparisons of analytic v. holistic scoring of student’s written responses, plus demonstrating the relative technical advantages to single-scoring student responses to two items v. double-scoring a student’s response to a single item.
Standard Setting
- Introducing the Body of Work standard setting method – or as it was called in its original form, the Student-Based Constructed-Response method.
- Sharing an approach for setting achievement standards on a portfolio-based alternate assessment.
- Presenting results of initial attempts at crowdsourcing standard setting.
The conference also offered the opportunity to debate big ideas and burning questions of the day, such as
- whether the NAEP scale was a “real” vertical scale or if they just wanted us to think it was; and did it work just as well either way. If it looks like a duck…
- the reliability (or lack, thereof) of accountability systems designed to meet the requirements of No Child Left Behind,
- the rush to reify alignment criteria while alignment was still very much an abstract concept,
- standard setting, was it, in fact, fundamentally flawed,
- how best to introduce and implement new state content and achievement standards (spoiler alert: not with a top-down approach); and of course,
- our still elusive search for a way to increase the utility of large-scale test results.
A Jury of Our Peers
We brought all of our latest emerging and developing ideas to large-scale to put them to the test in front of our peers. Those peers included colleagues from testing companies and other organizations involved in large-scale testing, assessment staff at state departments of education, USED staffers with knowledge of, and often experience in, large-scale testing, and an elite group of academics, giants in our field, actively involved in large-scale testing in some capacity.
And our ideas were tested at the conference. Large-scale was our Peer Review process, one much more consequential to us than the publications peer review process (who published?) and one that existed long before the current federal Peer Review process was ever imagined.
There was real give-and-take during sessions, usually planned, sometimes spontaneous. Of course, the vetting process and occasional kerfuffle began in earnest during the lengthy breaks between sessions (curse that damn xylophone) and continued after sessions ended for the day. There was a real fear that this critical feature (pun intended) would be lost as the conference grew too large for Boulder. That aspect of the transition from Boulder, however, went fairly smoothly.
The conversations over drinks that extended well into the night in the courtyard at the Harvest House in Boulder shifted seamlessly to pints at the Golden Bee or cocktails poolside at THE BROADMOOR in Colorado Springs. How many innovations in large-scale testing took shape over a frosted mug of root beer at Schilo’s in San Antonio, were refined while strolling through the North End in Boston, or blossomed in one of those artificial biomes beneath the dome at the Gaylord Opryland? [to clarify: the Colorado Springs conference was held at a nondescript hotel down the street, but all the cool kids hung out at THE BROADMOOR.]
The first indication that an idea shared at large-scale had been well-received was when the letters from CTB’s legal team trying to patent it arrived in the mail. After all, as everyone knows, a patent lawsuit is the sincerest form of flattery.
Ultimately, of course, our ideas had to perform and be accepted in the marketplace to be considered assessment innovations (i.e., they had to become part of state testing). Many ideas did perform and were accepted. Others weren’t adopted for good reason. Still others probably should have been adopted, but something got in the way. Figuring out why good ideas didn’t make it remains another critical component of the conference.
A Unique Opportunity
I cannot overstate the importance of the CCSSO Large-Scale Assessment Conference to those of us in the field. I am sure that it’s difficult for many to imagine, and perhaps even difficult for my contemporaries to remember, when attending large-scale was perhaps the only opportunity many of us had to share our latest thinking on issues related to large-scale assessment with colleagues in the field. This was a time in the not-so-distant past with no e-mail, no websites, and certainly no Zoom calls. Hell, at Advanced Systems we had to wait for the red lights on our phones to indicate that one of the handful of outside lines was available before we could even make a phone call.
And the opportunities to gather with colleagues in person were few and far between. States and testing companies were not getting together every other week for one consortium meeting or another. Yes, there was AERA and its red-headed stepchild, but those conferences were too huge, the topics were too broad and too narrow at the same time, and it was all way too academic (in that sense of the word). Plus, traveling to AERA on Easter Sunday each year was a much bigger ask than traveling to large-scale on Father’s Day. Large-scale was our conference.
Now in 2021, in a technology-driven world and under conditions that couldn’t be more different than those described above, the National Conference on Student Assessment is still the best opportunity to come together to share and test ideas. In a world with too much interaction and too much noise, a world that is just too much at times, NCSA offers an oasis. It is a time and a place (albeit a virtual place this year) to forget day-to-day activities and the fires you need to put out, to focus on complex and wicked problems, and to re-imagine assessment.
Image by susancycles from Pixabay
Embrace the Absurd 