On Wednesday, I will be presenting at the opening session of the Center for Assessment’s 2021 Reidy Interactive Lecture Series, RILS. My presentation, A Brief History of Innovation: Bursting the Bubble, and the focus of this year’s conference on Design Innovation feel particularly appropriate for the annual conference named in honor of Ed Reidy. For Ed, was not only “a lifelong educator who played a significant role in improving public education in the state of Kentucky and in the country as a whole.” In my mind and heart, Ed will forever be the man in Frankfort, a co-conspirator if you will, who made it possible for a small band of people who didn’t know any better to attempt to turn Rich Hill’s bold vision for innovation in state assessment into a reality.
This is my twentieth RILS conference since joining the Center in the summer of 2002. My presentation at my first RILS conference in 2002 was a last-minute addition to the program. I presented the empirical analyses, which I had been conducting with state assessment data from a handful of states, backing up Rich’s theoretical simulations on the reliability, or lack thereof, of accountability systems designed to meet NCLB requirements. I guess we naively thought that’s how the process was supposed to work. That work eventually resulted in the publication of my only EM:IP article the following fall.
Similarly, my presentation at my tenth RILS conference in 2011 resulted in an invitation to write an article, Managing Multiple Measures, for the spring 2012 edition of Principal.
I know that this year’s presentation will yield a CenterLine blog post.
It’s an open question whether this progression reflects diminishing or increasing returns.
RILS, however, is not about publications. From the beginning, RILS has been about bringing together practitioners (i.e., people actively engaged in an art, discipline, or profession) for a couple of days to discuss the complex and wicked technical and policy problems involving assessment and accountability that they were working on now and those they expected to be working on in the near future – openly sharing thoughts, ideas, questions, and sometimes frustrations, learning from each other to better support states in the year(s) ahead.
It is certain moments from those discussions, and the people involved in them, that I remember when I think of RILS. Here are just a few from the past 20 conferences.
It was about meeting and sharing a table for two days with Center Board member Linda Mabry – a person who brought a different and fresh perspective to the issues we faced. As I described at the time, a perspective that was not 180° from ours, like so many opponents of large-scale assessment and accountability, but perhaps 45° – 90°, enough to make for a lively and informative discussion.
There was the moment at my first RILS in 2002 when Stanley Rabinowitz made the point that it was critical to confirm that state accountability systems identified the right schools. A brash, young, state assessment director sitting in the front row immediately asked, “How do we know which schools are the right schools?” (I wonder what happened to him.) That exchange crystallized for me the idea that’s it’s imperative that we are able to validate the classifications made via our assessments and accountability systems with information that exists outside of those systems. Assessment and accountability cannot be a closed system.
At the 2008 conference, Henry Braun shook me to the core during his presentation, Vicissitudes of the Validators, when he calmly stated that the validity of an accountability system should be judged not by whether it identifies the right schools, but by the extent to which it fulfills the promises and accomplishes the desired goals for which it was implemented in the first place: “Thus, consequential validity is the ultimate criterion by which we should judge an accountability system.”
And at the same 2008 conference, the juxtaposition of Robert Triscari and Tracy Halka presenting the novel approach we used to set college-readiness achievement standards for Achieve’s ADP Algebra II test and Laurie Wise describing the much more expansive, but oh so very similar, approach being used to identify college- and career-readiness thresholds for NAEP. Can there be a better form of validation than NAGB taking the same approach that you used for your exam? As I learned from the lawyers early in my career, the most compelling case for the legal defensibility of a standard setting approach is that someone else had used it.
Then there was the RILS session with some guy from Colorado whom I had never seen named Derek Briggs talking about something I had never head of: axioms of additive conjoint measurement. I had no idea what he was talking about that day, but I knew that this was someone who was thinking about educational measurement in a different way and much more deeply than anyone I had encountered to that point in my life in large-scale testing.
There was Laura Slover, on behalf of Achieve, presenting research and making their case that that a very similar set of mathematics content knowledge and skills were needed for college- and career-readiness (an actual career through which a person could support a family). I never agreed with their conclusion that Algebra II was the answer (what is that anyway), but I was convinced by the overall argument which still resonates today.
And in the early days of NCLB, there was a spirited discussion with Dale Carlson, a mentor to so many of us, that raised questions about theories of action and beliefs about the purpose of public education which are still the crux of assessment and accountability policy:
To what extent do we believe that the success of public education in the United States is limited by social factors outside of its control as opposed to the belief that the purpose of public education in the United States is to provide students the opportunity to overcome social factors outside of its control?
The school accountability system that you design at one end of that belief continuum will (or should) look very different than the one you would design at the other end.
Where our beliefs about public education fall along that continuum will (or should) influence how we approach assessment and accountability policy, how we identify and define the problems to be solved, and the type of innovative solutions that we propose to solve those problems.
Those are some of the memories I take from my first nineteen RILS. What will I take from the twentieth?
Image by Larisa Koshkina from Pixabay
Embrace the Absurd 