In the wake of the events of 2020, the assessment and measurement community has made a commitment to do the right thing to ensure that tests are used appropriately. The community has vowed to be more proactive in speaking out against policies and test uses that result in inequitable outcomes or consequences, while actively promoting policies and test uses that improve student learning and increase equity. In practice, however, distinguishing between good and bad policies and test uses is not always easy. Questions that appear black-and-white at 30,000 feet often reveal themselves as 50 shades of gray at ground level. Decisions and actions that were so clear when laid out in a keynote address or journal article become cloudy when you realize that there are very fine people with persuasive arguments on both sides of the question. [all obvious allusions, references, and triggers in this paragraph were intended]
In this post, I would like to reflect on one test use policy that dogged me for the better part of my career: the use of the high school state assessment program as a high school graduation requirement. For the fifteen years from 1999 – 2014, I was on the outer edge of the inner circle of state policymakers in two states tasked with implementing a policy to use student performance on the high school state tests as one requirement for earning a high school diploma. Ultimately, the policies worked out quite differently in the two states, but the issues faced, the arguments raised, the sleepless nights, and the stress eating were the same in both cases.
High Stakes
By 1999, I had been involved in large-scale state assessment for a decade and was well aware of the problems associated with the high stakes use of such assessments for promotion, graduation, school accountability, and heaven forbid, teacher evaluation. And any facets of the argument that I had missed were covered in the new National Academies publication, High-Stakes: Testing for Tracking, Promotion, and Graduation. There were also regular communications with the folks at FairTest and there was a letter opposing the planned use of the new state tests signed by the faculty at the Lynch Graduate School of Education at Boston College – a couple of whom I highly respected and regarded as role models.
If the efforts to change minds were not enough, there were also the appeals to the heart. I still recall the agonizingly long drive home after a 12-hour day spent at state house hearings listening to story after story, in 3-minute snippets, of the students who would be irreparably harmed by the graduation requirement, of the injustice in holding student accountable for the failures of their schools and the larger education system. Who could not be affected by the individual students who would be figuratively, and sometimes literally, placed in front of you as examples of students who could not possibly be expected to “pass the test” and would be denied a diploma.
The Other Side of the Coin
We didn’t need complex statistical models, or even preliminary test results, to know that the students who would have the most difficulty meeting the test-based graduation requirement would be disproportionately students of color, low-income students, students with disabilities, and English learners. One could argue that the low performance of students in those subgroups, or the failure of schools to adequately serve those students, was a driving force behind the test-based graduation policy. That’s where the issues become cloudy. How could these students not be harmed by granting them a diploma when they lack the most rudimentary skills in high school English language arts and mathematics – when they will be forced to apply their limited financial aid to non-credit-bearing remedial courses at community colleges? Would these students benefit from support programs and public relations campaigns determined to “get everyone over the wall” without the graduation requirement?
What is fair?
Some of the strongest support for the high standards, high expectations, and high stakes (up to a point) came from the school districts and organizations serving the students described above. Many, but not all, urban superintendents viewed the requirement as a lever to obtain necessary funding and other resources. Advocacy groups for students with disabilities supported the increased focus that the policy placed on the need for high quality instruction for students with disabilities while attempting to make the test-based requirement as reasonable as possible for their students and enhancing opportunities for students who would not meet the requirement. It was a delicate balance for them.
What’s an Assessment/Measurement Professional to Do?
Presented with a situation in which a policy or intended test use has known risks, downsides, and threats to validity, but there are also negative consequences with the decision not to implement the policy, what is an assessment/measurement professional, test developer, or psychometrician to do? The answer is that we do what we are trained to do: evaluate the situation, work the problem, and arrive at the optimal solution. We do not adopt an attitude of indifference or reluctant compliance (“just following orders”). And we do not walk away or take our test and go home.
In addition to developing or adopting a high-quality assessment and appropriate administration procedures, there were five steps that I found useful when implementing the high school graduation requirement and similar high-stakes test-based policies.
- Have the infrastructure in place to support school improvement efforts based on curriculum and instruction before implementing the test-based requirement.
- In the two states in which I was involved in implementing a graduation test, education reform efforts had been in place for nearly a decade before the high-stakes graduation test was implemented. Districts and schools had ample time to implement programs to meet the needs of all students and the states had ample time to identify districts and schools in need of support.
- Determine which type of misclassification of student proficiency is the bigger “problem to avoid” and act accordingly.
- Whenever (and however) a yes/no, pass/fail, proficient/not proficient classification is made it will always be necessary to make a tradeoff between minimizing false positives or false negatives. In the case of test-based graduation requirements, states often regard misclassifying a student who actually has the required level of proficiency as not proficient as the more serious error. Passing scores on the test are set to minimize those types of errors.
- Ensure that students have multiple opportunities to meet the test-based graduation requirement.
- Given that the core content and skills that a state wants to ensure that all students have mastered via a large-scale test should not exceed the tenth grade level (personal opinion), it should not be difficult to offer students multiple opportunities to retake the state assessment or to meet the requirement through alternative assessments that the state has identified.
- Ensure that there are alternative pathways for students who cannot pass the test to demonstrate that they possess the required knowledge and skills.
- Remember that requiring students to pass the test is a means to a desired end (i.e., students who earn a diploma have the skills necessary for postsecondary success) and not and end in itself. It was always important to me that the Commissioner be able to declare truthfully that the test was not the reason that a student was denied a diploma (even if the Commissioner was not likely to make such a statement).
- Establish a plan for the ongoing evaluation of the impact of the policy, its intended and unintended consequences, and the extent to which it is achieving its goals.
- This step should go without saying for any policy or program, but is critical for one which has known negative unintended consequences. Districts and schools that were unsuccessful in getting students to proficient prior to the policy taking effect are most likely to explore inappropriate or ineffective approaches after implementation, particularly if there was a lengthy ramp-up period prior to implementing the test-based graduation requirement.
A final piece of advice is to establish an exit plan upfront (for the policy, not for yourself). Ideally, the conditions that made the test-based graduation policy necessary, or the best solution available at the time, should not exist forever. What will have to change for the policy to no longer be necessary? Identifying those exit criteria and working with districts and schools to achieve them will go a long way toward keeping the focus on the ultimate goal and off of the test.
(Image by No-longer-here from Pixabay)
Embrace the Absurd 