One in a Million, A Million to One

Interpreting Individual Student Performance on a Large-scale Assessment

This is the third, and final, installment in a series of three posts based on a workshop presented in April 2015 at the annual conference of the New England Educational Research Organization.

Across the land, there is a call for state assessments to provide more, better, and actionable information to inform instruction.  The Criteria for Procuring and Evaluating High-Quality Assessments published by the Council of Chief State School Officers (CCSSO) in March, 2014 includes the criterion that assessments should

Yield Valuable Reports on Student Progress and Performance – Providing timely data that inform instruction (D.2).

Reports are instructionally valuable, easy to understand by all audiences, and delivered in time to provide useful, actionable data to students, parents, and teachers.

In April 2015, Achieve, Inc. sponsored a webinar and released a set of materials focused on Communicating Assessment Results to Families and Educators.  Among the materials disseminated, Achieve produced sample assessment reports, including a model Mathematics Family Report. The two-page report provides the family with information about an individual student’s performance on the fifth grade state assessment in mathematics.  The first page of the report provides basic information about the assessment, explains what the results mean, and provides the student’s Overall Score (i.e., scaled score) and Performance Level classification.  So far, so good.  At the bottom of the first page, however, the family is instructed to “[t]urn to the second page to learn more about David’s knowledge and skills in mathematics.”  That is where the trouble starts.

At the top of the second page there is a display providing information on the student’s level of mastery in each of five Mathematics Scoring Categories.  As discussed in my previous post, What’s In A Name?, the capacity for a state summative assessment to support such claims of mastery is shaky, at best.

Achieve, however, does not stop there.  The model report also includes a section which contains a detailed description of the student’s individual strengths and weaknesses.

 

There is no doubt that information like this would be extremely valuable for David, his parents, and David’s former or new teacher (assuming the report arrives during the summer).  Unfortunately, we cannot expect to receive this level of information from a state summative assessment and should not raise expectations that this level of detailed information about individual student performance is an appropriate target for states and assessment contractors.  A state assessment report cannot provide the same type or level of information as a standards-based report card because a state assessment cannot produce that level of detail about an individual student’s performance.

The implication in the table of David’s Strengths & Areas For Improvement is that the information provided has been compiled specifically for David.  In reality, the list of strengths and weaknesses is likely to be similar to the lists of knowledge and skills provided in performance level descriptions.  That is, if that type of information is reported at all it will reflect probable strengths and weaknesses for students with an overall score and/or pattern of performance similar to David’s. However, if David’s performance is anywhere near the middle of the score range there are literally millions, if not trillions, of ways that students could arrive at the same score as David; and repeated analyses over the last fifteen years have shown us that on any given test there will be nearly as many unique response patterns as there are students taking the test.  The likelihood that we can identify and describe David’s specific strengths and weakness to the level of detail shown in the model report is a million to one long shot.

It would be even worse, however, if that list of strengths and areas for improvements were tied directly to David and based only on his performance on the one or two items on the test that measure multiplication and division of fractions, understanding of the concept of volume, or the use of line plot displays.  A student’s performance on a small number of items on a single test may suggest a particular strength or weakness, but by itself is certainly not sufficient to support the claims implied by the statements on the Family Report.

The inability to describe an individual student’s strengths and areas for improvement on the basis of performance on a single test is a limitation of state assessment, but it is not a flaw that we can correct. Yes, we can do a better job of communicating state assessment results to students, families, and teachers; but no, we cannot obtain accurate descriptions of an individual student’s strengths and areas for improvement from a single comprehensive assessment.  No, computer-adaptive tests are not the solution.  Computer-adaptive tests are designed to  provide more accurate estimates of a student’s location on the overall proficiency continuum, but in general, are not designed to provide more detailed information about an individual student’s strengths.

The call to provide families with the type of information included in Achieve’s model Family Report is admirable and appropriate.  The problem, however, was in positioning the Family Report as the report of results from a single assessment.  The Family Report, like a report card, can be based on a synthesis of information from multiple sources.  State assessment programs like PARCC and Smarter Balanced are offering tools that can be used throughout the year to collect more detailed information about student performance.  We should also be able to expect local districts, schools, and teachers to be able to provide families with accurate information on a student’s strengths and areas for improvement.  If local educators cannot provide that information, that is the reporting problem that we need to solve.  After all, even the model report directs families to contact David’s teachers for information on how best to support David’s learning.