assessment, accountability, and other important stuff

Archive for July, 2016

One Small Step

Charlie DePascale

Forty-seven years ago today, on July 20, 1969, Neil Armstrong and Buzz Aldrin became the first people to walk on the surface of the moon.  Their successful Apollo XI mission fulfilled the challenge proposed by President John F. Kennedy in a 1961 speech to Congress that “this nation should commit itself to achieving the goal, before this decade is out, of landing a man on the Moon and returning him safely to the Earth.”  The Apollo program has been hailed as one of the greatest achievements of the 20th century, perhaps one of the greatest achievements of all-time.  It serves as standard against which other significant challenges and programs are compared, one of the most recent examples being the Cancer Moonshot 2020 program to develop an effective vaccine-based approach to combat cancer by the year 2020.

We are not without similar challenging, ambitious goals in education. Fifteen years ago, President George W. Bush and Congress, proposed the challenge under No Child Left Behind that states should commit themselves to achieving the goal, by the year 2014, or having all students proficient in reading and mathematics.  The nation was not quite as successful at achieving that goal as in landing a man on the moon.  Results of the 2015 NAEP tests in Reading and Mathematics indicated that approximately half of U.S. students were proficient in reading and mathematics.  An improvement from pre-NCLB results, to be sure, but far short of the moon.

Although the NCLB goal of 100% of students proficient in reading and mathematics by 2014 was almost universally regarded as unattainable, it was by far a much less ambitious goal than those that preceded it.  Twenty-two years ago, President Bill Clinton signed the Goals 2000: Educate America Act into law.  Among other things, the act set the following goals, to be achieved by the year 2000:

  • The high school graduation rate will increase to at least 90 percent.
  • All students will leave grades 4, 8, and 12 having demonstrated competency over challenging subject matter including English, mathematics, science, foreign languages, civics and government, the arts, history, and geography, and every school in America will ensure that all students learn to use their minds well, so they may be prepared for responsible citizenship, further learning, and productive employment in our nation’s modern economy.
  • United States students will be first in the world in mathematics and science achievement.
  • Every school in the United State will be free of drugs, violence, and the unauthorized presence of firearms and alcohol and will offer a disciplined environment conducive to learning.
  • The nation’s teaching force will have access to programs for the continued improvement of their professional skills and the opportunity to acquire the knowledge and skills needed to instruct and prepare all American students for the next century.

Like the NCLB goal of 100% of students proficient in reading and mathematics, progress has been made on some of the goals set forth by Goals 2000, but in 2016 none have been accomplished.

In a 2011 commentary in EdWeek, Salvaging Race to the Top Assessment, I suggested that goals such as improving instruction and student learning to ensure college- and career-readiness were more challenging than landing a man on the moon and returning him safely to earth because a) the Apollo program had a clearly defined outcome, and b) the effort to land a man on the moon was largely self-contained.  In contrast, our goals in education are often much less well-defined (e.g., what do we mean by college- and career readiness?) and are by no means self-contained.  As I stated in the EdWeek commentary

The success of the assessment program now being developed by two multistate consortia with Race to the Top money will require extensive infrastructure changes in schools; the active support and involvement of states and local educators across the country; and the active support and involvement of students, parents, and perhaps communities. Additionally, the project will require significant behavior changes and resources for long-term, sustained implementation. In this regard, the War on Poverty may be the more appropriate 1960s comparison than the Apollo project in terms of RTT assessment’s scope and complexity.

Of course, the same can be said of the goals set forth in NCLB and Goals 2000.  Bringing about any major change in education requires the long-term commitment and active support of all parties, including the general public (i.e., taxpayers).  For some reason, however, the country no longer engages in discussions of long-term commitment and sacrifices needed to accomplish goals.  That was not the case, it appears, with the space program in 1961.  In the same speech to Congress in which President Kennedy called for a commitment to reach the moon, he also expressed this caution

I believe we should go to the moon. But I think every citizen of this country as well as the Members of the Congress should consider the matter carefully in making their judgment, to which we have given attention over many weeks and months, because it is a heavy burden, and there is no sense in agreeing or desiring that the United States take an affirmative position in outer space, unless we are prepared to do the work and bear the burdens to make it successful. If we are not, we should decide today and this year.

This decision demands a major national commitment of scientific and technical manpower, materiel and facilities, and the possibility of their diversion from other important activities where they are already thinly spread. It means a degree of dedication, organization and discipline which have not always characterized our research and development efforts. It means we cannot afford undue work stoppages, inflated costs of material or talent, wasteful interagency rivalries, or a high turnover of key personnel.

New objectives and new money cannot solve these problems. They could in fact, aggravate them further-unless every scientist, every engineer, every serviceman, every technician, contractor, and civil servant gives his personal pledge that this nation will move forward, with the full speed of freedom, in the exciting adventure of space.

We have not had a similar conversation about what it will take to achieve goals such as ensuring that all students graduate from high school college- and career-ready, closing achievement gaps, or improving the nation’s teaching force. Unfortunately, I do not expect that conversation to happen any time soon.  There has certainly been nothing in the current presidential election campaign to suggest otherwise.

So, let’s set aside the grand long-term goals for a minute and in commemoration of Neil Armstrong’s historic step from the lunar module to the surface of the moon, let’s focus our attention on “one small step” or a few small steps that might lead the way to a giant leap forward for education.  Let’s commit to taking a few key steps that certainly can be accomplished in the three years between tonight and the 50th anniversary of the Apollo XI moon landing in July 2019.

  1. Develop a clear and complete definition of college- and career-readiness.
  2. Agree on the limited amount of content area knowledge and specific skills that are critical for all adults to possess.
    1. Acknowledge that much of the content taught in schools is intended to support the acquisition of complex thinking skills, but could be interchangeable with other content.
  3. Meet President Clinton’s 1996 technology goals for schools:
    1. All teachers and students will have modern computers [devices] in their classrooms.
    2. Every classroom will be connected to the information superhighway.
    3. Effective and engaging software and on-line resources will be an integral part of every school curriculum.
    4. All teachers will have the training and support they need to help all students learn through computers.

Perhaps those three steps are not really that small, but they are worth our commitment and our effort because we want our children in this nation to know that the only limit to your achievements is the strength of your dreams and your willingness to work for them. (Source unknown)

 

We hold these truths to be self-evident…

Charlie DePascale (assisted by the words of Thomas Jefferson)

We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable rights, that among these are life, liberty and the pursuit of happiness.

 

IMG_1377

When in the course of human events, significant resources are allocated to educational assessment and important decisions are made on the basis of the results of educational assessment, it becomes necessary for the people to understand the Laws of Nature which govern educational assessment. Further, a decent respect for the people and the field requires that those with appropriate knowledge and understanding place before the people the common sense of the subject, in terms plain and firm.

Therefore, in educational assessment

We hold these truths to be self-evident, that reality is contextual, that we cannot eliminate uncertainty, that educational assessment is based on modeling, and modeling is not measurement.

In short, the fundamental truth that we hold to be self-evident is that there is no truth.

As the stakes of education have risen, so too have the demands placed on educational assessment.

  • Whereas large-scale assessments were once administered annually, or even every two years, to students in select grade levels, we are now in a place where administering assessments to all students in every grade level multiple times per year is being considered as a viable option to reduce testing.
  • Whereas it was once sufficient to report average performance for large groups of students on loosely defined content, it is now expected that assessment programs will produce detailed information about the achievement and growth of individual students against well-defined content and achievement standards.
  • Whereas it was once acceptable to not assess large groups of students who did not fit the design of the assessments, it is now expected that assessments will be accessible to and provide actionable information about the performance of each individual student.
  • Whereas large-scale assessments previously had few, if any, consequences for districts, schools, and teachers, the results of those assessments are now the cornerstone of accountability and education reform.

For the most part, the increase in demands placed on educational assessment has received the tacit approval of the educational measurement community.  That is not meant to suggest that there have not been cautions raised on occasion and that individual measurement specialists or even groups of measurement specialists have not voiced opposition to certain policies; or to ignore to that some measurement specialists have based their research agendas on demonstrating flaws in assessment-based educational policy and that at least one prominent individual with a background in measurement has disowned the educational measurement community.  The reality is, however, that as the demands increase, the tests keep coming.

Perhaps it is hubris – a belief that we can rise to the challenge and build assessments to meet any demands.  Perhaps it is fear – a belief that if we are honest about the limitations of our field, the entire field will be rejected.  Perhaps it is a matter of economics – we give the people what they want (or in a more negative light – we can convince the people that we have what they need and want). Perhaps it is simply resignation – a belief that we are powerless to stop the inevitable.  Whatever the explanation, the tests keep coming.

Today, the focus is on acknowledging the self-evident truths of educational measurement; and by doing so, provide the people with information that they need to begin to become literate consumers and users of educational assessment.

Reality is contextual

When discussing assessment results, we tend to talk in terms of absolutes.  We declare that Martha is proficient in mathematics; Alexander is college-and-career ready; or that Washington High School is an A+ school.   Of course, the meaning of each of those classifications and the inferences one can draw from them is dependent on context.

There is no universally accepted definition of the specific knowledge and skills that mathematics comprises.  Mathematics for a particular context is defined by the set of content standards adopted at a given point in time for a specific purpose.  Similarly, the meaning of proficiency in mathematics is tied directly to those content standards.  One can engage in a chicken-or-egg argument about whether the concept of proficiency should flow from the content standards or vice versa, but together content and achievement standards often form a closed system.  It is clear that proficiency on the basic skills mathematics standards from the 1970s was different from proficiency on the world-class standards established by some states in the 1980s and 1990s, which was different still from the concept of demonstrating college-and-career readiness on the Common Core State Standards in mathematics.

In addition, an unfortunate facet of the current system is that the reality of the concept of proficiency in mathematics is often defined by the assessment instrument rather than by the content standards or achievement level descriptions.  Yes, we strive for alignment between the assessment and the content and achievement standards, but any educator will tell you that the meaning of proficiency does not become reality until they see the first set of assessment results.

We cannot eliminate uncertainty

In reporting assessment results, we do not totally shy from the concept of uncertainty, but we do our best to downplay it.  We routinely report results with relatively small error bands and explain that a student’s true score falls within that band.  We explain that all assessment results are imprecise and contain measurement error, but suggest that such error can be controlled through better test design and/or accounted for through statistical techniques.  In other words, we discuss the precision of the things that we can and do measure on the assessment.

Rarely, however, do we engage openly in discussions about the level of uncertainty surrounding the construct.  Despite all of the attention devoted to alignment over the last 15 years, as a field we still have little understanding of the impact of varying degrees of alignment or of how to define and measure alignment to complex constructs. (Hint: It is not through counting items.)

More important, we have an insufficient understanding of  how student performance on assessments can and should relate to student learning and achievement in a particular subject area over time.  Whether one views the content from third grade through high school as a single construct, a set of interrelated constructs, or a network of learning progressions, there is a great deal of uncertainty about how student learning should progress on complex constructs, and that uncertainty is totally unrelated to the assessment.

Modeling is not measurement

The complexity of the constructs we hope to understand is one reason why our science of educational measurement is based primarily on modeling, probability, and prediction.  The goal is to uncover and understand relationships among factors included in the model. Our field, however, favors discussing the precision and certainty implied by measurement rather than having to deal with the uncertainty of modeling.

Like the modeling used to predict the weather or the number of games our team will win next year, there is usefulness in the modeling associated with educational assessment.  However, as in those systems, there is also uncertainty.  There are multiple models and there is variability across the models.  Beyond the models themselves, however, there is inherent uncertainty – factors that the models cannot predict. We all have experienced a day when a weather front moved more slowly or quickly than expected, ruining our plans for a cookout, ball game, or commencement ceremony.

Embracing uncertainty

We, therefore, specialists in educational measurement, must be willing to solemnly publish and declare that there is uncertainty in educational assessment, uncertainty that cannot be eliminated simply by building better assessments or better assessment models.  And by doing so, pledge our support to preparing educators and policy makers who as consumers of assessments and assessment results are better prepared to work with uncertainty.  As Voltaire wrote, “doubt is not a pleasant condition, but certainty is absurd.”