Citius, Altius, Fortius

Charlie DePascale

 

We have barely recovered from the withdrawal symptoms that accompany the end of each Olympics games, and suddenly summer is over and another school year is upon us.  As our attentions shifts back to designing new and improved school accountability systems under ESSA, this is a perfect time to reflect on lessons learned from the Olympics.  Lessons that extend beyond the Olympic motto, Faster, Higher, Stronger and the ephemeral feats of the athletes to the very heart and soul of the games – the rules that determine success in each of the individual sports.  In particular, let’s focus on the values reflected in the rules related to progressing through the various heats or rounds and the rules related to breaking ties.

We tend to view the Olympics as a monolith. In reality, the Olympics games are the simultaneous convening of championship meets or tournaments by a host of international sports federations, each with its own set of rules.  One could think of the games as a giant athletic flea market in which each of the vendors, in exchange for a fee and the agreement to abide by a set of house rules, has been allocated space and time to market their wares.  Or perhaps one could think of the games and the rules established by the various sports federations as analogous to individual states designing accountability systems that meet the requirements of ESSA under the oversight of the US Department of Education.  Within the parameters established by the IOC, each sports federation makes value judgments regarding the indicators that it will use to determine who can call themselves an Olympian and ultimately, who receives a bronze, silver, or gold medal and will forever be known as an Olympic champion.

To some extent, rules are determined by a combination of tradition, practicality, and the nature of the game.  In sports such as basketball, volleyball, or soccer (i.e., football) each contest is a match between two opponents which ends with an individual winner – although in soccer there can be ties in the preliminary, or pool, games.  In volleyball, in which the match is a series of independent sets, the winner is the team that wins the most sets.  By design, there are an odd number of sets so that the match cannot end in a tie.  In basketball and soccer, the winner is determined by total score at the end of a specified period of time.  In the event of a tie, both sports continue with an overtime (or extra time) period.  The overtime period is shorter than periods during the game, but in most other respects follows the same rules as the rest of the game.  In basketball, overtime periods continue until there is a winner.  In soccer, however, after two extra time periods, the winner of the game is decided by a series of penalty kicks.  Penalty kicks are exciting and arguably provided one of the most dramatic moments in Rio and one of the top ten goals and most iconic images in U.S. soccer history.  For some, however, the concept of deciding a major championship on the basis of a skills competition does not sit well. Hope Solo, for example, may have preferred a tie-breaker more similar to boxing in which the match is awarded to the participant who was more dominant or aggressive throughout the match.

The need for the penalty kick solution in soccer is a function of the nature of the game.  Championship games have to have a winner and goals (and even shots) are often rare.  For example, in its final two games in Copa America 2016 earlier this summer, the U.S. men’s national team scored no goals and took a total of two shots on goal.  Prior to penalty kicks, soccer championships were determined by replaying the entire game (as is the case in the U.S. Open golf tournament) or by a coin flip (which is a method generally reserved for deciding the winner of Democratic party caucuses).  So, the penalty kick solution may not be optimal, but it was more practical and seems fairer than previous solutions, and as noted above, is very exciting.

Moving beyond overtime rules in matches between two participants, there are a plethora of rules that suggest that faster, higher, stronger are not the only values at play in various sports.

Gymnastics is a hotbed of interesting rules.

• Qualifying for the All Around championship – Current rules allow only two gymnasts per nation to compete for the title of All Around Champion. Although three U.S. gymnasts ranked in the top 5 of all gymnasts competing in the each of the last two Olympics games, only two were advanced to the field of 24 gymnasts competing for the Olympic gold medal and title of All Around Champion.
• Breaking ties in the All Around competition – The winner of the All Around women’s competition is determined by the sum of the scores across the four events. In the event of a tie, the first tiebreaker is to identify the winner by dropping the score from each gymnast’s lowest scoring event. The consequence of this rule is that in a tie between two competitors, the gymnast with the lowest score on one of the four events is declared the All Around champion.
• Breaking ties in an Event final – In Event finals, the winner is the gymnast with the highest composite difficulty and execution score. If a tie occurs, the gymnast with the higher execution score is declared the winner.  By definition, this rule rewards the gymnast who took less of a risk and performed the easier routine, but made fewer errors.

Progressing in Swimming v. Track & Field (Athletics)

At first glance, Swimming and Track & Field seem quite similar in their structure and goals (i.e., to have the fastest time).  Events in both sports contain multiple heats in which athletes must succeed to progress to the next round, and ultimately make it to the finals, or medal round.  Although in both sports, the winner is determined by total time, the qualifying rules in the two sports are quite different.

• Swimming – In Swimming, the top eight times across all heats advance to the next round. In theory, all eight swimmers from one heat could advance or no swimmers from one heat could advance dependent upon the overall ranking of times.
• Track & Field – In contrast, in Track & Field, the top runners in each heat are guaranteed a spot in the next round. The remaining spots in the next round are filled by the non-automatic qualifiers with the best times across all heats (as in swimming).  The result is that the finals in Track & Field may not contain the eight runners with the fastest times in the semi-final.

The Importance of Consistency

In sports with multiple heats or rounds like Swimming, Track & Field, Gymnastics, or Diving consistent performance is valued – to a point.  Obviously, some level of consistency is required to meet the standard needed to move from one round to the next and make the finals.  Under current rules in each of those sports, however, performance in previous rounds does not count as the athletes move forward.  Ultimately, it is only performance in the final round that determines who receives the medals.  This was not always the case. In Gymnastics, for example, scores from the preliminary were included in the final composite score through the 1988 Olympic Games.

Faster, Higher, Stronger – at the right time

Just as consistency is valued, but only to a point, the same can be said for excellence.  As diver Tom Daley can attest, an athlete may set an Olympic record or even a world record in a preliminary round and ultimately, not earn a medal or even make it to the semi-final round.  The gold medal winner is the individual or team that performs best in the final round, which may not have been the best performance displayed during the meet.

Generalizability

In one sense, there is not an expectation within the sports world that the results from the Olympics Games are necessarily generalizable beyond the Olympics.  Until a more extensive body of evidence is compiled, people will expect Tom Daley to perform very well in championship diving meets. Performance in the Olympics may have an inordinate impact on an athlete’s fame, fortune, and legacy.  However, a non-repeated performance (good or bad) in a single, high-profile meet should not have a lasting impact on the evaluation of that athlete’s overall ability or level of proficiency within their athletic community.   (That is not to say that an exceptionally good or poor performance at a high-profile event like the Olympics might not have a lasting impact on an athlete and her or his performance.)

On the other hand, the semi-finals and finals at the Olympic Games are the culmination of a long filtering and pruning process.  As implied above, there is an expectation that athletes competing in this championship meet will largely be the same group excelling in other meets throughout the season.  In other words, there is generalizability in the process that discriminated between these athletes and non-Olympians in their sport even if there is variation in performance within this elite group across individual meets.

Comparability

Although it would make little sense to use different criteria to compare the performance of two gymnasts, there is not an expectation that the same criteria will be applied to gymnasts, swimmers, and runners. Despite variations in rules, each of the sports contested during the Olympic Games produces a winner who earns a gold medal and is proclaimed Olympic champion.  Debates about whether the swimmer or runner is a better athlete or whether the basketball or soccer team is a better team can make for interesting exchanges across the dinner table, at a bar, or on Twitter (and mind-numbing discussions on sports radio), but that is not really the point of the games.  Different sports require different skills, have different groups of competitors, and may have different values built into their rules and procedures.  We understand and accept that.

Four Lessons Learned for School Accountability

Like the sports federations, states will be operating under a common set of rules as they design their accountability systems and those systems will produce a common set of outcomes (e.g., identify the bottom 5% of schools, identify subgroups within schools requiring targeted assistance).  Although the indicators used by states will largely be the same, the judgments and decisions that states make regarding the design, scoring, and reporting of results from those indicators can lead to very different outcomes.  The first lesson learned from the Olympics is to ensure that the decisions made throughout the system on individual indicators and in on the composite score accurately reflect the values of the state.

A second lesson learned is to really understand the first lesson.  A state should be able to explain the decisions made throughout the design process.  There should be a rationale for each decision, and ideally there should be evidence to support the decision.  However, as in the case of penalty kicks in soccer, the rationale may be that nobody has thought of a more practical solution – yet.  For example, practical or political constraints may dictate that the annual mathematics assessment must be administered in no more than 2.5 hours, cost no more than $15 per student, and results must be returned within 10 days of the end of the testing window. Those constraints will certainly impact other decisions and outcomes.  That’s not a problem, as long as the impact of those constraints and decisions are understood and can be communicated clearly.

The third lesson is to understand the level of discrimination that the system is being asked to make and is capable of making.  First, a system designed to discriminate among the top 8 or top 24 schools is not likely to be the same system that one might need to identify the lowest performing 5% of schools.  Second, discriminating among schools within the top 24 or bottom 5% is much different than building a system that differentiates between those two groups.  To return to the sports analogy, in the former case it may be necessary to time races to the thousandth of a second, but in the latter tenths of seconds or even whole seconds might be sufficient.  Build the system that does what you need it to do and understand what it does not do.

The fourth lesson learned is that stuff happens that may impact results.  Your system may not be able to account for high winds during a diving competition or driving rain during a track meet.  Things may happen that positively or negatively impact a school’s accountability results in any given year.  As long as no serious and long-lasting consequences result from a single year’s accountability results and those results are presented in context, all can be well.

With those lessons in hand, we will be ready when the winter of 2018 brings us the next Olympics games and the administration of assessments for our new ESSA accountability systems.

Let the games begin!