Friday, April 18, 2014
Friday, August 3, 2012
Who: Alexander Holtzman
What: Stanford University honors senior thesis
IRV FactCheck Rating: NOT CREDIBLE
Ranked Choice Voting (RCV) opponent Terry Reilly has recently been distributing and promoting by email and news article a 2012 undergraduate thesis by Alex Holtzman, titled “The Unanticipated Inequalities of Electoral Reform: Racial and Ethnic Disparities in Voting Behavior under Oakland's Ranked Choice Voting Program ”.
As a result of his paper, Holtzman was one of twenty-eight students awarded Stanford University's Firestone Medal for Excellence in Undergraduate Research in 2012. Terry Reilly has highlighted the paper in support of RCV repeal efforts in Oakland and San Francisco (both repeal efforts failed) and to discourage the use of RCV in other parts of the country.
You can probably look just about anywhere and find voting behavior differences along some set of demographic characteristics, regardless of which election method is being used. However racial and ethnic differences still deserve special vigilance. But racial and ethnic disparities are generally known to still exist with traditional election methods. So what should be of particular interest in evaluating RCV implementations is not just whether flaws exist, but how any flaws have changed, what are the causes of those changes, and what are good ways to mitigate the flaws and continue the improvements.
Much of Holtzman's quantitative analysis purports to compare Oakland, California voter turnout data for the mayoral contests in 2006 and 2010. The 2006 contest used a traditional June/November runoff system and was won in June; there was no November runoff. The 2010 contest was decided with one RCV election in November. In both years, local elections were consolidated with state-wide elections in June and November.
Overall Holtzman finds mixed support for his hypothesis that an alleged increase in complexity associated with voting under RCV causes a decline in voter participation, especially among traditional racial and ethnic minorities. The effects he finds tend to be small and sometimes don't support his hypothesis. But a bigger problem is that his quantitative analysis appears to be seriously flawed from the start.
Holtzman does not actually use any turnout data from the June 2006 mayoral contest. Instead he uses November 2006 turnout data, when there was no mayoral contest, and compares it to November 2010 turnout. He also uses overall turnout data for each election rather than contest-specific data. As a result, he finds an overall increase in turnout of only 1 percentage point. There was actually an increase of 14.1 percentage points between the two mayoral contests (46.0% for June 2006, 60.09% for November 2010, a 30.6% relative increase).
A quantitative analysis that starts with grossly erroneous data and understates the overall effect by more than an order of magnitude (1% vs. 14.1%) just can not provide a credible analysis of the components of that effect. This problem with the choice of data spoils both the within-city analysis of Oakland alone and also the difference-within-difference model that additionally introduces turnout data from Long Beach, California.
The principle claim that RCV proponents make about increased turnout, compared to a traditional two-election runoff system, is not that RCV necessarily brings more voters out to vote, but that it allows the entire contest to be decided in a single election that can be scheduled when the most voters are likely to be voting anyway. RCV helps avoid the one lower-turnout election. In Oakland, the June primary was the low-turnout election. In San Francisco, which used a November/December runoff system before upgrading to RCV, the December runoff was most often the low-turnout election. By using the wrong data, Holtzman's paper fails to investigate this claim.
The 2006 versus 2010 mayoral comparison for Oakland has been written about previously, both overall and by city council district, showing significant improvements in voter participation. Perhaps the best overview of how RCV has helped increase voter participation is a poster produced by the Ella Baker Center for Human Rights and Oakland Rising.
Holtzman also does a ballot usage analysis that only looks at the Oakland 2010 mayoral data. However his analytical framework is flawed and too often he extends his conclusions beyond what his data shows. Marking a choice for only one candidate does not mean that the ballot will be exhausted or that the voter is disenfranchised or confused about RCV. Holtzman does not investigate actual vote exhaustion rates. Without comparisons to Oakland's experience with its traditional two-election runoff system, the results Holtzman does find beg the question of whether the RCV experience was an improvement. RCV often has lower rates of exhausted votes than comparable contests using a traditional two-election runoff system. The advantage tends to grow as the number of competitive candidates increases and is evident in Oakland.
Because of these and other serious flaws with the paper, most of the core results should not be relied upon and likely are significantly wrong. I contacted both Holtzman and his thesis adviser, Clayton Nall, to share my concerns about the paper. Both declined to discuss the substance of those concerns.
So Holtzman's undergraduate thesis gets an overall IRV FactCheck rating of NOT CREDIBLE.
Some of the ancillary results in the thesis may have some validity, but without a demonstrated comparison to traditional runoffs and a consideration of compensating effects, neither those findings nor the thesis as a whole support Terry Reilly's contention that RCV is a failed reform.