Berkeleyan
Statistician's new method will test election outcomes
'Auditing for confidence' on the basis of observed discrepancies
| 06 February 2008
This week's California primary offered the first test of a new procedure for conducting hand tallies to verify election outcomes, finally giving election officials a reliable way to judge the accuracy of the vote count or to tell them how much to expand hand tallies in the event of a close race.
Fewer than half of all states require or allow audits, and many of the 19 that do, including California, mandate a hand count of a flat percentage of precincts selected at random in each county in the state.
"Despite the fact that these audit laws have been on the books, the laws haven't really told election officials what to do when they find problems," says Philip Stark, professor of statistics and author of the new post-election auditing procedure. "When you count by hand, what happens if you find mistakes? In California, the law doesn't say. If you find big errors, do you have to look for more? When do you stop counting? It's entirely up to election officials to certify or not certify the elections in their jurisdictions."
"My method," he continues, "is the first that, given discrepancies you observe in the sample counted by hand, tells you whether you need to sample more or not, until eventually you get to the point where you can be, for the sake of argument, 99 percent confident that a full recount wouldn't give you a different outcome. It tells you when to stop auditing, how to know when you've found little enough error in a big enough sample that you can be confident that the results are right."
Stark obtained the cooperation of three counties, including Marin County, to test his method on election day this week; he also will look at statewide results after the fact. If the method proves workable, it may be tested more broadly in the June 3 direct primary and could end up being used statewide in the Nov. 4 general election, Stark says.
"I can't audit the primaries in real time unless all counties are willing to play along," which would be difficult to arrange, he says. "Luckily, Marin County has a measure on the ballot that is entirely contained within the county. That is crucial to testing the approach, because we have to increase the amount of auditing incrementally across the set of precincts that comprise the race, depending on the discrepancies that are found."
Stark was introduced to the problem of post-election audits after being appointed in June 2007 by California Secretary of State Debra Bowen to the Post-Election Audit Standards Working Group, which was tasked with developing standards and best practices for random hand tallies (audits) used to assess the accuracy of machine vote counts. The need for a statistically sound way of judging the accuracy of vote counts became more imperative after Bowen in August recertified some voting systems but insisted that hand audits be conducted to assure a certain level of confidence that the outcome is right.
"The working group recommended that a method that does what my method does should be used, but no method existed to solve the problem," Stark says of the working group's final report, issued July 27. "So I developed a method."
According to Stark, other experts have tackled this problem, too, but primarily from the perspective of computer science, not statistics. Computer scientists figured out how much to audit in order to detect an error if the outcome is wrong - though in practice, he says, audits always find errors.
Stark approached the problem from the opposite perspective, asking the question, "Assuming there is error, given the amount of error you find, how confident are you that the counting equipment found the right winner?"
Says Stark: "What you really need to know is, Are the errors you uncover in an audit small enough to give you comfort that, overall, there isn't enough error to throw the election? Or are they large enough to cast the outcome into doubt? Or do you need to count more to tell?"
Using Stark's method, election officials will have rules to follow based on the margin in the race, the number of precincts in the race, and the vote counts in each precinct. The approach allows errors in different precincts to be treated differently to reflect voting technology or precinct sizes.
For example, if a hand tally of a randomly selected set of precincts differs from the machine count, election officials can increase the number of audited precincts until a chosen level of confidence in the outcome is reached - and then certify the count - or until a complete recount reveals the true winner. Hand recounts trump machine counts in most states, he says.
Neck-and-neck races, such as the 2000 Florida face-off between George Bush and Al Gore, would require a complete recount, according to Stark's method. At the other extreme, races with large margins would not need even the 1 percent hand tally mandated by California, or the minimum percentages required in other states.
Two tests in '06
In a paper accepted for publication in the journal The Annals of Applied Statistics, Stark reports tests of his approach with two races from November 2006: the U.S. Senate race in Minnesota and, in California, a Sausalito-Marin City School District school-board race, a small contest in which voters could vote for up to three candidates.
For that school-board race, which involved nine precincts and a razor-thin margin of 43 votes, a total recount would have been necessary to be 99 percent confident of the outcome, he found. Instead, the county conducted an audit of only 1 precinct, as required by law. In the Minnesota race, Stark's method showed that Minnesota's auditing rules, mandated for presidential and congressional races, provided high confidence that the right winner was named.
Stark said that the California Secretary of State's office is looking closely at his approach, and will provide him with the hand tallies from this week's election that the state now requires counties to submit. The real-time test in Marin County and the post-election look at statewide races, which should be analyzed by spring, will be key tests of the approach.
"From those data, I am going to be able to go back and answer the question, 'What is our confidence that we found the right winner, given the discrepancies that the audit uncovered?' " Stark says.
Stark acknowledges that many counties, particularly large ones like San Diego and Los Angeles, may balk at the additional auditing that his "auditing for confidence" approach may require. As a result, it may take some convincing to get the state legislature to accept his approach, he says.
On the other hand, New Jersey's new auditing law seems to require a method like Stark's. In an effort to limit somewhat the cost and bother of hand tallies, New Jersey requires a lower level of confidence (90 percent) for lesser races compared to major contests (99 percent).
"If machine counts usually find the right winner, say, 99 percent of the time, then even if you only audit for 50 percent confidence, on average less than one in 200 certified outcomes will be wrong," he says.