Rotten Luck and Self-Inflicted Wounds – How the U.S. Pollsters Failed in the 2016 Presidential Election and Why a Similar Result is Unlikely in Australia

Martin O’Shannessy is a partner at OmniPoll and a former CEO of Newspoll with a 100 per cent record of calling state and federal election outcomes. Polling is now part of so many aspects of our daily lives – from product pushing to poll pushing. The art of the pollster is complex and many factors are at play across the globe. How could the opinion polls in the by-election for Longman be so inaccurate – leading to false expectations and the loss of a prime minister weeks later. In an address to The Sydney Institute on Tuesday 11 September 2018, Martin O’Shannessy gave a comprehensive overview of how opinion political polling works.

ROTTEN LUCK AND SELF-INFLICTED WOUNDS – HOW THE U.S. POLLSTERS FAILED IN THE 2016 PRESIDENTIAL ELECTION AND WHY A SIMILAR RESULT IS UNLIKELY IN AUSTRALIA.

MARTIN O’SHANNESSY

Most polls conducted in the run up to the 2016 U.S. presidential election predicted a Clinton win. Previously successful poll aggregators (who look at groups of polls) including Nate Silver’s Five-Thirty-Eight predicted a 70 per cent to 90 per cent chance of a Clinton win on the night before the polls opened. This was affirmed by an almost unanimous consensus among pundits like Frank Luntz, who Tweeted a 99 per cent chance of Clinton win, that a Trump victory was impossible. The surprise result that saw Trump take 57 per cent of electoral college votes has been cited as a reason to disbelieve polls in the United States and possibly Australia. So, what are the reasons for the failure of polls to predict the U.S. election and will the Australian polls be subject to the same issues as we move to our next election?

To answer these questions, we need to look at how to do polling well; whether it was done well in the US 2016 context; what features of the US system add complexity and how this contributed to the failure of polls to point to the correct outcome. We can then look at the Australian situation to compare the complexity of the two systems and the polling landscape. We can then ask whether, in view of the differences, we can better trust the Australian polls to truly guide us to the next Australian election.

Polling done well 

Rather than indulge in a lecture, I’m going to tell you a story about George Gallup.

Gallup, father of modern polling is best known among we pollsters for the 1936 Presidential race between Democrat, Franklin Delano Roosevelt and Republican Alf Landon. In the run up to the election, the prominent magazine, Literary Digest, conducted a poll that attracted over 2.3 million responses predicting a clear win to Landon. The Gallup poll was based on just 1500 interviews and predicted a win to Roosevelt.

The Roosevelt win is now history and Landon is all but forgotten but how did Gallup’s minute sample give such accuracy compared to the huge sample of the Literary Digest poll?

The simple answer is that the Literary Digest polled the wrong people. Their poll was based on a mail out to registered car owners and telephone subscribers (the most well-heeled of American society in 1936), the Gallup poll had been conducted among people entitled to vote from all walks of life including both sexes, African Americans and other less well-heeled citizens. The Gallup poll was conducted in a wide range of locations and was randomly recruited so every potential voter had a chance to participate in the survey. The Literary Digest poll excluded the vast majority of voters from any chance of participating because the only people eligible to participate were on the mailing lists used to recruit respondents for their survey.

So, Gallup showed in practice by this poll that to do polling well, certain basic rules must apply. First, every potential voter needs a chance to participate in the poll; second, the structure of the sample must reflect the population.

But the complexity of the U.S. system makes it hard to reflect the population

It is certainly harder in the United States than Australia.

In Australia, we have 150 federal electorates. Thanks to the Australian Electoral Commission (AEC), each electorate has approximately the same population, returns the same number of representatives (one), has separate elections and ballot papers for each house of parliament and turnout is compulsory and predictable. So, nationally sampled polls predict national two-party vote in this system with great accuracy and the national two-party vote is a reliable guide to the number of seats to be won by each party.

This differs from the US system in every important respect. The voting unit is a state; each of the 50 states has a different population and returns a different number of electoral college votes. A vote for an individual presidential candidate is actually a vote for his/her slate of electors effectively electing both The House of Representatives and Senate with one ballot choice. In addition, while most states take a “winner takes all” approach to awarding their electoral college votes to the overall winner within the state, two states don’t (Nebraska and Maine who allocate electors proportionally) and while the popular vote is traditionally followed by electors, there is no constitutional compulsion on electors to allocate their vote in any particular way in the Electoral College process. For individuals, voting, even registration is not compulsory, so turnout is unpredictable and voluntary. The national two-party vote has proven (in 2016 at least) to be a very poor indicator of the number of electoral college votes to be won by each party

I make no comment on whether either national practice serves democracy better. But I do say that the Australian system greatly simplifies the task of pollsters to create polls that can reflect the population at reasonable cost and for reasonable effort. The US system is significantly more complex and requires many more (educated) guesses to estimate and many more dollars if it is to be to be polled well.

To illustrate the risk introduced by complexity let us consider the impacts of making a 1 per cent error in two U.S. states and two Australian electorates.

Let us say that you, the pollster have correctly called 48 out of 50 states but have got it wrong by 1 per cent in Florida and Wisconsin. Each state is estimated by you at 51-49 in favour of Clinton. However, the real result is 49-51 in favour of Trump. The statistical error you have committed at a national level would be invisibly small. But the number of Electoral College votes you called for the wrong party would be 39 votes (Florida 29, Wisconsin 10) out of the total of 270 required to win the presidency a prediction error of 14 per cent from an estimation error of 1 per cent.

In Australia things are very different. Should you, the pollster, call two marginal electorates to the wrong party, they will simply cancel one another out in 50 per cent of cases. Even in the 50 per cent of cases where they don’t cancel one another, the error generated by getting it wrong in two Australian seats represents just 2.6 per cent of the total seats required to form government. Quite different from the 14 per cent error that would have resulted from the same statistical error afflicting the pollster in the United States.

So, similar levels of statistical error can have much higher impacts on the ability of a poll in the United States to predict an outcome compared to Australia.

The practical benefit of our less complex system is less complex polling. The vast majority of federal election polls I have conducted have been based on a single, national poll of about 1500 responses supplemented by a poll of up to 1500 in marginal seats to improve our chances of avoiding small errors near the margin where seats change hands and new governments are formed. These polls have been consistently accurate to less than 1 per cent in my time and averaged +-1.5 per cent across the history of Newspoll to my departure.

One-off issues making it harder in 2016

Adding to the inherent difficulty of polling the U.S. system, several one-off features also made it harder to conduct good polls and interpret them well in 2016.  Trump supporters were under-polled but made an unusually high turn at out the ballot box. There was also a strong divide in voter preference based on education levels in 2016. This had not been the case in 2012 and many pollsters therefore did not add this into their calculations. There was also a higher turnout of disaffected voters who had been previously “un-pollable” meaning that polls just missed some Trump voters and could not take their views into account.

What is the right approach to polling the U.S. System?

Given the need to reflect the structure of the voting system being polled, there should have been up to 50 (one in each state) highly accurate polls conducted door to door or on the telephone.

National polls, while fascinating and ironically twice as accurate as the state polls, should have been ignored by the pundits as they are not an inherently good predictor of electoral college votes.

How was it polled in 2016?

There were at least 39 national polls published. These were, on the whole well managed, well conducted, high quality, accurate to within 2 per cent and irrelevant.

There were at least 208 state level polls. These should have predicted the outcome, but they were, on the whole done cheaply online. Only 12 per cent of all the state level polls were conducted using high quality methods such as human operated telephone.  The other 88 per cent were conducted using cheaper and unrepresentative online panels or other low-cost, low-quality methods. So, the critical state polls were mostly wrong with errors averaging 5 per cent across the board while the irrelevant national polls were mostly right with errors averaging just 2 per cent.

So how did they get the state polls so wrong?

The polling industry and their main customers, the media can be excused for having some very rotten luck – which I will discuss next – but they also stand convicted of mainly self-inflicted wounds in the form of poor technique.

The bad luck…

Pollsters in the United States are forced to weight their results by expected turnout. This is because a person may say they favour Trump or Clinton when polled but the fact is their vote only counts if they turn out and vote. As stated earlier, the 2016, turnout nationwide grew more in Republican counties than Democratic counties, relative to 2012.

In 2016, many pollsters assumed similar turnout ratios to 2012 and weighted their results to reflect the 2012 turnout across parties. This unusually high Republican turnout was not predicted causing under-estimation of Republican votes and over-estimation of the Democrat vote in a number of counties.  In short, on turnout many pollsters zigged when they should have zagged.

The self-inflicted wounds

Poor survey technique – It seems amazing to say it (at least to me), but many state polls did not weight for education. This was a serious, avoidable error. At Newspoll (and now OmniPoll) our education weighting matrix is part of our secret sauce as education levels are intimately tied to voter perceptions. In the US, education levels are also tied to that other complicator, voter turnout. In 2016 a stark, education related divide in voter preference was evident at the ballot box (Trump lower educated and Clinton higher) but many state pollsters did not allow for this in their analysis.

Misleading poll results were amplified by aggregators – Poll aggregators, who appeared to do so well in 2012 are not pollsters. These included Real Clear Politics and Nate Silver’s Five-Thirty-Eight. Aggregators input the published work of the pollsters to their models to predict outcomes. The mathematics used in these models is usually very good. Some of the maths used in the early US nuclear programme known as Monte Carlo simulations feature in the models used by aggregators.  Each has their own way of weighting published polls for recency, quality and credibility of the pollster within their models.

The caveat is, that however good the model may be, the prediction it makes can never be better than the inputs you give it. In 2016, the vast majority of the relevant state polls used by aggregators were of poor quality. No matter how previously successful the aggregator model may have been, the old adage of “Garbage in, Garbage out” always applies and it did so in spades in 2016. Five-Thirty-Eight’s prediction of a 70 per cent to 90 per cent chance of a Clinton victory was regarded as a bit less confident than some others.

There was a vicious cycle of self-confirming inaccuracy – aggregator predictions are influenced by the volume of polls they access. Whatever weights you use in a model, there is a very high chance that inputting many polls that say Clinton will win, will result in the model predicting a Clinton win.

The inaccurate state polls were echoed by the aggregators to reinforce the pre-formed view of many media outlets that Trump could not win.

This was backed up by the national polls that predicted accurately but irrelevantly that Clinton would win the national vote as she did. And, in the glow of 20/20 hindsight, we also now see that many media outlets were focussing on the polls that confirmed their judgement about the outcome and those outlets predominantly favoured Clinton.

So why wouldn’t this happen in Australia?

The single most important reason for Australian polls to be more accurate on the whole is that our system is simpler and therefore more easily measured.

The effect of our compulsory enrolment and turnout is to create a homogenous system.  Every citizen, virtually everybody we reach in our surveys, is a voter who is highly likely (nay compelled) to turn out on election day. The result is that national polls most often reflect voter behaviour on election day. The homogeneity of our system means that most times, reading our electoral pendulum for seats to change hands has proved a good pointer to the result. Because of these factors, a good indicator of any state or federal election outcome can be achieved with two good polls; one national and one in marginal seats (or at most 8 for our states and territories but not the potential 50 polls required to be sure about every US state).

Because the Australian system is easier to measure, it is also cheaper to measure well. As a result, our polling landscape favours quality to a greater degree. This leads to a few dozen, higher quality polls being published during a federal election campaign. This will be done on behalf of four or five media outlets not hundreds.

As a result of these factors, the vast majority of polls I have conducted or come in contact with have had adequate budget for live phone interviews either wholly or with some opt in panel. Sampling and weighting are also generally of high quality (especially because the AEC publishes accurate data on the age and sex of electors which all pollsters can access when they design their samples for very accurate sampling and precise weighting).

So how have Australian polls performed lately?

Technically the margin of error we expect on a national poll of say, 1200 responses is up to 3 per cent either side of the real result. Statistics tell us that, most often, the difference between the poll and the outcome will hover around 1.5 per cent. Federal election polls published since I took over Newspoll have generally been accurate to within 1 per cent or less. In every case since the foundation of the company in 1985 and in the last election, the results given by the major polling companies have accurately predicted the national vote and been a good guide to the outcome of the election. This compares very favourably to an average error of just over 2 per cent for the US national polls and just over 5 per cent for the state level polls in 2016.

But surprises can happen – even in Australia

It is worth remembering that a poll most often tells us the overall result in the electorate or the whole nation. In Australia it is possible but rare for the actual two-party vote and the actual electoral outcome to diverge. The pollsters can, as they did in the United States accurately tell us the vote but not the winner. Three recent Australian elections reflect this. Opinion polls accurately reflected the actual state or national vote in each case but the distribution of votes at an electorate level gave us a different Prime Minister or State Premier to the one we were expecting. These elections were 1990, Federal Labour won with Bob Hawke on 49.9 per cent of the actual vote; 1998, A Federal Coalition win to John Howard on 49 per cent of the vote; 2010, Labor in South Australia for Mike Rann on 48.4 per cent.

Prospects for polling in 2018-19

Pollsters, on the whole do their best to accurately reflect what is current opinion and the reality we face is that no poll is truly predictive of the future.

While some bad luck and significant poor practice have been found in the U.S. polls in 2016, two major issues for U.S. pollsters are cost pressure and a complex environment that is expensive to measure well.

I also advance the proposition that the tendency to view polls as “content” by media proprietors is contributing to an interest in volume not accuracy on the part of research buyers who would prefer to have many cheap polls to boost circulation many times rather than one good poll to predict the election outcome.

The Australian pollster faces some of the same cost pressures as the U.S. pollster but for now, our system remains endemically easier for a diligent pollster to measure well at a reasonable cost.

Assuming that there is no dramatic decline in the budgets provided by media proprietors for the polls they publish, I believe that there is every reason to expect the polls we see in the coming year to be a good guide to the outcome we may expect at the next federal election.

Looming challenges

The Australian electoral system and polling landscape lend themselves to better quality outcomes more often than the U.S. system. However, Australian pollsters do face a major challenge as technology consigns the land line phone to history. You will recall that a good sample is based on giving everybody in the population of interest a chance to be selected in the survey. In the recent past, this mainly involved selecting phone numbers from published lists that had well known geography (permitting effective quota sampling) and were generally available to researchers – that is the land line phone as listed in the White Pages. Fewer people now use landlines and the ability of the landline to represent everybody is falling as using a landline sample now excludes at least 30 per cent of adults from the possibility of being surveyed. Because of this, using the landline is becoming more costly to do well and will ultimately become unreliable.

Some market researchers have dropped phone altogether and opted for online panel samples. These samples are often used to provide cheap, indicative data but they exclude 98 per cent of the population from a survey run on a panel making them unsuitable for the highly accurate (and highly scrutinised) business of polling.

The mobile phone, which now outnumbers people in Australia is the viable alternative as using the mobile phone does include virtually everybody (93per cent and rising). However, use of the mobile phone presents one key problem that government has been slow to address despite years of warnings and lobbying by the research industry. Mobile phones reach almost everybody but there is no published phone book covering mobiles that would allow researchers to build samples based on geography. Without this, the very accurate methods required to do quality national polls or any good level of state or electorate poll is impossible on the mobile phone.

There is a global phone book known as the IPND that could be used for sampling. However, unlike the White Pages it is not published, and special permission is required to access it. This permission is rarely given and comes at high cost. Industry needs access to de-identified mobile phone lists from the IPND for research purposes and has worked with the Department of Communications and The Arts over ten years to develop a mutually agreed scheme to safely and privately provide that access. The scheme has been developed and recommended by the Department but repeated failures to finally act on these recommendations at a legislative level have to date left industry in a steadily worsening position.

This problem faced by pollsters and market researchers is an issue for the taxpayer. About a third of all the work done by major market research operators in Australia is done to plan services, measure outcomes and design communications for government. When CEO of Newspoll and Chair of our industry body AMSRO, I estimated that the extra work required to cope with poor sample quality added up to 20 per cent to the cost of our high-quality government work. The taxpayer stands to benefit along with the economy generally if this problem is finally solved by government adopting the recommendations jointly developed by industry and the Department.

Australian pollsters are still getting it right because of the nature of our system and polling landscape. We face issues than can be solved legislatively and, as in all things, complacency at any level will be rewarded with failure.

The major reference for this article is an inquiry into the issue conducted by the American Association for Public Opinion Research

An Evaluation of 2016 Election Polls in the U.S. Ad Hoc Committee on 2016 Election Polling

https://www.aapor.org/Education-Resources/Reports/An-Evaluation-of-2016-Election-Polls-in-the-U-S.aspx