An opinion poll or an exit poll is basically a survey based on a small proportion of the population. And the prediction from the survey is not likely to match exactly the true result that could be obtained by interviewing all members of the population. The margin of sampling error quantifies how close one may expect a survey result to fall relative to the population value. A “margin of error” of ±3 percentage points at a 95 per cent confidence level indicates that if the survey is conducted 100 times, one might expect the result to be within 3 percentage points either side of the true population value 95 of those times, on an average.
Thus, with a ±3 per cent margin of error, for example, a 60 per cent prediction indicates a range of 57 to 63 per cent is actually predicted — be that for voting or seat percentage. Clearly, a ±3 per cent prediction in an election having only 70 seats indicates that the prediction should be as precise as within ±2 seats. However, the pollsters even provided a prediction with a range of ±5 or ±7 seats (which is a range of ±10 per cent) for Delhi election, and they are now celebrating their success! These predictions were very loose by any standard. Since the overall trend was highly skewed favouring AAP
in Delhi, such predictions don’t appear to be bad-looking.
Yet, imagine if pollsters predict a 32 (±6) seats for any party in an election having 70 seats with only two contesting parties, and both 38 seats (which is victory) and 26 seats (a defeat) in the final outcome are cheered as correct predictions. Statistically, this is what happened in Delhi. And, what’s more, if a ±15 per cent margin of error is allowed, the prediction has a wide range covering 30 percentage points, which is absurd. Yet, people are branding it a success! Certainly, the margin of error can be reduced by increasing the sample size. Calculation of the number of seats from the voting percentages is a more complicated process. However, pollsters need to calculate for the seats also, separately for each election, if they aspire to predict the number of seats by different parties in our first-past-the-post system. And, if a 20 or 30 percentage points of error range is projected even for an effectively two-party contest like in Delhi, the predictions will be more off the mark in a multi-party contest, and in states with more economic, religious, and social heterogeneity. Let’s consider a simple illustrative example. Suppose, a 55 per cent support favouring a party is estimated with ±10 per cent margin of error using a sample size of 95. Then, a ±5 per cent margin of error could be achieved by increasing the sample size to 380. And, the margin of error could be made to ±3 per cent by a sample size of 1,056. Simple statistical theory can be employed for such sample size calculations, and these are needed a-priori to achieve a ±3 per cent projected margin of error considering the heterogeneity of the concerned population appropriately — be it for a pan-Indian vote, or the states.
This is a problem not limited to Indian pollsters as highlighted in an article published in the Journal of the American Statistical Association in 2018, written by Professor Andrew Gelman of Columbia University jointly with Houshmand Shirani Mehr and Sharad Goel — both from Stanford University — and David Rothschild of Microsoft Research. They compared 4,221 late-campaign polls including 608 state-level, presidential, Senate and governor’s races of the US between 1998 and 2014, to find the historical margin of error to be plus or minus 6 to 7 percentage points. This means that the error range was 12 to 14 points for most poll predictions in the context of the US — not the typically reported 6 per cent. Pollsters in the US, too, are criticised for their not-so-accurate estimation. The same standard should be applied for their Indian counterparts.
The writer is a professor of statistics at the Indian Statistical Institute, Kolkata