Ipsos MORI June 2008 Methodology Review

Political Polling Methodology Review
June 2008

Overview

Ipsos MORI has a long history of leading the industry in terms of our transparent approach to political polling. We believe that in order for our clients, the media and other interested observers to be able to utilise and understand the outcomes of our research, we must make it clear how we came by these findings.

We also have an excellent record of producing accurate political research over four decades. Our final pre-election poll at the last British general election was within two-thirds of a percentage point on average in predicting the shares of the three major parties, and the exit poll that we conducted jointly with NOP for the BBC and ITV provided the data that enabled the exact prediction of the government’s majority in seats (66).  We are especially pleased about this, considering that 10% of the public tell us in post-election polls that they decide for whom to vote in final 24 hours of the election. In addition, our on-the-day poll of the 2004 London Mayoral election also correctly indicated the final result to within one percentage point. We are naturally proud of this record and want to maintain it.

The challenges to conducting accurate research are constantly changing and evolving, and it is good practice for pollsters to periodically review their methodologies and if necessary make amendments or adjustments to ensure that their findings continue to be reliable. In recent years, for example, the polling industry has had to face hurdles such as the growth in the number of mobile-only households and higher refusal rates, and more subtle changes in the way the public behave or take part in surveys may have an effect in less obvious ways. In addition, we would be remiss not to undertake experimentation and work around online political polling, and this work is ongoing. The best way to tackle challenges as they arise is to continually review and improve our methodologies, growing and adapting along with changing cultural and technological norms.

This review of our polling methodology is therefore part of an ongoing process. It arises partly from some apparent inconsistencies in voting intention findings between our face-to-face and telephone surveys, which we began to investigate at the start of 2008, and also considers some questions raised by polling during the London Mayoral election campaign this spring. After some months of research, experimentation and debate, we have come to a number of conclusions, which are outlined below together with the steps we intend to take as a result. However, there remain some outstanding questions, and our investigations continue: as such, we will continue to publish findings and make adjustments as necessary in future.

This review is primarily concerned with the measurement and interpretation of voting intentions. Many aspects of this question are specific to it, such as the effect on responses of the question ordering and the best means of distinguishing those respondents who will vote from those who will not; these have no direct bearing on our other research. But issues such as sample representativeness naturally have a wider relevance. Nevertheless, all references to ‘polling’ in this document refer exclusively to political polling at Ipsos MORI, and conclusions are not necessarily applicable to other types of research.
 
Issues addressed in the review

The review has attempted to address a number of issues raised by the findings of our recent polls.  

1. Sample representativeness
We regularly review the representativeness of our samples for national surveys, updating targets for weighting, revising the composition of the sampling points used in our face-to-face Omnibus survey, and checking our findings and sample profiles where possible against independent sources. This became recognised procedure especially following the MRS Review of the 1992 election with which several of us were involved, among other contributions, introducing the ‘Spiral of Silence’ concept to the review.

We carried out such a review of sample representativeness in the first months of 2008, looking at both the face-to-face Omnibus on which our Political Monitor survey has been run, and the telephone surveys which we frequently use for ad-hoc political polls (including our telephone Omnibus).

2. Polling during the 2008 London election campaign
We did not conduct a final “predictive” poll in the London Mayoral elections: fieldwork for our final survey was completed a week before polling day, and hence (as we made clear at the time) could not be considered a suitable basis for a prediction of the election result. However, even at this stage in the election, the ‘turnout’ projections our polls suggested were unrealistically high, with 61% of the public saying they were ‘certain’ to vote on election day. Turnout on election day was actually closer to 45%. Our polls also found slightly higher support for the incumbent Mayor, Ken Livingstone, than he achieved in the eventual election, which may also have been related to turnout since anecdotal evidence suggested that Boris Johnson’s team performed better at turning out their vote. We have therefore investigated possible improvements to our methods for identifying likely voters, though also bearing in mind that voters can behave very differently in local elections from general elections, and that any lessons derived from the one may not necessarily apply to the other.

3. Inconsistencies between telephone and face-to-face voting intention findings
For the past 12 months, we have noticed a difference between the findings of our (published) face-to-face and (mostly unpublished) telephone voting intention data. While not always so large as to be statistically significant between a given pair of polls, the discrepancy has been consistent, with the face-to-face data showing higher Labour shares than our telephone data. This is a new development, as our telephone and face-to-face polls were achieving consistently comparable results at the time of the 2005 general election.  

4. Presentation and understanding of polling
Like all polling organisations, we are largely dependent on the media to disseminate the findings of our surveys, and usually have little direct control over the way in which these are reported. Poll findings are sometimes misunderstood by journalists, or reported under a misleading headline or in such a way as to ignore the limitations inherent in the research. It is our duty to prevent misreporting or misinterpretation of our research to the best of our ability, but at the same time to encourage the widest reporting of our published findings providing this can be done accurately. We have therefore reconsidered the way in which we publish and present our political poll findings, in the hope that we can make them – and their limitations – easier to understand.

Findings

1. Sample representativeness

Examination of some of our surveys over the last couple of years, including some work carried out for trade unions, has revealed that some of our political polls contain higher proportions of public sector workers than are present in the population. This appears to be the case with both our telephone and face-to-face quota surveys. There is no clear evidence of the reason for this, but it may be that public sector workers are more likely than the rest of the public to agree to take part in surveys if approached. Consultation with our international Ipsos colleagues reveals that a similar phenomenon is already familiar to pollsters in other European countries.

Therefore we will be adding a weighting for public sector workers to future Omnibus surveys on which our political polls run, and to our ad hoc political surveys. The weighting target is taken from the latest Economic & Labour Market Review published by the Office of National Statistics.

While it is too early to say whether this change will have a consistent effect on voting intention measurements, in the June 2008 Political Monitor survey the effect of adding the new weight was to increase Conservative share by one percentage point and to reduce Labour share by one point when measured on the basis of those who say they are certain to vote. (This adjustment cannot be applied to our past polls as the public/private sector question has not regularly been included in our previous surveys.)

Examination of other demographic and contextual data from our recent national political surveys has revealed no other discrepancies when compared with reliable independent sources. We are confident that with the addition of a public sector weight, our telephone surveys will be reporting on a robust and representative sample. Nevertheless, we will continue to conduct further experiments and to search for other questions which could be added to our surveys to produce verifiable measurements against independent data sources. Should we discover any other anomalies we will, of course, take steps to correct them.

Similarly, our analysis of the London election poll data found no significant divergences among the sample from the known population profile. (Such analysis is, however, restricted by the inevitably small number of demographic questions that were included in the polls.) The data was also analysed in relation to characteristics of the interviewers, to eliminate the possibility that interviewer effects might have distorted the findings. None of the available interviewer characteristics proved to be significantly related to declared voting intentions.

2. Voting intention measurements by telephone

(a) Overview

Making accurate election predictions from voting intention poll data involves not only ensuring that the sample is representative, but also interpreting the responses and converting them to a projection of behaviour. This can involve different challenges in different circumstances, but we believe the most pressing one in Britain at the moment to be distinguishing those respondents who will vote from those who will not.

But predictions can only be made in relation to specific elections, and normally only from eve-of-poll surveys. Between elections, we can measure voting intentions – a measurement of what the public think they would do – but because the election in question is a hypothetical one there can be no ‘right’ answer as to what adjustments, if any, should be made to allow for turnout.

Our practice in recent years has been to publish two trend figures – the voting intentions of the whole public, and (to provide a figure that is more nearly comparable with the calculations that would be made in an eve-of-poll prediction) the voting intentions of those who say they would be absolutely certain to vote in an immediate general election. Neither of these trend figures should be interpreted as a prediction of the result uuof a future election, as they collect a ‘snapshot’ view from a given moment in time, and cannot in any way predict future outcomes.
 
We have concluded that our methodology for measuring voting intentions is fundamentally sound, but that we should revise our presentation of the data to make the meaning and limitations of mid-term political polling clearer. We may, however, introduce extra considerations into our calculation of eve-of-poll predictions at future elections; many factors which influence turnout at elections (such as being canvassed) cannot be polled on during mid-term polls – see section (c) below for more details.

We continue to see no case for weighting our data by reported past vote. Respondents' reports of their votes are highly inaccurate and can change in line with their political opinions. This means that all weighting targets are necessarily based on subjective judgement of the responses that would be likely to be obtained from a representative sample, rather than the objective facts that we prefer to use in weighting. Weighting our polls by reported past vote at the 2005 General Election would have made them less, not more, accurate.

(b) London 2008

Reviewing our telephone polls during the run-up to the London mayoral elections, it is possible that our samples were slightly distorted in favour of Ken Livingstone by an over-representation of public sector workers in the sample (a problem that we have found has been affecting our national polls, see above), but this cannot be confirmed as no question on industry sector was included in the original polls.

However, we think the main cause of the difference between voting intentions measured during the campaign and the final outcome was differential turnout. It is plain that our certainty of voting question did not prove accurate in identifying those who would eventually turn out, since it would have considerably overstated the turnout if taken literally. However, it is not clear whether it would have exaggerated turnout of all parties equally, in which case it would have given a true picture of the strength of the candidates at the time the poll was taken. Media reporting during the final week of the election (after our last poll was conducted) gave a firm impression that the Conservative campaigners were more effective at contacting and getting out their vote than their Labour opponents. This was of course not a factor that could have been measured in advance in our polling. (We are still completing analysis of a recall survey of respondents to the last of our polls during the campaign, and it is possible that this might provide further evidence one way or the other, but vote recall polls are usually not too reliable and hence any evidence offered is unlikely to be conclusive.)

(c) General election 2005

Voters behave differently in local and general elections, and the relationship between turnout and certainty of voting as declared in a poll is probably not the same; certainly, our polls over many years have found a much higher exaggeration of claimed turnout at local elections than at general elections, both in pre-election polls and in reported voting behaviour post-election. Therefore we cannot assume that methods suited to predicting turnout in the London Mayoral election, should we be able to develop any, could be applied directly to general election polling. Our final polls during the last general election, using the same certainty of voting filter as we use now, were highly accurate.

Nevertheless, one of the lessons of the London Mayoral election would certainly seem to be applicable. Analysis of the academic British Election Study (BES) from the 2005 general election, which has the advantage of having had the turnout of its individual respondents validated against the marked-up electoral register, also shows that whether or not a respondent had been contacted by party campaigners during the election run-up was one of the best predictors of whether or not he or she would vote, and taken in conjunction with certainty of voting was a more effective predictor than certainty of voting alone.

However, the BES data also confirms that the single best predictor of turnout (though by no means perfect) is a ten-point certainty of voting question, and that other attitudinal indicators (such as perceived importance of the election, difference between the parties or belief in a civic duty to vote) are less effective independent discriminators of turnout than certainty of voting, reported contact by campaigners and reported past turnout. These sometimes however help to ‘fine tune’ predictions of voters’ behaviour.

(d) Consequently:

(i)    We will continue to use some of the questions detailed above in the calculation of our final poll prediction at future general elections (and will report the results, and their likely implications for translating voting intentions into voting behaviour, in our earlier polls during the campaign if this seems appropriate).
(ii)    We will continue to report voting intentions in our “peacetime” polls calculated on two bases, those absolutely certain to vote and all those giving a voting intention. We will consider adding trends showing voting intention calculated on other bases (and are perfectly happy to accept requests within reason to calculate it for any given poll on a different basis), but will make no change for the moment.
(iii)    We will continue to experiment with the effects of adding additional behavioural and attitudinal questions into our turnout filter, and where appropriate to report the results. This will inform our preparations for the next general election and increase our knowledge of the interaction between various factors in current political thought. Should we identify a survey question or questions which appear sufficiently useful to justify our adding them to the political surveys on a regular basis, we will also consider whether they offer a worthwhile alternative means of calculating voting intention, in which case we would add a new trend series to the two existing ones.
(iv)    We will take steps in the presentation of our poll data that attempts to emphasise the distinction between “snapshot” polls and predictions, and to warn against misinterpretation of our figures.

3. Voting intention measurements on the face-to-face Public Affairs Monitor omnibus

There has been a consistent discrepancy over the last few months between the voting intention figures produced by our face-to-face surveys (published as the Ipsos MORI Political Monitor) and our telephone omnibus (mostly unpublished). The data has tended to agree on Conservative share, but the face-to-face surveys have tended to give Labour a higher voting intention share at the expense of the Liberal Democrats or minor parties. The telephone data has tended to be closely in line with that of the published telephone polls by other pollsters.

Since the telephone voting intention findings seem more robust and more closely reflective of recent election results, we have decided to transfer our monthly voting intention polling to the telephone for the moment. To give context to the data and provide comparable trend data by which the telephone voting intention figures can be judged, we are now releasing the results of a number of previously unpublished monthly surveys conducted since the autumn of last year. We will also transfer the satisfaction ratings polling to the telephone surveys, since the cross analysis of satisfaction by voting intention is a useful measure.

It is important to note that most other non-voting questions and measures (including demographic characteristics) are consistent between our telephone and face-to-face findings, which is why we feel comfortable moving the Political Monitor onto the telephone and also why we will continue to do robust social issues data collection on our face-to-face omnibus.

Our investigations have not yet provided an explanation for the current differences in findings, although a number of possibilities have been ruled out. It does not appear to be caused by the difference between using showcards in the face-to-face polls and a read-out voting intention question on the telephone. Nor is it apparently an effect of sampling error in the selection of sampling points for the face-to-face survey. It is not caused by the ordering of the ‘certainty of voting’ and voting intentions questions (which comes first). It not caused by any differences between the voting intentions of all adults and of that section of them who live in a household with a landline telephone.

The sampling for the two types of polls is based upon the same principles although it necessarily varies in the details; otherwise apart from the method of the interview itself the methodology used in the two methods is identical. In the past we have found that the two methods produce comparable results, and we consider that polling by two different methods offers an important safeguard against the possibility of some undetected new bias affecting the measurement of voting intentions.

The “most important issues facing the country” question, however, will remain on the face-to-face survey. Open-ended questions requiring multiple answers can be particularly vulnerable to mode effects, and transferring this question to the telephone survey would disrupt the long-term trends to no useful purpose. This survey will in future be known as the Ipsos MORI Issues Index; it will continue to be published monthly, on the same timetable as previously, and the data will be made publicly available.

Ipsos MORI Contacts

Julia Clark,
Head of Political Research
Email

Dr Roger Mortimore
Email