Dr Henry Durant, the man who introduced opinion polling to Britain, once described it as "the stupidest of professions" - for who else is stupid enough to publish a prediction on Thursday morning that may be proved wrong on Thursday evening?
Durant ran the first polls for British Gallup (originally the British Institute of Public Opinion) in 1937, just two years after its American forefather was founded. Gallup began in both the USA and in Britain as a means of journalism, but it quickly expanded its horizons. Durant explained:
People constantly asked us to put questions on our regular surveys, and at the beginning I was stupid enough to regard these as a nuisance: then I suddenly realised that this was a beautiful way of making money. It grew and soon had its own omnibus survey: today it's one of the things that researchers live off.
Even in the early days the questions covered a wide-ranging subject matter, but the directly political questions quickly won their place. By October 1938, they had begun testing the public's satisfaction with the Prime Minister (57% were satisfied with Neville Chamberlain, 43% dissatisfied), and the first national voting intention question was introduced in February 1939 (when 64% answered that they would vote for the Government "if there were a general election tomorrow"). Meanwhile, just as George Gallup and his rivals had established their position in the USA by successful prediction of the 1936 Presidential election, so Durant's British Gallup polls moved into election polling, at the behest of the News Chronicle, and secured a firm foothold by correctly challenging the received wisdom. An early triumph was to forecast Edith Summerskill's victory at the West Fulham by-election in 1938. Even more unexpected, of course, was Churchill's defeat at the 1945 General Election.
Gallup showed us that Attlee was going to win with the Labour Party. Nobody believed us, including all the News Chronicle people.
But in the event Gallup was right, in fact slightly underestimating Labour's lead. Despite the occasional hiccup, the polls generally since then have a good record of election 'prediction': only twice in 15 elections has the average error of the "poll of polls" in measuring the parties' vote shares exceeded 2%, or the error on the Conservative-Labour lead exceeded 6%.
In 1945, Gallup was the only player in the field, but rivals soon began to move in, and many - ICM, MORI, NOP and Harris - are now household names. All produce regular polls commissioned by and published in the media. Scotland, Wales and Northern Ireland, too, have become used to more or less regular polling for their own media, both by the London-based pollsters and by smaller local companies equally well-known within their own markets.
Meanwhile, the political parties finally woke up to the potential of polling for their own purposes (finding out in private what they could do to win the next election instead of reading in the newspaper how certain it was they were going to lose it). Both main parties were commissioning research by the end of the fifties, and all the main polling firms have worked for one or other at some period since then - or, indeed, for both, though not of course at the same time! Academia, too, began to use polling techniques to generate data for political research, especially the British Election Study under Butler and Stokes and their successors.
One technique which has remained mainly confined to private polling, but which over the last few years has come to the fore, is the use of focus groups. This is not because they mark a sea change with the methods of the past: focus groups have a long history in social and market research. What has moved them into the headlines is the use of focus groups by Tony Blair to explore voter attitudes and test key messages in the period leading up to the 1997 election - something which has continued in Government. In The Unfinished Revolution, Blair's polling advisor, Philip Gould, talks about the use of focus groups.
I nearly always learn something new and surprising. People do not think in traditional ways or conform to conventional prejudice. In a group it is possible to test out the strength and depth of feeling about an issue, which can be more difficult, although not impossible, in a conventional poll
On the other hand:
Not all groups work as they should. On one occasion, two strangers seduced each other as the group was going on. Everyone pretended not to notice. (Philip Gould: The Unfinished Revolution)
Most published polling, though, remains based on the quantitative survey. Gallup's first polls were postal surveys, but these were quickly supplanted by the familiar pen and clipboard for face-to-face interviews. Telephone polling came later - indeed, it was not until 1997 that the majority of the main monthly newspaper polling series switched to telephone interviews - and the nineties also saw the increasing replacement of the clipboard with Computer Assisted Personal Interviewing (CAPI) for face-to-face interviews. At the same time, unfortunately, has come the explosion of the meaningless but ubiquitous self-selecting polls, at first by mail-in coupons, then newspaper phone-in polls (far too easily and too frequently misreported as "a telephone poll"), and now increasingly similar polls on the Internet, often given spurious credibility by association with otherwise reputable sites or publications.
But the misreporting or simplification of our data is a cross we always have to bear. Some simplifications seem to catch the imagination of the press, and stick - for example, as ideotypes of the key swing voters, "Essex Man", "Mondeo Man" or "Worcester woman". Press reports this year have brought us "Swindon man" - apparently some market researchers have come to the conclusion that Swindon is ideally representative of Britain as a whole, and have descended upon this almost blameless town in droves. The story is all too reminiscent of "Magic Town", surely the only Hollywood feature film about political research methodology: James Stewart, the pollster-hero, discovers a town that is the perfect microcosm of the USA, bases all his polls there (with consequent savings in time and cost), and eventually learns a harsh lesson about panel-conditioning effects.
But in the end, it is a simple business, really - all we have to do is ask the right questions, to the right people, and add up the figures correctly; and if we can get all the figures within 1%, as Henry Durant did in that first poll at West Fulham, we may even get some credit for it. But even Dr Durant called that "beginner's luck".