How Can a Poll of Only a Thousand Americans Represent Millions of People?

Andrew Gelman, a professor in the departments of statistics and political science at Columbia University, explains:

A "3 percent margin of error" means that there is a 95 percent chance that the survey result will be within 3 percent of the population value. To put it another way, you would expect to see a less than 3 percent difference between the proportion of people who say "yes" to the survey question and the proportion of people in the population who would say "yes" if asked.

How is it that a survey of only 1,000 people can reach this level of accuracy? You must first assume that the survey respondents have been sampled at random from the population, meaning that people are selected one at a time, with all persons in the U.S. being equally likely to be picked at each point. For most polls, this is approximated by calling phone numbers generated randomly by computer.

The margin of error depends inversely on the square root of the sample size. That is, a sample of 250 will give you a 6 percent margin of error and a sample size of 100 will give you a 10 percent margin of error. A 10 percent margin of error is not so useful. It would give you vague claims such as, "The proportion of Americans who support the death penalty is somewhere between 60 percent and 80 percent." Pollsters thus spend the money to get a reasonably large sample. In the other direction, by surveying 4,000 people, you can get the margin of error down to 1.5 percent. This sounds appealingly precise (for example, "The proportion is between 68.5 percent and 71.5 percent"), but it is generally a waste of time because public opinion varies enough from day to day that it is meaningless to attempt too precise an estimate. Indeed, to do so would be like getting on a scale in the morning and measuring your weight as 173.26 pounds.

The margin of error is a mathematical abstraction, and there are a number of reasons why actual errors in surveys are larger. Even with random sampling, people in the population have unequal probabilities of inclusion in the survey. For instance, if you don't have a telephone, you won't be in the survey, but if you have two phone lines, you have two chances to be included. In addition, women, whites, older people and college-educated people are more likely to participate in surveys. Polling organizations correct for these nonresponse biases by adjusting the sample to match the population, but such adjustments can never be perfect because they only correct for known biases. For example, "surly people" are less likely to respond to a survey, but we don't know how many surly people are in the population or how this would bias polling results.

Finally, the 3 percent margin of error is an understatement because opinions change. On January 3, 2004, the Gallup poll included 410 Democrats, 26 percent of whom supported Howard Dean for president. The margin of error was 5 percent, and so we can be pretty sure that on that date, between 21 percent and 31 percent of Democrats supported Dean. But a lot of them have changed their minds. A poll is a snapshot, not a forecast.

--Originally published: Scientific American Online, March 15, 2004