How Sample Surveys Go Wrong
HOW SAMPLE SURVEYS GO WRONG
Random sampling eliminates bias in choosing a sample and allows control of variability. So once we see the magic words “randomly selected” and “margin of error,” do we know we have trustworthy information before us? It certainly beats voluntary response, but not always by as much as we might hope. Sampling in the real world is more complex and less reliable than choosing a Simple Random Sample (SRS) from a list of names in an exercise. Confidence statements do not reflect all of the sources of error that are present in practical sampling. Most sample surveys are afflicted by errors other than random sampling errors. These errors can introduce bias that makes a confidence statement meaningless. Good sampling technique includes the art of reducing all sources of error. Part of this art is the science of statistics, with its random samples and confidence statements. In practice, however, good statistics isn’t all there is to good sampling. Let’s look at sources of errors in sample surveys and at how samplers combat them.
• Sampling errors
Random sampling error is one kind of sampling error. The margin of error tells us how serious random sampling error is, and we can control it by choosing the size of our random sample. Another source of sampling error is the use of bad sampling methods, such as voluntary response. We can avoid bad methods. Other sampling errors are not so easy to handle. Sampling begins with a list of individuals from which we will draw our sample. This list is called the sampling frame. Ideally, the sampling frame should list every individual in the population. Because a list of the entire population is rarely available, most samples suffer from some degree of under-coverage. If the sampling frame leaves out certain classes of people, even random samples from that frame will be biased. Using telephone directories as the frame for a telephone survey, for example, would miss everyone with an unlisted telephone
Random sampling eliminates bias in choosing a sample and allows control of variability. So once we see the magic words “randomly selected” and “margin of error,” do we know we have trustworthy information before us? It certainly beats voluntary response, but not always by as much as we might hope. Sampling in the real world is more complex and less reliable than choosing a Simple Random Sample (SRS) from a list of names in an exercise. Confidence statements do not reflect all of the sources of error that are present in practical sampling. Most sample surveys are afflicted by errors other than random sampling errors. These errors can introduce bias that makes a confidence statement meaningless. Good sampling technique includes the art of reducing all sources of error. Part of this art is the science of statistics, with its random samples and confidence statements. In practice, however, good statistics isn’t all there is to good sampling. Let’s look at sources of errors in sample surveys and at how samplers combat them.
• Sampling errors
Random sampling error is one kind of sampling error. The margin of error tells us how serious random sampling error is, and we can control it by choosing the size of our random sample. Another source of sampling error is the use of bad sampling methods, such as voluntary response. We can avoid bad methods. Other sampling errors are not so easy to handle. Sampling begins with a list of individuals from which we will draw our sample. This list is called the sampling frame. Ideally, the sampling frame should list every individual in the population. Because a list of the entire population is rarely available, most samples suffer from some degree of under-coverage. If the sampling frame leaves out certain classes of people, even random samples from that frame will be biased. Using telephone directories as the frame for a telephone survey, for example, would miss everyone with an unlisted telephone