Non-response bias in Survey Research

This post is only part of a conversation I have been having with statistician and librarian and gentleman of many hats, Ray Lyon.  We are discussing Total Survey Error. Non-response bias is only one of the pitfalls in poor survey methodology and the biased data some surveys reveal.  I expect more will come on this matter…

A recent development in conversations among members of the survey researcher’s professional organization and governing body, the American Association of Public Opinion Research, has been a renewed discussion concerning non-response bias. While non-response bias has been continually researched and reported on for many years, the renewed vigor in recent weeks is the result of some criticism Pew and other major survey research organizations sustained regarding methodological constraints present in survey research, particularly non-response bias. The problem of accurately representing a given population is felt by all for whom survey research is, in fact, a scientific endeavor. In some cases, careful weighting for demographic characteristics helps to restore the representativeness of data, so that the data can be generalized to the larger population. However, there is the unknown factor that statisticians must deal with. We cannot fully know the characteristics of the sample population who choose not to answer our questions. Scientific rigor is on the line, and it is very much at the forefront of discussion among professional survey researchers.
National survey organizations must sample enough respondents to reach a ‘golden number’ that, then, is often generalized to the total population. Since response rates may be as low as 20-40%, (meaning that many more potential respondents need to be contacted to reach that golden number of actual respondents), 60-80% of the population that was contacted have no representation in the survey results. As researchers, we ask ourselves, what are the responses that we did not get? Would those non-response answers skew our results? Have we truly gotten a clear picture of the population with regard to all our questions? This is non-response bias and it is a serious matter. The best researchers in the country have been researching non-response bias for years and continue to do so. We are very aware of the problem, but there are no easy answers. We know that increased response rate is important to reduce non-response bias, but it does not eliminate the problem of non-response bias unless response rates approach 100%. With a good response rate, we can statistically weight our final dataset to more closely resemble the total population and even get a better representation of the total data. However, there is still some non-response that represents an unknown factor social scientists and statisticians are not comfortable with. We have statistical methodology that is very good at compensating for many things, but there are limitations. A poor response rate can call all the data analysis into question.

, , ,

No comments yet.

Leave a Reply