IJE Advance Access originally published online on September 13, 2007
International Journal of Epidemiology 2007 36(5):966-968; doi:10.1093/ije/dym181
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Cochrane Column
South African Cochrane Centre, Medical Research Council, PO Box 19070, Tygerberg 7505, South Africa. E-mail: taryn.young{at}mrc.ac.za
This month, we feature the Cochrane Review assessing the effect of strategies to increase the response to postal questionnaires. Cochrane methodology reviews are published in the Cochrane Database of Systematic Review alongside the more usual Cochrane Reviews of the effects of healthcare interventions. They are produced by the Cochrane Methodology Review Group. Cochrane methodology reviews have a similar structure to the healthcare reviews, but with a few subtle changes to their section headings, to reflect the fact that they cover studies assessing the methodology of research in health and social care, rather than the care itself. They are preceded by published protocols and produced with the same rigour and attention to detail as Cochrane Reviews of healthcare. For example, evidence from methodological research is included or excluded on the basis of explicit criteria. Each review covers a specific and well-defined area of methodology and data from included studies might be combined statistically to increase the power of the findings. In such cases, the review may include graphs or tables presenting the data from each individual study, along with the overall average.
If you are interested in contributing to the Cochrane Column or The Cochrane Collaboration, contact me at the South African Cochrane Centre.
Increasing response to postal questionnaires
 |
Background |
|---|
 Top Background MethodologyResultsConclusionsReferencesReferences |
|---|
Questionnaires are the ‘mainstay of epidemiologic data collection’.1 They are often the most efficient data collection method because they can be self-administered in literate populations; and if there is a reliable postal system they can be mailed to large, geographically dispersed populations.2 However, if people in the study sample do not respond to the questionnaire, the research results will be less precise and may be biased. Because non-response can affect the validity of epidemiological studies, the assessment of the proportion that responds is an important dimension in the critical appraisal of health research. For the same reason, the identification of effective strategies to increase response to postal questionnaires could improve the quality of health research. This review sought to identify such strategies.
|
Methodology |
|---|
|
TopBackgroundMethodologyResultsConclusionsReferencesReferences |
|---|
Search strategy
We aimed to identify and include all randomized controlled trials of any strategy designed to increase the response to a postal questionnaire. We searched 14 electronic databases to February 2003 (including Cochrane Controlled Trials Register, ERIC, PsycLit, MEDLINE and EconLit), the reference lists of relevant trials, and all issues of American Journal of Epidemiology and Public Opinion Quarterly. We contacted the authors of all trials to enquire about any unpublished trials, and, where necessary, to ask for clarification of results presented and methods of allocation used. Two authors assessed the eligibility of each trial using pre-defined criteria.
Data extraction and analysis
For each trial identified, we extracted data on the participants, intervention, numbers randomized to intervention and comparison groups and method of allocation concealment. For each strategy, we estimated pooled odds ratios and 95% confidence intervals in a random effects model. Heterogeneity in trial results was assessed using a chi-squared test and the degree of inconsistency between trial results was quantified using I2.
|
Results |
|---|
|
TopBackgroundMethodologyResultsConclusionsReferencesReferences |
|---|
We found 372 eligible trials evaluating a total of 98 different strategies to increase response to postal questionnaires. There was substantial heterogeneity among the results of trials of many strategies. A sample of the strategies found to be effective is shown in Table 1. Analyses to investigate heterogeneity in results of trials for some of the strategies have been completed.3–5
|
|
Conclusions |
|---|
|
TopBackgroundMethodologyResultsConclusionsReferencesReferences |
|---|
Researchers can increase response to postal questionnaires by using strategies shown to be effective in this review. For example: monetary incentives could be offered with questionnaires; participants could be contacted in advance; letters and questionnaires could be made more personal; a stamped-return envelope could be included; follow-up contact could be made with non-respondents and reminders could include a second copy of the questionnaire. Some of the strategies require additional materials and administrative time, however, others can be implemented at no extra cost.
The review was first published in The Cochrane Library in 2001 and was updated in 2003. It will be updated again in 2008.
The full text of the Cochrane Review is available in The Cochrane Library: Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R, Kwan I, Cooper R. Methods to increase response rates to postal questionnaires. The Cochrane Database of Methodology Reviews 2007, Issue 2. Chichester, UK: John Wiley & Sons, Ltd. DOI: 10.1002/14651858.MR000008.pub2.
|
References |
|---|
|
TopBackgroundMethodologyResultsConclusionsReferencesReferences |
|---|
1 Hartge P, Cahill J. Field methods in epidemiology. In: Modern Epidemiology—Rothman KJ, Greenland S, eds. (1998) 2nd. Philadelphia: Lippincott-Raven.
2 Armstrong BK, White E, Saracci R. Principles of Exposure Measurement in Epidemiology. Monographs in Epidemiology and Biostatistics (1995) 21. New York, NY: Oxford University Press.
3 Edwards P, Roberts I, Sandercock P, Frost C. Follow-up by mail in clinical trials: does questionnaire length matter? Contr Clin Trials (2004) 25:31–52.[CrossRef]
4 Edwards P, Cooper R, Roberts I, Frost C. Meta-analysis of randomised trials of monetary incentives and response to mailed questionnaires. J Epidemiol Community Health (2005) 59:987–99.
5 Scott P, Edwards P. Personally addressed hand-signed letters increase questionnaire response: a meta-analysis of randomised controlled trials. BMC Health Services Res (2006) 6:111.[CrossRef]
Commentary: Methods to increase response rates to postal questionnaires
1Post-graduate fellow, Faculty of Medicine, University of Toronto, Toronto, Ontario.
2Senior science officer, Jamieson laboratories, Windsor, Ontario.
The use of postal questionnaires is undoubtedly an important data gathering tool for many types of health-care research. In particular, epidemiological literature, especially large scale studies, rely almost exclusively on postal questionnaires. The main problem with postal questionnaires is the poor response rates that can introduce systematic biases in the data and thus skew the findings causing poor external validity. Edwards et al. address this problem by systematically reviewing the methodological literature that tests various techniques at improving response rates to postal questionnaires. This study included 372 trials evaluating 98 different ways of increasing response rates to postal questionnaires. This is a large task, including many hundreds of studies and the authors should be commended for undertaking the project.
What did they find? The odds of getting a response to a postal questionnaire were at least doubled when using monetary incentives, recorded delivery, a teaser on the envelope and a more interesting questionnaire topic. Also, the odds of response were substantially higher with pre-notification, follow-up contact, unconditional incentives, shorter questionnaires, providing a second copy of the questionnaire at follow-up, mentioning an obligation to respond and university sponsorship. The odds of response were also increased with non-monetary incentives, personalized questionnaires, use of coloured as opposed to blue or black ink, use of stamped return envelopes as opposed to franked return envelopes, an assurance of confidentiality and first class outward mailing. Finally, the odds of response were reduced when the questionnaire included questions of a sensitive nature, when questionnaires began with the most general questions, or when participants were offered the opportunity to opt out of the study. This study reports 98 separate analyses that can be referred to for detailed numerical findings for individual strategies. As would be expected, there was substantial heterogeneity among trials for many of the strategies.
This study is important for several reasons. First, it is the first systematic and comprehensive meta-analysis of all available studies examining this question. Second, it uses strong methodology and provides specific findings. Third, the results of this article can inform study design and thus improve response rates allowing for improved validity. This latter point deserves careful elaboration. Since postal questionnaires are a primary method of data acquisition in many research designs, strengthening response rates immediately improves the methods of the study. Improving methods strengthens the validity of the findings and the implications of those findings. Many large-scale epidemiological studies have had immediate, important impacts on public health measures or clinical practice. Also, epidemiological studies often serve as hypotheses generation tools that are then tested in randomized controlled trials (RCTs). The better the data acquisition, the stronger the potential for public health impact and the more they lend to RCTs. This meta-analysis reports findings that can improve the response rates to postal questionnaires. Epidemiologists and clinical trialists would be wise to place this article among their resources for designing and implementing studies that employ postal questionnaires.
Commentary: Methods to increase response rates to postal questionnaires
Institute of Health and Society, Newcastle University.
Postal questionnaires are commonly used for data collection in epidemiological research. Low response rates are a matter of concern. The precision of estimates is inversely proportionate to sample size. Non-response is a significant source of bias, since those who do not respond typically differ from those who do with respect to demographic and other characteristics. Edwards and colleagues (2007)1 report on strategies to increase postal questionnaire response rates.
The focus of their review, conducted to Cochrane standards, comprised randomized controlled trials, published up to 2003, of any strategy to improve postal questionnaire response rates. Three hundred and seventy-two eligible trials, evaluating 98 different strategies, were identified. The strategies could be conveniently subdivided into: length, appearance and content of questionnaire; mode of delivery (e.g. franked versus stamped and postage class); number and timing of contacts; incentives; origin of questionnaire and style and content of communication (e.g. assurances of confidentiality and portrayal of benefits of response).
The most effective strategies were: pre-notifying recipients of the impending arrival of the questionnaire; sending questionnaires by first-class mail or recorded delivery; provision of a stamped return envelope; personalization of questionnaires and letters; short questionnaires; provision of an incentive, preferably monetary and the use of one or more reminders, including a duplicate questionnaire to initial non-respondents.
The quality of the review is high, and the recommendations for practice are clear, explicit and, in principle at least, capable of being implemented in health surveys. However, a number of points are worthy of further consideration. Edwards and colleagues do not analyse findings from health-related and other studies separately. For some strategies, all reviewed studies relate to general populations and/or non-health-related topics; generalizability to health-related surveys may be called into question. There is some evidence (cited by McColl et al., 2001)2 that health-related surveys are intrinsically more interesting to respondents; strategies to enhance response rates that are desirable in other settings may not always be required. Second, some strategies––in particular the provision of monetary incentives––may be unacceptable to research funders or ethics committees, for increasing the cost of the research or exerting undue pressure on individuals to participate in what should be a voluntary activity. Finally, and reflecting the lack of an economic evaluation of strategies in the studies they reviewed, Edwards and colleagues do not consider the cost-effectiveness of the various strategies. When the marginal costs and marginal benefits of some approaches, particularly those that are more resource intensive, are weighed up, an apparently effective strategy may no longer be so appealing.
|
References |
|---|
|
TopBackgroundMethodologyResultsConclusionsReferencesReferences |
|---|
1 Edwards P, Roberts I, Clarke M, et al. Methods to increase response rates to postal questionnaires. In: Cochrane Database of Systematic Reviews (2007) Art. No.: MR000008. DOI: 10.1002/14651858.MR000008.pub3.
2 McColl E, Jacoby A, Thomas L, et al. Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health Technology Assessment (2001) 5:i–256.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  M. Osler, M. Juhl, R. Lund, A.-M. N. Andersen, M. Kriegbaum, and U. Christensen Effect on response rates of stamps vs a pre-printed business reply envelope in a mailed follow-up survey among middle-aged Danish men Int. J. Epidemiol., August 1, 2009; 38(4): 1156 - 1157. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||