IJE Advance Access originally published online on January 13, 2005
International Journal of Epidemiology 2005 34(1):87-88; doi:10.1093/ije/dyh406
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
IJE vol.34 no.1 © International Epidemiological Association 2005; all rights reserved.
Commentary |
Commentary: Empirical evidence of attrition bias in clinical trials
1 Department of Social and Preventive Medicine, University of Berne, Finkenhubelweg 11, CH-3012 Bern, Switzerland
2 MRC Health Services Research Collaboration, Department of Social Medicine, University of Bristol, Canynge Hall, Whiteladies Road, Bristol BS8 2PR, UK
* Corresponding author. Department of Social and Preventive Medicine, University of Berne, Finkenhubelweg 11, CH-3012 Bern, Switzerland. E-mail: egger{at}ispm.unibe.ch
Excluding the 5 patients who died and the 10 who had permanent postoperative deficits, there remained 79 patients available for follow-up and at risk of subsequent persistent stroke and death.
This is how, in 1970, Fields and colleagues analysed the data from the Joint Study of Extracranial Arterial Occlusion, which had randomly allocated patients with bilateral carotid stenosis to carotid endarterectomy or medical treatment.1 Among the patients who had survived surgery and were ‘available for follow-up’, a 26% reduction in the risk of recurrent transient ischaemic attacks, stroke, or death was observed, compared with patients who had received conventional treatment (P = 0.02). Around the same time Bradford Hill, in the ninth edition of his Principles of Medical Statistics, pointed out that excluding patients after ‘admission to the treated or control group’ may affect the validity of clinical trials and that ‘unless the losses are very few and therefore unimportant, we may inevitably have to keep such patients in the comparison and thus measure the ‘intention to treat’ in a given way, rather than the actual treatment’.2 Indeed, when several years later Sackett and Gent3 re-analysed the study according to this intention to treat principle the results were less convincing: the reduction in the risk was 17% (P = 0.09) (Figure 1).
|
In more recent years, the debate has shifted from anecdotal evidence of bias in single trials to more sophisticated ‘meta-epidemiological’ research, based on many trials and meta-analyses.4 Schulz and colleagues5 pioneered this approach when they assessed the methodological quality of 250 trials from 33 meta-analyses from the Cochrane Pregnancy and Childbirth Database and examined the association between dimensions of trial quality and estimated treatment effects. Compared with trials in which authors reported adequately concealed treatment allocation, failure to prevent foreknowledge of treatment allocation was associated, on average, with an exaggeration of treatment effects by 30–40%. Trials that were not double-blind also yielded larger effects. These bias effects were subsequently confirmed in several other studies.6 There is thus convincing empirical evidence that trials of lower methodological quality that are susceptible to selection bias, performance bias, or detection bias produce, on average, larger treatment effects than trials of higher quality that avoided or minimized such biases.
What about attrition bias? So far three studies examined the influence of this type of bias, which, as the example of the Joint Study of Extracranial Arterial Occlusion shows can be of great importance. Schulz et al. compared trials that reported exclusions with trials that either explicitly reported no exclusions or gave the impression that no exclusions had taken place.5 The other two studies assessed the quality of reporting, rather than methodological quality. Kjaergard et al. compared trials that reported adequately on attrition (independent of whether exclusions occurred) to trials with inadequate reporting.7 Similarly, Balk et al.8 assessed whether dropouts had been recorded (either explicitly or by reporting the number enrolled and the number evaluated). Schulz et al. found little difference in effect estimates whereas the Kjaergard and Balk studies found trends in opposite directions. The methods used to assess attrition were unsatisfactory in all cases, because reports often omit important methodological details.6 In particular, it is problematic to deduce that a trial was analysed according to the intention to treat principle if no exclusions are reported.9 And it is equally problematic to assume that an analysis described as intent to treat did in fact include all patients, according to random allocation.10
The elegant study by Tierney and Stewart11 for the first time directly assesses the impact of attrition bias. Stewart and colleagues have co-ordinated individual patient data meta-analyses of clinical trials in oncology for many years and now present a comparison of the results from their analyses, which invariably followed the intention to treat principle with those done by the original investigators, which often excluded some or many patients. The results confirmed what methodologists had been suspecting for some time: pooled analyses of trials with patient exclusions showed more beneficial effects of the experimental treatment than analyses based on all or most patients randomized. It therefore seems likely that the earlier studies5,7,8 either suffered from measurement error due to incomplete reporting or assessed an inappropriate proxy measure, reporting quality.12 The effects of attrition bias in these meta-analyses were modest in most cases, but, as Tierney and Stewart point out, they may well underestimate the extent of bias in other situations. Investigators who participate in collaborative meta-analyses of cancer trials are probably methodologically more astute and not representative of all investigators in this field, and certainly not representative of trialists in general. Of note, although the bias will in general lead to exaggerated treatment effects, Stewart and Tierney also show that it can go in either direction, and predicting its effect in a specific situation will be difficult.
Individual patient data meta-analyses have been described as the ‘yardstick’ against which other forms of systematic review and meta-analysis should be measured.13 Tierney and Stewart's paper is another example of the power of such analyses, which in a number of cases have produced definitive answers that might not have been obtained in any other way.14 Their approach should be applied more widely to further develop our understanding of the mechanisms that introduce bias in clinical trial research.
 |
References |
|---|
 Top References |
|---|
1 Fields WS, Maslenikov V, Meyer JS, Hass WK, Remington RD, Macdonald M. Joint study of extracranial arterial occlusion. V. Progress report of prognosis following surgery or nonsurgical treatment for transient cerebral ischemic attacks and cervical carotid artery lesions. JAMA 1970; 211:1993–2003.
2 Bradford Hill A. Principles of Medical Statistics, 9th edn. London: The Lancet Limited, 1971.
3 Sackett DL, Gent M. Controversy in counting and attributing events in clinical trials. N Engl J Med 1979; 301:1410–12.[Web of Science][Medline]
4 Sterne JAC, Jüni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Stat Med 2002; 21:1513–24.[CrossRef][Web of Science][Medline]
5 Schulz KF, Chalmers I, Hayes RJ, Altman D. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273:408–12.
6 Jüni P, Altman DG, Egger M. Assessing the quality of controlled clinical trials. BMJ 2001; 323:42–46.
7 Kjaergard LL, Villumsen J, Gluud C. Reported methodological quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001; 135:982–89.
8 Balk EM, Bonis PA, Moskowitz H et al. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA 2002; 287:2973–82.
9 Schulz KF, Grimes DA. Sample size slippages in randomised trials: exclusions and the lost and wayward. Lancet 2002; 359:781–85.[CrossRef][Web of Science][Medline]
10 Hollis S, Campbell F. What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ 1999; 319:670–74.
11 Tierney JF, Stewart LA. Investigating patient exclusion bias in meta-analysis. Int J Epidemiol 2005;34:79–87.
12 Huwiler-Müntener K, Jüni P, Junker C, Egger M. Quality of reporting of randomized trials as a measure of methodological quality. JAMA 2002; 287:2801–04.
13 Chalmers I. The Cochrane Collaboration: preparing, maintaining and disseminating systematic reviews of the effects of health care. Ann NY Acad Sci 1993; 703:156–63.[Web of Science][Medline]
14 Clarke MJ, Stewart LA. Obtaining individual patient data from randomised controlled trials. In: Egger M, Davey Smith G, Altman DG (eds). Systematic Reviews in Health Care: Meta-analysis in Context, 2nd edn. London: BMJ Books, 2000.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  E. Nuesch, S. Trelle, S. Reichenbach, A. W S Rutjes, E. Burgi, M. Scherer, D. G Altman, and P. Juni The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study BMJ, September 7, 2009; 339(sep07_1): b3244 - b3244. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||