IJE Advance Access originally published online on October 26, 2006
International Journal of Epidemiology 2006 35(6):1589-1590; doi:10.1093/ije/dyl227
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Letters to the Editor |
Reply to Roger Marshall
1 Department of Epidemiology, University of California, Los Angeles, CA, USA
2 Department of Statistics, University of California, Los Angeles, CA, USA
3 Department of International Health, Boston University School of Public Health, Boston, MA, USA
4 Department of Epidemiology, Boston University School of Public Health, Boston, MA, USA
5 Geriatrics Section, Boston Medical Center, Boston, MA, USA
6 Department of Medicine, Boston University School of Medicine, Boston, MA, USA
* Corresponding author. E-mail: mfox{at}bu.edu
While appreciating Dr Marshall's comments1 on our manuscript, we are concerned that his approach understates the potential distortions engendered by misclassification. In particular, while we see no flaw in Dr Marshall's algebra, we disagree strongly with his conclusion that
adopting scenario situations based on specified sensitivity and specificity values can be misleading and pessimistic; specificity and sensitivity are not absolutes, they are restricted in their range of possible values, depending on the measured exposure prevalence. The key to considering whether bias is present is not in terms of differentiality of specificity and sensitivity but in differentiality of their chance-corrected counterparts
Sensitivity and specificity are indeed not absolutes, which is one reason we treated them as random variables.2 Nonetheless, Dr Marshall's model1,3 and conclusion focus on algebraic rather than contextual (substantive) properties. We believe this focus is potentially misleading and leads to overly optimistic conclusions. In particular, we think he has overlooked optimistic biases in his own literature analyses.4
To say that a classification method is nondifferential with respect to disease means it has identical operating characteristics among cases and controls, so that sensitivity and specificity do not vary with disease status. We expect this property to hold when the mechanisms that determine the classification are identical among cases and controls. In particular, we expect nondifferentiality when the disease is unrelated to exposure measurement. This expectation is reasonable when the mechanisms that determine the classification precede the disease occurrence and are not affected by uncontrolled risk factors. Thus to say there is no recall bias (as when exposure data are collected from records that predate the outcome) means that neither disease nor uncontrolled risk factors prompt more false-positive responses from cases than from controls. Put more abstractly, nondifferentiality means that the classification X is independent of the outcome Y (i.e. Y conveys no information about X) conditional on the true exposure E and whatever has been controlled, not that some ‘chance-corrected’ measure is the same in cases and controls. This condition may be rarely met, but at least it can be criticized based on qualitative mechanistic considerations.
It is intuitions and judgments about the role of the outcome Y in exposure classification errors that are the basis for priors about measurement behaviour. These judgements provide one reason to express such priors in terms of sensitivity and specificity, despite the greater clumsiness of algebra and statistics when compared to expressions based on predictive values5 or on ‘chance-corrected’ measures like Q1 and Q0. Furthermore, statisticians have traditionally focused on sensitivity and specificity because, when combined with other assumptions (such as independence of errors across variables), their nondifferentiality can preserve validity of the test of the null,6 although this result is often over-interpreted.7
With regard to Dr Marshall's empirical meta-study of bias,4 we think he has over-generalized limited findings from a highly select group of 16 reports found in the original source.8 First, we should expect that the very presence of a validation substudy (required for inclusion in the meta-study) would mark the study as atypical, e.g. conducted with greater care and in a population amenable to such a substudy. Second, studies of exposures in which a near-gold standard may be available (e.g. prescription drugs and medical treatments), which included most of the studies, are no guide to general practice, since the precise nature of the exposure involved facilitates accurate recall. Third, ‘validation’ studies of other measurements compare two flawed measures and are therefore really studies of agreement. The compared measures often have correlated error components, and will share any bias in selection for the ‘validation’ substudy. We thus expect such studies to overstate validity, even when analysed as agreement studies (rather than as validation studies), and find it unsurprising that the results from such studies appear optimistic. This concern is especially worrisome for exposures for which genuine validation data have never been and probably never will be available, such as lifetime histories of nutrient intake, exercise, and environmental and occupational exposures.
Regardless, to claim that the relation of Q1 and Q0 to bias should lead us to express nondifferentiality in terms of Q1 and Q0 is almost circular: independence of Q1 and Q0 from disease status Y directly implies no bias. The essential scientific question is as follows: what are the substantive bases for such an assumption? For example, what exposure-assessment mechanisms (as opposed to algebraic relations) should lead us to expect this independence condition to hold? One might base an analysis using the studies in Marshall4 as a prior for Q1 and Q0, but this usage could be quite misleading, given the limitations of the studies and that both Q1 and Q0 depend on prevalence of X, which is bound to vary across study populations, and even across strata within a study. The mistake here is similar to naively applying predictive values from one population to another.
In summary, we doubt that Marshall's theoretical and empirical observations provide valid guidance to the impact of unknown classification error in typical studies, especially in lifestyle, environmental, and occupational epidemiology. When no genuine validation data are available, sensitivity and specificity may often be the most natural expressions on which to base prior expectations, including expectations about the impact of the outcome on classification errors. When building a prior for errors (such as illness effects on memory), the exposure prevalence in Q1 and Q0 or in predictive values renders these measures less generalizable and removes them further from the mechanisms affecting those errors.
References
1 Marshall RJ. Letter Re: ‘Fox MP, Lash TL, Greenland S. A method to automate probabilistic sensitivity analyses of misclassified binary variables.’ Int J Epidemiol 35:1589–90.
2 Fox MP, Lash TL, Greenland S. A method to automate probabilistic sensitivity analyses of misclassified binary variables. Int J Epidemiol 2005;34:1370–6.
3 Marshall RJ. Assessment of exposure misclassification bias in case-control studies using validation data. J Clin Epidemiol 1997;50:15–19.[CrossRef][Web of Science][Medline]
4 Marshall, RJ. An empirical investigation of exposure measurement bias in case-control studies. J Clin Epidemiol 1999;52:547–50.[CrossRef][Web of Science][Medline]
5 Marshall RJ. Validation study methods for estimating exposure proportions and odds ratios with misclassified data. J Clin Epidemiol 1990;43:941–7.[CrossRef][Web of Science][Medline]
6 Bross IDJ. Misclassification in 2 x 2 tables. Biometrics 1954;10:478–86.[CrossRef]
7 Greenland S, Gustafson P. Adjustment for independent nondifferential misclassification does not increase certainty that an observed association is in the correct direction. Am J Epidemiol 2006;164:63–8.
8 Chouinard E, Walter S. Recall bias in case-control studies: an empirical analysis and theoretical framework. J Clin Epidemiol 1995;48:245–54.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  A. M Jurek, S. Greenland, and G. Maldonado Brief Report: How far from non-differential does exposure or disease misclassification have to be to bias measures of association away from the null? Int. J. Epidemiol., April 1, 2008; 37(2): 382 - 385. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||