IJE Advance Access originally published online on November 18, 2005
International Journal of Epidemiology 2005 34(6):1376-1377; doi:10.1093/ije/dyi228
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Commentary |
Commentary: About that measurement problem
Cancer Prevention and Population Sciences, Roswell Park Cancer Institute, Elm and Carlton Streets, Buffalo, NY 14263, USA
E-mail: james.marshall{at}roswellpark.org
The goal of epidemiological inquiry is to estimate the extent to which conditions and exposures affect our risks of the diseases that afflict us; we seek to know how much of the risk of disease might be attributed to a given exposure or condition. Given finite resources and options, we want to identify the most important among malleable risk factors, so that we can focus on those.
This is true of the occupational setting from which Fox et al.1 drew their example: given that we cannot eliminate all hazards to workers, or to those who may be exposed to industrial by-products, we want to know the extent to which a specific risk is increased by the exposure to a given toxin. We seek to know whether other exposures might be less hazardous; we may wish to consider whether the increased risk is more than we in our society are willing to accept. In the case of asbestos, we may wish to weigh the risk associated with asbestos exposure against that generated by exposure to other fire retardants, or against increased risk of immolation due to non-use of asbestos.
An important characteristic of epidemiological exposure data is reliance on records of exposure—records that were in general designed for purposes other than epidemiological precision—or on reports by study subjects. In either case, the precision of the exposure is almost always less than perfect. In dietary epidemiological studies, for example, measures of intake of a nutrient are usually dominated by measurement error rather than by actual exposure. In general in dietary epidemiology, <40% of the variance observed in the intake of any one nutrient reflects actual variance in intake; 60% of the observed variance will be noise.2–5
It is critical that failure to observe an association can result from the inability to accurately measure exposures, even those with massive impacts on risk. It doesn't matter how strong the epidemiological effect; there exists a well-behaved, random error structure that will completely obscure that effect. Building on observations by Bross,6 we have generally understood that the net effect of well-behaved imprecision is benign, resulting in the underestimation of the strength of the exposure–disease association. This may in some circumstances be true, as when only one exposure and one outcome are of concern. It is not true in the case of associations confounded by other associations.7,8
In addition, the assumption that errors are well behaved is a strong assumption. Slightly ill behaviour on the part of measurement errors can, as Fox et al. point out, and as others have noted,9 have massively distorting effects. The only way to avoid this problem is to measure exposures and outcomes without error. One can also measure repeatedly; under a variety of circumstances, repeat measurement will substantially decrease the bias resulting from imperfect measurement.10
Given that measurement without error may be difficult, the interpretation of observed associations becomes problematic. To take the example from Fox et al., an observed odds ratio of 1.8 describing the association of resin exposure and lung cancer means that exposure multiplies the risk of disease 1.80 times, if resin exposure is measured perfectly among both cases and controls, if case status is measured perfectly, and if no other biases intrude.1 The adjustment of the odds ratio proposed by Fox et al. enables deriving the meaning of the observed odds ratio. The observed odds ratio of 1.80 means that exposure multiplies the risk 1.11 times, if the sensitivity and specificity of exposure, respectively, among cases, are 0.9 and 0.8, while the sensitivity and specificity of exposure, respectively, among controls, are 0.8 and 0.9. If, on the other hand, the sensitivity and sensitivity of exposure among cases, respectively, are 0.9 and 0.8, and the sensitivity and specificity of exposure, respectively, among controls are 0.9 and 0.8, this observed odds ratio of 1.8 means that exposure actually multiplies the risk of disease 10.7 times! These interpretations are not dependent on sample size; one could enlist all the disease cases in the world, along with thousands of controls for each case, and the meaning of that observed odds ratio of 1.8 would still depend on the sensitivity and specificity of exposure measures among cases and controls. If one had conducted the massive aforementioned study and then did what epidemiologists are wont to do, one would assemble a confidence interval around this 1.8 odds ratio and then conclude by discussing the implications of this very precise (but almost certainly incorrect) estimate of the effect of exposure on risk. The discussion might cite other sources to legitimize the claim that the measures of resin exposure and lung cancer are sufficiently valid, as was recently proposed for dietary data,11 then proceed to suggest that, nonetheless, this 1.8 relative risk might be a slight underestimate of the true impact of resin exposure on the risk of lung cancer.
The device proposed by Fox et al. permits the direct exploration of different measurement-error combinations, so that the investigator can test what might, according to these combinations, have generated what was observed; sampling variability can be incorporated into these tests.
Nodding shrugs that the data are probably sufficiently valid represent an unfortunate acceptance of imprecision that may well consign epidemiological inquiry to the scientific sidelines. Fox et al., however, remind epidemiologists how important data quality is to the conclusions that will be extracted from those data. Fox et al. demonstrate that statistical variability is by no means the most important source of inconsistency in epidemiological data, showing that interpretation of a dataset depends on the epidemiologist's understanding of the accuracy of exposure reports: they offer us a ready means of assessing the implications of interpreting our often very imprecise data.
What may result from this new and readily accessible instrument is that, in many instances, the epidemiologist may have the means to understand the dependence on measurement error of his or her epidemiological data. We can hope that epidemiologists serious about their science will begin to use the instrument.
 |
References |
|---|
 Top References |
|---|
1 Fox M, Lash T, Greenland S. A Method to automate probabilistic sensitivity analyses of misclassified binary variables (this issue). Int J Epidemiol, 2005;34:1370–76.
2 Willett WC, Sampson L, Stampfer MJ et al. Reproducibility and validity of a semiquantitative food frequency questionnaire. Am J Epidemiol 1985;122:51–65.
3 Byers T, Marshall J, Fiedler R, Zielezny M, Graham S. Assessing nutrient intake with an abbreviated dietary interview. Am J Epidemiol 1985;122:41–50.
4 Byers T, Marshall J, Anthony E, Fiedler R, Zielezny M. The reliability of dietary history from the distant past. Am J Epidemiol 1987;125:999–1101.
5 Willett W. Nutritional Epidemiology. 2nd edn. New York; Oxford: Oxford University Press, 1998.
6 Bross I. Misclassification in 2x2 tables. Biometrics 1954;10:478–86.[CrossRef]
7 Greenland S. The effect of misclassification in the presence of covariates. Am J Epidemiol 1980;112:564–69.
8 Marshall J, Hastrup J. Mismeasurement and the resonance of strong confounders: uncorrelated errors. Am J Epidemiol 1996;143:1069–78.
9 Marshall J, Hastrup J, Ross J. Mismeasurement and the resonance of strong confounders: correlated errors. Am J Epidemiol 1999;150:334–40.
10 Marshall J. The use of dual or multiple reports in epidemiologic studies. Stat Med 1989;8:1041–49.[Web of Science][Medline]
11 Byers T. Food frequency dietary assessment: how bad is good enough? Am J Epidemiol 2001;154:1087–88.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  G. B. GORI Dimensional errors of metaphorical measurements. Can they be resolved? Int. J. Epidemiol., December 1, 2006; 35(6): 1590 - 1592. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||