IJE Advance Access originally published online on August 25, 2008
International Journal of Epidemiology 2008 37(6):1422-1429; doi:10.1093/ije/dyn173
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Quantifying the potential role of unmeasured confounders: the example of influenza vaccination
1 Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, The Netherlands.
2 Veterans Affairs Medical Center, Minneapolis, MN, USA.
* Corresponding author. UMC Utrecht, Julius Center for Health Sciences and Primary Care, PO BOX 85500, 3508 GA, Utrecht, The Netherlands. E-mail: r.h.h.groenwold{at}umcutrecht.nl
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionFundingReferences |
|---|
Background The validity of non-randomized studies using healthcare databases is often challenged because they lack information on potentially important confounders, such as functional health status and socioeconomic status. In a study quantifying the effects of influenza vaccination among community-dwelling elderly we assessed whether additional information on not routinely available covariates was indeed associated with exposure to influenza vaccination and could, therefore, have led to residual confounding in healthcare databases.
Methods We randomly selected 500 persons aged 65 years and older from the computerized Utrecht General Practitioner database. Information on exposure status and on demographics, co-morbidity status, prior healthcare use and medication use was extracted from the database. A questionnaire was used to obtain additional information on not routinely available risk factors [e.g. functional health status (SF-20), smoking status and alcohol consumption]. Missing data from the questionnaire was imputed and multivariable logistic regression analysis was applied to quantify the influence of covariates on the prediction of exposure to influenza vaccination. Within an existing dataset the potential impact of functional health status on the relation between influenza vaccination and mortality was simulated.
Results We obtained questionnaire data from 365 of 500 (73%) subjects. The model including routinely available data from the database appeared accurate in predicting exposure to influenza vaccination (c-statistic 0.86, 95% CI: 0.82–0.89). Functional health status was the only additional characteristic measured with the questionnaire that was not similar in vaccinated and unvaccinated subjects. However, extending the multivariable regression model with functional health status did not significantly improve the prediction of exposure to influenza vaccination, nor did it affect the relation between influenza vaccination and mortality.
Conclusion The potential for unmeasured confounding on the association between influenza vaccination and health outcomes as quantified in healthcare databases seems small for non-randomized intervention studies within extensive and reliable databases.
Keywords Confounding, bias, influenza vaccines
Accepted 28 July 2008
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingReferences |
|---|
Non-randomized or observational studies assessing the effects of interventions by definition follow daily medical practice. In the doctor's office, initiated interventions are therefore a result of both physician and patient preferences. Consequently, severely ill patients, i.e. the ones with the strictest indication, are more likely to receive the perceived beneficial intervention as compared to those in better health. This may result in incomparability of prognosis of patients who are exposed and unexposed to the medical intervention and therefore findings may be biased.1 This so-called ‘confounding by indication’ typically results in an underestimation of the treatment effect.2–4 In contrast, ‘preventive’ or patient-initiated interventions like statin use, low-fat diets or vitamin supplements are more likely to be taken by those with healthy lifestyles. Effect estimates in non-randomized studies pertaining to these interventions are usually artificially increased (so-called ‘healthy user bias’).5,6
Different methods, notably data analytical tools, have been proposed to control for confounders available in observational studies.4,7 Unmeasured confounding, however, cannot be adjusted for in the data analysis.8,9 Ideally, therefore, all potential confounders should be measured accurately. Such measurements, however, can become quite laborious and expensive. Many non-randomized intervention studies are based on large existing computerized healthcare databases, containing routinely obtained information on, for example, demographics, prior healthcare consumption and medication use of patients. However, the decision to prescribe a specific intervention in daily medical practice is often also determined by patient characteristics not (routinely) available, such as functional status, smoking or alcohol consumption. Hence, these may act as residual (and unmeasured) confounders.
For example, in non-randomized studies on influenza vaccine effectiveness, routinely available covariates such as age, gender, risk-elevating medical conditions, prior healthcare utilization and prescribed medication are commonly treated as potential confounders,2,10 since these are related both to the risk for experiencing clinically relevant outcomes (such as mortality or hospitalization) and to exposure to influenza vaccination.2,10,11 Such ‘confounding by indication’ may dilute the beneficial effect of influenza vaccination or even artificially induce an adverse effect.5 However, it has recently also been suggested that persons with a lower functional health status (a characteristic not routinely available in healthcare databases) are less likely to receive the vaccine,12 leading to overestimations of the beneficial effect of influenza vaccination. A confounder is associated with the exposure as well as the outcome under study. If one of these associations is absent the covariate cannot confound the association between exposure and outcome. Thus, absence of an association with exposure indicates no potential for confounding. We, therefore, set out to quantify the association between functional health status and other relevant not routinely available covariates and exposure to influenza vaccination among community-dwelling elderly. Furthermore, we simulated the potential impact of functional health status in an existing dataset on the association between influenza vaccination and mortality.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingReferences |
|---|
Setting
The computerized medical database of the University Medical Center Utrecht General Practitioner Network includes information on approximately 60 000 patients enlisted with 33 General Practitioners. This database contains medical as well as basic demographic information. Diagnoses are coded according to the ICPC (International Classification of Primary Care) system. The system complies with Dutch guidelines on the use of medical data for research purposes and has shown to be valid in influenza vaccine effectiveness studies.10,13
Study population
In The Netherlands, recommendations for influenza vaccination include those aged 65 years and older and specific risk groups (notably those with established pulmonary, cardiovascular and kidney disease) and uptake is around 75%.10 From the computerized medical database we randomly selected 500 patients aged 65 years and older as of September 1, 2006 who had received (250 patients) or not received (250 patients) influenza vaccination in 2006. Vaccination status was based on the presence or absence of the ICPC-code for influenza vaccination (R44.1). Earlier studies have shown a high agreement between the presence of this code in the medical database and vaccination status (
= 93%).10
Measurement of potential confounders
As in previous healthcare database studies,2,10,11 we extracted demographic and medical information from the electronic database. Demographic information included age and sex. Medical information included the following classes of risk-elevating co-morbidity based on the ICPC-coding system: cardiovascular co-morbidity [acute myocardial infarction (K75), congestive heart failure (K77), other cardiovascular diseases (K74, K76, K78–K80, K82–K84) or stroke (K90)], pulmonary co-morbidity [lung cancer (R84, R85), asthma or chronic obstructive pulmonary disease (R91, R95, R96)], diabetes (T90), malignancies (B72, B73, B74, D74–77, S77, T71, U75–77, X75–77, Y77) and renal insufficiency (U99). Co-morbidity status was ascertained by the presence of consultations encoded with one or more of the above-mentioned ICPC-codes during the 12 months preceding the influenza vaccination (November 1, 2005 to October 31, 2006). Furthermore, the healthcare consumption (in terms of the number of GP consultations) and medication use in the year preceding the influenza vaccination period were recorded as well as the vaccination status in the previous year (2005).
All randomly selected subjects received a questionnaire within 6 weeks after the influenza vaccination period in 2006. Within 3 weeks non-responders received a reminder. The questionnaire comprised questions on social characteristics (number of household members, educational level, etc), smoking and alcohol consumption, use of homeopathic treatment, as well as functional status. Functional status was assessed using the shorter form of the RAND-36, the MOS SF-20, which was validated for The Netherlands.14,15 This part of the questionnaire consisted of 20 items divided into six domains: physical functioning (six items), role functioning (two), social functioning (one), health perception (five), mental health (five) and body pain (one). All items were combined into one overall score. Possible values ranged from 28 through 82. A higher score on this questionnaire implies a better functional status. Functional status, as derived from the SF-20, has been shown to be related to mortality.16
Data analysis
Data were analysed using R for Windows (Version 2.4.1). Information that was missing from the returned questionnaires was assumed to be missing at random (MAR). This was determined by comparing data from patients without any missing value with data from patients with at least one missing value.17,18 Missing data were handled using multiple imputation, using the MICE algorithm.19 Ten imputed datasets were constructed. A baseline table, which was the average of the 10 imputed datasets, was constructed. Based on this baseline table, univariate associations of potential predictors of vaccination status were determined. Predictors with univariate associations with vaccination status (P < 0.15) were included in the multivariable logistic regression analysis.
In the analysis, we distinguished two types of data: data from the electronic patient file (routinely available potential confounders) and data from the questionnaire (not routinely available potential confounders). We developed multiple multivariable models that included different combinations of potential confounders to predict exposure to influenza vaccination. The simplest model included only demographics (age and sex) and the most extensive model based on routinely available variables included demographics, co-morbidity status, prior healthcare utilization and medication use. Each model was then compared with an extended model in which the potential confounders from the questionnaire were added.
On each of the 10 datasets the models were fitted. Based on these results final models were constructed (averaged betas), which were tested on each imputed dataset.17 Pooled results for each model were then compared using the Hosmer Lemeshow goodness-of-fit test for the likelihood ratios and by determining the discriminative value of each model (c-statistic). These were corrected for optimism by validating the model in 50 bootstrap samples.20
Sensitivity analysis was conducted to estimate the size of potential bias by selective non-response. Non-response was assumed to be associated either with a high functional health status or a low functional health status. Therefore, health status scores for non-responders were sampled either from the highest quartile of observed functional health status (i.e. non-response assumed to be associated with high functional health status) or from the lowest quartile (vice versa). This was repeated 100 times and results of the analyses were pooled.
The impact of non-routinely collected variables on the association between influenza vaccination and mortality was assessed by means of simulation. For this, the dataset of the PRISMA study was used.10 In short, the PRISMA study is a nested case–control study among 6676 community-dwelling elderly (age 
65 years). Functional health status was simulated such that the association with vaccination status mimicked the univariate association that was found in the questionnaire data. Furthermore, a low functional health status was assumed to increase risk of mortality 3-fold.12 Within the PRISMA dataset influenza vaccine effectiveness was estimated by logistic regression analysis, adjusting for routinely collected confounders (age, sex, co-morbidity status, medication and prior healthcare use) and additionally adjusting for functional health status as well.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingReferences |
|---|
In all, 365 of the 500 (73%) subjects returned a questionnaire. Four persons had either died or moved to an unknown address before sending the questionnaire, and were excluded from the analysis. Vaccine uptake was slightly lower in non-responders than responders, but other characteristics did not differ appreciably (Table 1). Only 47 of the 365 (13%) questionnaires had missing values for one or more items. Responders with complete data slightly differed from responders with incomplete data, for example with regard to age, sex and the number of prior GP consultations (Table 2) and we imputed the missing data using the known subject data.
|
|
Univariate analysis showed that age, co-morbidity status, prior healthcare and medication use were significantly associated with exposure to influenza vaccination. In addition, functional health status was lower among vaccinated [median: 63, interquartile range (IQR): 54–70] than unvaccinated subjects (median: 68, IQR: 61–75). In contrast, smoking, alcohol consumption, use of homeopathic substances, educational level, living situation, ethnicity and type of insurance were not associated with vaccination status (Table 3).
|
Since functional status was the only questionnaire variable associated with vaccination status (P < 0.001), we added just this variable to the multivariable models. Five different models were constructed, ranging from a simple model including only demographic information to a model including all medical database information (demographics, co-morbidity, prior healthcare and medication use). The simple model (including only demographics) was moderately discriminative in predicting vaccination status (c-statistic 0.65) and inclusion of the other routinely available covariates in the model increased the discriminative value (c-statistic for the full model 0.86). Additional inclusion of functional health status (or one of the subscales) as predictor of influenza vaccination status did not increase discriminative performance of the latter model (Table 4). Only the accuracy of the simple models (including demographics or demographics and co-morbidity status) in predicting vaccination status improved by including functional health status (c-statistic increased from 0.65 to 0.66, and from 0.68 to 0.69, respectively). Furthermore, in the extensive models (model D and E) functional health status was not an independent predictor in the multivariable model [β (95% CI): –0.02 (–0.05 to 0.01)].
|
In the sensitivity analysis of the size of potential bias by selective non-response inclusion of functional health status to the full model (model E) did not relevantly change the discriminative performance. The discriminative performance (c-statistic) of the full model (model E) without functional health status was 0.82 (95% CI: 0.79–0.85, based on 496 observations). When non-response was assumed to be associated with low functional health status or with high functional health status discriminative performance did not appreciably change: c-statistic 0.83 (95% CI: 0.79–0.86) and c-statistic 0.83 (95% CI: 0.80–0.86), respectively.
Within the PRISMA dataset, influenza vaccination reduced mortality risk by 48% [odds ratio (OR): 0.52, 95% CI: 0.38–0.71] after adjustment for routinely collected confounders (i.e. sex, age, co-morbidity status, medication use and prior healthcare use). After additional adjustment for functional health status influenza vaccine effectiveness did not materially change (OR: 0.47, 95% CI: 0.35–0.65).
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingReferences |
|---|
Non-randomized intervention studies are often based on information from healthcare databases. Discussions on unmeasured confounding in such studies frequently focus on patient characteristics, such as functional health status, not routinely available in these databases.2 Collection of such missing information, however, is often considered too laborious and expensive. Our example study showed that, in non-randomized intervention studies where extensive and reliable databases are being used (e.g. within the Dutch healthcare system), inclusion of such potential confounders did not improve the prediction of influenza vaccination status. In The Netherlands physicians used to support influenza vaccination especially among those with the highest risk for mortality (e.g. presence of cardiovascular co-morbidity). This is supported by our data, showing a clear contrast between vaccinated and unvaccinated subjects. In non-randomized studies from the United States this contrast is less clear.2 Therefore, the findings from such studies might be more prone to unmeasured confounding. Importantly, the model including only routinely available potential confounders was already highly predictive of vaccination status (c-statistic 0.86) and the other confounders from the questionnaire such as functional health status appeared highly correlated with the routinely available confounders. Consequently, after adjustment for routinely collected confounders, functional health status hardly changed the effect of influenza vaccination on mortality.
Recently, functional health status was proposed to be an important not routinely collected confounder of the association between influenza vaccination and mortality risk.12 Lower functional health status was assumed to be associated with lower vaccination rates while at the same time lower functional status is associated with higher mortality rates. Adjustment for functional status in a multivariable model without medical data moved the association between influenza vaccination and mortality towards the null (‘healthy user bias’).5,12 The authors therefore concluded that the available studies are biased by such unmeasured confounding and suggested the vaccine to be ineffective. Our findings indicate four important messages: first, the multivariable model including extensive routinely available data can adequately predict influenza vaccination status. Second, the distribution of potential confounders not routinely available was often similar in vaccinated and unvaccinated patients. Third, even variables that differed between exposure groups (i.e. functional health status) did not improve the prediction of vaccination status (nor did it affect the relation between influenza vaccination and mortality), and fourth this indicates that much of the information of this latter possible confounder is already captured in the model. Fourth, if the association between functional health status and mortality would be stronger than what we assumed, this would (if any) increase vaccine effectiveness, instead of reducing it (as has been suggested).5,12
Similarly, additional inclusion of certain variables in a multivariable model-relating influenza vaccination to clinical outcomes was found not to change the effect estimate for vaccination.4 Indeed, after inclusion of a limited number of routinely available confounders the effect estimate stabilized and additional inclusion of potential confounders is unlikely to result in relevant changes.4 Such stabilization of the effect estimate is a strong argument against residual, unmeasured confounding. Prior healthcare consumption was highly correlated with functional health status. In the Dutch healthcare system healthcare consumption can be measured accurately, since each patient is registered with one GP, whose database contains all medical information of that patient. However, in countries with other healthcare systems, functional health status may add information to a multivariable model. Furthermore, other not routinely collected variables than we measured in our study (i.e. other than smoking status, alcohol consumption, etc) may still confound the relation between influenza vaccination and mortality.
When subscales of functional health status, instead of sum scores, were included as covariates in the model only inclusion of the subscales ‘health perception’ and ‘body pain’ increased the performance of the model. Nevertheless, inclusion of these covariates did not increase the discriminative value of the model. Since health perception is inversely related to both vaccination status and mortality16 confounding by this covariate would underestimate vaccine effectiveness. Similarly, body pain is inversely related to both vaccination status and mortality and, therefore, confounding by this covariate would underestimate vaccine effectiveness as well.
There was no full response to the questionnaire. If this response was selective, this may have resulted in biased findings. It is possible that functional health status is related to the likelihood of responding to the survey. However, this seems unlikely since functional health status is most probably related to prior healthcare consumption and co-morbidity status. These covariates, however, are well balanced between responders and non-responders (Table 1). Nevertheless, responders and non-responders differed with respect to vaccination status and prior vaccination status. However, in the total population of 496 responders and non-responders 234 out of 300 prior vaccinated subjects received a new vaccination (78%), while 14 out of 196 subjects who were not vaccinated in the previous year received vaccination in the current year (7%). Among the responders these proportions were similar, 79 and 7%, respectively. We, therefore, believe selection bias will only (if any) have a limited effect on the results of this study. Furthermore, sensitivity analysis of the size of potential bias by selective non-response did not show a change in the discriminative value of the model when non-responders were assumed to have either extremely low functional health status, or extremely high functional health status.
A potential confounder is, by definition, unequally distributed between those receiving and those not receiving the intervention and at the same time related to the health outcome under study. If equally distributed a confounding effect is absent by definition. However, in our study, functional health status was unequally distributed among vaccinated and unvaccinated subjects. Furthermore functional status, as derived from the SF-20, is known to be related to mortality.16 Thus, functional status is a potential confounder. However, when the information of the potential confounder is already contained in other variables included in the multivariable model, a confounding effect by that variable is not to be expected since addition of this variable will not improve the prediction of exposure status. This is supported by the observation that functional health status did hardly change influenza vaccine effectiveness. We propose to routinely report the predictive (or discriminative) ability (for example summarized in the c-statistic or area under the receiver-operating curve) of observed confounders in predicting exposure status to inform readers about the potential for residual confounders in observational studies. A high discriminative value in predicting exposure status leaves little room for increase by including additional variables, and, thus, for residual confounding. Then, the discriminative ability of observed confounders in predicting exposure status can be a directive in the discussion on unmeasured confounding in observational intervention studies. However, future studies are warranted to further support the validity of this approach.
Our study shows that the discriminatory power of routinely collected confounders (i.e. demographics, co-morbidity status, healthcare use and medication use) in predicting influenza vaccination status is already high. Therefore, unmeasured confounding in studies on influenza vaccine effectiveness in which these variables are adjusted for is likely to be very small or absent.2,10 Results indicating that influenza vaccination reduced mortality risk, therefore, seem valid. However, since the discriminatory performance was still not optimal, unmeasured confounding might be present. This (probably small) potential for unmeasured confounding can be studied, for example, by using ‘off season’ data, since outside the influenza epidemic period no effect of influenza vaccination is expected.2,21
In conclusion, since non-randomized intervention studies are prone to confounding, information on as many potential confounders as possible should be collected. Therefore, not only information from healthcare databases, but also information on not routinely collected variables, such as functional health status, should ideally be obtained. We believe our study shows this to be feasible. However, if the discriminative value of a model predicting exposure status is already high, the collection of such information does not seem to be worth the effort.
|
Funding |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingReferences |
|---|
Netherlands Scientific Organization (VENI no. 916.56.109 to E.H.).
Conflict of interest: Dr Nichol reports having served as a consultant to the influenza vaccine manufacturers Sanofi Pasteur, MedImmune, GSK and Novartis. She has received grant support from Sanofi Pasteur and GSK.
KEY MESSAGES
|
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingReferences |
|---|
1 Miettinen OS, Cook EF. Confounding: essence and detection. Am J Epidemiol (1981) 114:593–603.
2 Nichol KL, Nordin JD, Nelson DB, Mullooly JP, Hak E. Effectiveness of influenza vaccination in the community-dwelling elderly. N Engl J Med (2007) 357:1373–81.
3 Salas M, Hofman A, Stricker BH. Confounding by indication: an example of variation in the use of epidemiologic terminology. Am J Epidemiol (1999) 149:981–83.
4 Hak E, Verheij ThJM, Grobbee DE, Nichol KL, Hoes AW. Confounding by indication in non-experimental evaluation of vaccine effectiveness: the example of prevention of influenza complications. J Epidemiol Community Health (2002) 56:951–55.
5 Hak E, Hoes AW, Nordin J, Nichol KL. Benefits of influenza vaccine in US elderly–appreciating issues of confounding bias and precision. Int J Epidemiol (2006) 35:800–02.
6 Toh S, Hernandez-Diaz S. Statins and fracture risk. A systematic review. Pharmacoepidemiol Drug Saf (2007) 16:627–40.[CrossRef][Web of Science][Medline]
7 Klungel OH, Martens EP, Psaty BM, et al. Methods to assess intended effects of drug treatment in observational studies are reviewed. J Clin Epidemiol (2004) 57:1223–31.[CrossRef][Web of Science][Medline]
8 Stram DO, Huberman M, Wu AH. Is residual confounding a reasonable explanation for the apparent protective effects of beta-carotene found in epidemiologic studies of lung cancer in smokers? Am J Epidemiol (2002) 155:622–28.
9 Vandenbroucke JP. When are observational studies as credible as randomised trials? Lancet (2004) 363:1728–31.[CrossRef][Web of Science][Medline]
10 Hak E, Buskens E, van Essen GA, et al. Clinical effectiveness of influenza vaccination in persons younger than 65 years with high-risk medical conditions: the PRISMA study. Arch Intern Med (2005) 165:274–80.
11 Hak E, Wei F, Nordin J, Mullooly J, Poblete S, Nichol KL. Development and validation of a clinical prediction rule for hospitalization due to pneumonia or influenza or death during influenza epidemics among community-dwelling elderly persons. J Infect Dis (2004) 189:450–58.[CrossRef][Web of Science][Medline]
12 Jackson LA, Nelson JC, Benson P, et al. Functional status is a confounder of the association of influenza vaccine and risk of all cause mortality in seniors. Int J Epidemiol (2006) 35:345–52.
13 Hak E, van Loon S, Buskens E, et al. Design of the Dutch prevention of influenza, surveillance and management (PRISMA) study. Vaccine (2003) 21:1719–24.[CrossRef][Web of Science][Medline]
14 Ware JE Jr, Sherbourne CD. The MOS 36-item short-form health survey (SF-36). I. Conceptual framework and item selection. Med Care (1992) 30:473–83.[Web of Science][Medline]
15 Kempen GI. [Assessment of health status of the elderly. Application of a Dutch version of the MOS scale]. Tijdschr Gerontol Geriatr (1992) 23:132–40.[Medline]
16 Franks P, Gold MR, Fiscella K. Sociodemographics, self-related health, and mortality in the US. Soc Sci Med (2003) 56:2505–14.[CrossRef][Web of Science][Medline]
17 Crawford SL, Tennstedt SL, McKinlay JB. A omparison of analytic methods for non-random missingness of outcome data. J Clin Epidemiol (1995) 48:209–19.[CrossRef][Web of Science][Medline]
18 Greenland S, Finkle WD. A critical look at methods handling missing covariates in epidemiologic regression analyses. Am J Epidemiol (1995) 142:1255–64.
19 van Buuren S, Oudshoorn C. Available at: http://web.inter.nl.net/users/S.van.Buuren/mi/hmtl/mice.htm (Accessed on October 30, 2007).
20 Clark TG, Altman DG. Developing a prognostic model n the presence of missing data: an ovarian cancer case study. J Clin Epidemiol (2003) 56:28–37.[CrossRef][Web of Science][Medline]
21 Simonsen L. Taylor RJ, Viboud C, Miller MA, Jackson LA. Mortality benefits of influenza vaccination in elderly people: an ongoing controversy. Lancet Infect Dis (2007) 7:658–66.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  R. H. H. Groenwold, A. W. Hoes, and E. Hak Impact of influenza vaccination on mortality risk among the elderly Eur. Respir. J., July 1, 2009; 34(1): 56 - 62. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||