IJE Advance Access published online on June 29, 2007
International Journal of Epidemiology, doi:10.1093/ije/dym119
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Placebo-controlled trials of Chinese herbal medicine and conventional medicine—comparative study
1 Institute of Social and Preventive Medicine (ISPM), University of Berne, Switzerland.
2 Department of Rheumatology, Inselspital, University of Berne, Switzerland.
3 MRC Health Services Research Collaboration, Department of Social Medicine, University of Bristol, UK.
* Corresponding author: Institute of Social and Preventive Medicine (ISPM), University of Berne, Switzerland. E-mail: egger{at}ispm.unibe.ch
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Background Chinese herbal medicine (CHM) is increasingly used in the West, but the evidence on its effectiveness is a matter of debate. We compared the characteristics, study quality and results of clinical trials of CHM and conventional medicine.
Methods Comparative study of placebo-controlled trials of CHM and conventional medicine. Eleven bibliographic databases and searches by hand of 48 Chinese-language journals. Conventional medicine trials matched for condition and type of outcome were randomly selected from the Cochrane Controlled Trials Register (issue 1, 2003). Trials described as double-blind, with adequate generation of allocation sequence and adequate concealment of allocation, were assumed to be of high quality. Data were analysed using funnel plots and multivariable meta-regression models.
Results 136 CHM trials (119 published in Chinese, 17 published in English) and 136 matched conventional medicine trials (125 published in English) were analysed. The quality of Chinese-language CHM trials tended to be lower than that of English-language CHM trials and conventional medicine trials. Three (2%) CHM trials and 10 (7%) conventional medicine trials were of high quality. In all groups, smaller trials showed more beneficial treatment effects than larger trials. CHM trials published in Chinese showed considerably larger effects than CHM trials published in English (adjusted ratio of ORs 0.29, 95% confidence intervals 0.17–0.52).
Conclusions Biases are present both in placebo-controlled trials of CHM and conventional medicine, but may be most pronounced in CHM trials published in Chinese-language journals. Only few CHM trials of adequate methodology exist and the effectiveness of CHM therefore remains poorly documented.
Accepted 4 May 2007
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Traditional Chinese medicine (TCM) is a system of health care with a unique theoretical and diagnostic basis that originated in China some 2500 years ago. The core concepts suggest that disease is the result of imbalances in the flow of the body's vital energy, or ‘qi’ (pronounced ‘chee’), and that the human body is a microcosm of the basic natural forces at work in the universe. In modern China, both TCM and conventional medicine are practised, with TCM accounting for about 40% of all health care delivered.1 TCM is also increasingly popular in industrialized countries.2,3 Acupuncture and Chinese herbal medicine (CHM) are most widely used both in China and in the West.
Although TCM has a long history, its effectiveness continues to be debated. The evidence on CHM is particularly controversial. Common criticisms include the poor methodological quality of trials, and the biased dissemination of their results.4–7 The situation is complicated by the fact that many clinical trials in CHM are published in Chinese, and therefore inaccessible to most Western researchers.8
Bias in the conduct and reporting of trials is a possible explanation for positive findings of both placebo-controlled trials of CHM and conventional medicine.5,910 We compared the characteristics and results of placebo-controlled trials of CHM, including trials published in Chinese, with a matched sample of conventional medicine trials, and assessed the quality of trials and publication and related biases.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Literature searches
We searched 11 electronic databases, covering the periods from inception to January 2003: MEDLINE, EMBASE, CINAHL, AMED, MANTIS, Toxline, PASCAL, BIOL, Science Citation Index, CENTRAL and SIGLE. The search terms in MEDLINE were [‘Medicine, Chinese Traditional’(MESH) and phytotherap* or herbal or herb or herbs or ‘plant extracts’ or ‘plant extract’] or [‘Drugs, Chinese Herbal’(MESH) or ‘Chinese herbs’ or ‘Chinese herb’] and [placebo* or placebos(mesh) or ‘placebo effect’(mesh) or sham]. Search terms were similar for the other databases. We checked the reference lists of relevant articles and contacted experts in the field. No language restrictions were used.
Since many medical journals published in China are inaccessible in Europe, we performed additional searches at Shanghai University of Traditional Chinese Medicine. We used CBMdisc, a Chinese-language electronic database, to identify journals that publish clinical trials of CHM. CBMdisc goes back to 1980 and is the most comprehensive and accessible literature database in China. We found 63 potentially eligible trials, which were published in 27 TCM journals and 16 conventional medicine journals. Library staff in Shanghai indicated that an additional five TCM journals might be relevant. With local help (Acknowledgements section), we therefore searched 48 journals by hand. Since few placebo-controlled trials were published before 1990, all issues from January 1990 to March 2003 were searched, except for two journals: ‘Zhongguo Zhong Xi Yi Jie He Za Zhi’ and ‘Zhong Yi Za Zhi’. These two journals are the leading Chinese-language journals in TCM and were searched back to 1980.
We searched the Cochrane Central Controlled Trials Register (CENTRAL) to identify a matched placebo-controlled trial of conventional medicine for each eligible CHM trial, using keywords relevant to condition and outcome. CENTRAL is a bibliographic database of controlled trials maintained by the Cochrane Collaboration.11 We searched issue 1, 2003 of CENTRAL which contained 353 809 bibliographic references.
Study selection
We defined inclusion and exclusion criteria a priori and applied the same criteria to trials of CHM and conventional medicine. Inclusion criteria were (i) controlled trials of treatments or preventative measures with clinical outcomes, (ii) parallel group design with placebo control, (iii) available written report, such as a journal publication, abstract, thesis, conference proceeding, unpublished report, book chapter or monograph and (iv) sufficient information to allow the calculation of odds ratios (ORs). We excluded cross-over trials, trials in healthy volunteers and N-of-1 trials.
Selection of outcomes and matching procedures
We used pre-specified criteria for identifying outcomes for inclusion in analysis. The first choice was the main outcome measure, which was defined as the outcome used for sample size calculations. If a main outcome was not specified, we selected alternative outcomes, in the following order: (i) patients’ global assessment of improvement; (ii) physicians’ global assessment of improvement; (iii) the clinically most relevant other outcome measure (for example, the occurrence or duration of an illness). Outcomes were selected at random if several outcomes were considered equally relevant.
For each CHM trial, we identified matching trials of conventional medicine, which enrolled patients with similar conditions and assessed similar outcomes. We used computer-generated random numbers to select one out of several eligible trials of conventional medicine. Selection of outcomes and matching of trials was done without knowledge of trial results.
Data extraction and definitions
We used a piloted data extraction sheet. Except for Chinese-language trials, which were assessed only by one of us (AS), data extraction was done independently by two observers, with discrepancies being resolved by consensus. CHM was classified into individualized therapy, partially individualized therapy, non-individualized therapy and unclear type of CHM. In trials of individualized therapies, remedies were chosen according to individual signs and symptoms. Partially individualized therapy consisted of a basic remedy that was used for all patients and individualized by adding or omitting some herbs. If all patients used the same remedy, the intervention was defined as non-individualized. Diagnosis was based on TCM (for example, ‘qi’ deficiency), Western diagnosis (for example, hypertension) or a combination of the two. Other information collected on CHM interventions included whether Chinese names of herbs were mentioned, whether the dosage of each herb was reported and whether remedies were changed during the study.
Assessment of study quality
Assessment of study quality focused on key domains of internal validity:9,12 randomization (generation of allocation sequence and concealment of allocation) and blinding (of patients, therapists and outcome assessors). Use of a random-number table, computer-generated random numbers, minimization, coin tossing, shuffling cards and drawing lots were classified as adequate methods for the generation of the allocation sequence. Sealed, opaque sequentially numbered assignment envelopes, central randomization, independently prepared and coded drug packs of identical appearance, and on-site computerized randomization systems were considered adequate methods of allocation concealment. Descriptions of other methods were coded either as inadequate or unclear, pending on the level of detail provided. Trials described as double-blind, describing adequate methods for the generation of allocation sequence and adequate concealment of allocation were classified as of high methodological quality.
Graphical and statistical analysis
CHM trials and conventional medicine trials were analysed separately. We expressed results of each trial on the OR scale and used the method described by Hasselblad and Hedges13 to convert differences in continuous outcomes to ORs. We recoded outcomes if necessary, so that ORs <1 always indicated a beneficial effect of treatment. We examined heterogeneity between trials using the I-squared statistic.14 We investigated the association between study size and trial results in funnel plots, by plotting ORs on the horizontal axis (on a logarithmic scale) against their standard errors on the vertical axis.15 The extent to which study-level variables were associated with log ORs was examined by fitting univariable and multivariable meta-regression models separately for CHM trials and conventional medicine trials.16 The following variables were considered: standard error of log OR, language and year of publication, indexing of publication in MEDLINE and trial quality (blinding, generation of allocation sequence and concealment of allocation).
Results are given as ORs, ratios of ORs or asymmetry coefficients with 95% confidence intervals (95% CI). Ratios of ORs of <1 correspond to a smaller OR for trials with the characteristic and hence a larger apparent benefit of the intervention. Funnel plot asymmetry was measured by the asymmetry coefficient: the ratio of ORs per unit increase in standard error of log OR.17 All analyses were performed in Stata version 9.0 (Stata Corporation, College Station, TX, USA).
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
We identified 334 potentially eligible reports of placebo-controlled trials of CHM and excluded 199 reports. The most frequent reasons for exclusion were other type of intervention examined, no clinical outcomes reported and no placebo group (Figure 1). We included 135 publications, which reported on a total of 136 independent trials of CHM and 136 publications of 136 matched trials of conventional medicine. The bibliographic details of these trials are given in Appendices 1 and 2 (available at: www.ispm.ch/downloads).
|
Trial characteristics
Cardiovascular, gynaecological and obstetrical disorders were the most common conditions studied in pairs of trials of CHM and conventional medicine (Table 1). Close matching of outcomes was not possible in some instances, leading, for example, to global assessments of response being analysed in 66 (49%) of CHM trials, but only in 51 (38%) of trials of conventional medicine (Table 2).
|
|
The characteristics of CHM trials depended on language of publication: trials published in Chinese had larger sample sizes but were less likely to be indexed in MEDLINE than CHM trials published in English. The reported methodological quality of Chinese-language CHM trials tended to be lower than that of English-language CHM trials. Only one CHM trial published in Chinese and two CHM trials published in English were classified as of high methodological quality. Conventional medicine trials published in other languages tended to be smaller (median sample size 39 compared with 60) but there were few differences regarding methodological quality. Only 10 (7%) conventional medicine trials were of high quality (Table 2).
Characteristics of CHM interventions are presented in Table 3. The majority of trials was based on Western diagnosis (103, 76%). Most CHM remedies were taken orally. Reporting of the names of herbs was incomplete in 74 (55%) trials and only few trials (34, 25%) described the preparation of remedies. A change of remedies during follow-up was not reported in any trial. Only two trials identified interventions as individualized CHM, three were partially individualized. Most conventional medicine trials examined drugs (125, 92%), seven (5%) were concerned with vitamins or dietary supplements, two with the evaluation of a vaccine and two with immunotherapy.
|
Graphical and statistical analyses of treatment effects
Most ORs indicated a beneficial effect of the intervention. The degree of between-trial heterogeneity was similar for CHM and for conventional medicine. The proportion of total variation in the estimates of treatment effects due to between-study heterogeneity (I-squared)18 was 83% for CHM and 84% for conventional medicine.
Funnel plots were asymmetrical, with smaller trials (larger SEs) in the lower part of the plot showing more beneficial treatment effects than larger trials (smaller SEs, Figure 2). In meta-regression models, the association between SEs and treatment effects was stronger for CHM trials published in Chinese than trials published in English: the asymmetry coefficients were 0.09 (95% CI 0.04–0.17) and 0.38 (95% CI 0.06–2.31), respectively. Therefore, for each unit increase in the SE, the OR decreased by a factor of 0.09 for Chinese-language CHM trials and factor 0.38 for English-language CHM trials. The asymmetry coefficient was 0.29 (95% CI 0.14–0.61) for conventional medicine.
|
In meta-regression analyses, the standard error of the log OR (asymmetry coefficient) was the dominant variable in both groups. In CHM (but not in conventional medicine) language of publication continued to be an important, independent predictor of treatment effects. The ratio of ORs comparing Chinese with English CHM trials was 0.28 (95% CI 0.15–0.52) in univariable analysis and 0.29 (95% CI 0.17–0.52) in multivariable analysis adjusted for the standard error of the log OR. In multivariable analyses of both the CHM and conventional medicine trials, there was little evidence (P > 0.10) for an association of treatment effects with other variables, including study quality.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
We compared the characteristics and quality of published placebo-controlled trials of CHM with comparable trials of conventional medicine and examined the presence of bias due to inadequate methodology and selective publication. We found that, in general, smaller trials showed more beneficial effects than larger trials. In trials of CHM, study quality and results depended on language of publication: CHM trials published in English were of higher methodological quality and showed smaller effects than trials published in Chinese. There were very few placebo-controlled trials of CHM with adequate methodology. It is therefore not possible, based on the currently available placebo-controlled trials of CHM to confirm or exclude beneficial effects of CHM. Similarly, most of the placebo-controlled trials of conventional medicine had methodological deficiencies and only few trials were of high quality.
Strengths and weaknesses
To our knowledge, this is the first study directly comparing the presence of biases and their influence on effect estimates from clinical trials of CHM and conventional medicine. Our electronic search of the literature was comprehensive and, importantly, complemented by an extensive search by hand of a large number of journals published in China. Indeed, most of the trials were published in Chinese and identified through the search conducted in China. We acknowledge that the identification of unpublished studies, or studies not indexed in the relevant databases is notoriously difficult, and it is possible that we missed some unpublished trials, for example trials published in Japanese or Korean. Conventional medicine trials were randomly selected from the largest existing database of clinical trials (the Cochrane CENTRAL registry), and matched to herbal medicine trials for clinical area and type of outcome.
A limitation of our review is the focus on the beneficial effects of CHM and conventional medicine, rather than on both benefits and risks. However, the trials included in our study were small and lacked the power to reveal infrequent but important adverse effects. Furthermore, reporting on adverse effects has been shown to be inadequate even in larger trials.19 It is, therefore, unlikely that a comprehensive and valid assessment of adverse effects would have been possible within the framework of the present study. We stress that we did not examine the validity of the complex diagnostic system that is part of CHM.
Different sources of bias are difficult to disentangle. The methodological quality of randomized trials cannot reliably be assessed from published articles because reporting on important aspects of methodology is often incomplete, and the quality of reporting is an inadequate proxy measure for methodological quality.12,20 Deficiencies in methodology of smaller trials that were either not reported by the authors or not assessed by us may therefore have contributed to the asymmetrical shape of the funnel plots. Small studies of CHM may show more beneficial effects than larger ones because it is more feasible in small studies to treat patients individually and change remedies according to the change of their symptoms. However, none of the 136 trial reports mentioned changes in remedies during follow up and only two trials were identified as using individualized treatments.
Findings in context with other studies
In a similar study, we recently compared placebo-controlled trials of homoeopathy and conventional medicine.21 In contrast to the present study, the degree of funnel plot asymmetry was similar in trials of homoeopathy and conventional medicine. Trials of homoeopathy tended to be of higher methodological quality than conventional medicine trials, although most trials of either type of medicine were also of low or uncertain quality. Vickers and colleagues,22 in a review of controlled studies examined whether certain countries produce only positive results. They found that published clinical trials conducted in China almost never report an experimental treatment to be equal or inferior to control. Our results confirm their findings for placebo-controlled trials of CHM and indicate that greater publication bias and the lower methodological quality of trials conducted in China might explain this phenomenon.
Tang and co-workers8 identified almost 3000 randomized controlled trials in a search of 28 Chinese journals of traditional Chinese medicine. Most trials examined herbal treatments, however, they generally compared two herbal preparations and did not express effectiveness in numerical terms.8 Our search covered 48 Chinese journals, but we identified only 136 parallel-group, placebo-controlled trials with clinically relevant and quantifiable outcomes. In line with Tang et al.'s survey8, most Chinese-language clinical trials of CHM were not placebo-controlled.
Several systematic reviews of trials of CHM have been done by the Cochrane Collaboration and other groups in recent years. The conclusions make depressive reading: ‘based on one low quality trial, the medicinal herb ... may have an antiviral activity;’23 ‘Because the trial methodology of these studies was often inadequate or insufficiently documented, it is difficult to recommend the use of CHMs ... ,’6 ‘At present, it is unclear whether Chinese herbal treatments ... do more good than harm.’7 These conclusions are not surprising in the light of our study: if only very few large high-quality trials exist, then systematic reviews of individual CHM interventions will be based on a few small trials of low quality.
Implications and future research
We agree with Ernst24 that in the light of the popularity of herbal medicine, more research is required to clarify the proper place of herbalism, including CHM, in modern health care systems. Both large high-quality randomized trials to examine the effectiveness, or otherwise, of herbal preparations, and research that aims to identify the active component or components of herbal medicines are needed.
|
Acknowledgements |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
We are grateful to Dr Fritz Grossenbacher for his valuable help with electronic literature searches and to Xiaoyan Guo and Yibiao Zhou for their help with hand searching in China. This study was funded by the Complementary Medicine Evaluation Program [Programm Evaluation der Komplementärmedizin (PEK)] of the Swiss Federal Office for Public Health.
Conflict of interest: None declared.
KEY MESSAGES
|
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
1 Hesketh T, Zhu WX. Health in China. Traditional Chinese medicine: one country, two systems. BMJ (1997) 315:115–17.
2 Eisenberg DM, Davis RB, Ettner SL, et al. Trends in alternative medicine use in the United States, 1990-1997: results of a follow-up national survey. JAMA (1998) 280:1569–75.
3 Nestler G. Traditional Chinese medicine. Med Clin North Am (2002) 86:63–73.[CrossRef][Web of Science][Medline]
4 Linde K, ter Riet G, Hondras M, Vickers A, Saller R, Melchart D. Systematic reviews of complementary therapies - an annotated bibliography. Part 2: Herbal medicine. BMC Complement Altern Med (2001) 1:5.[CrossRef][Medline]
5 Pittler MH, Abbot NC, Harkness EF, Ernst E. Location bias in controlled clinical trials of complementary/alternative therapies. J Clin Epidemiol (2000) 53:485–89.[CrossRef][Web of Science][Medline]
6 Liu C, Douglas RM. Chinese herbal medicines in the treatment of acute respiratory infections: a review of randomised and controlled clinical trials. Med J Aust (1998) 169:579–82.[Web of Science][Medline]
7 Armstrong NC, Ernst E. The treatment of eczema with Chinese herbs: a systematic review of randomized clinical trials. Br J Pharmacol (1999) 48:262–64.[CrossRef]
8 Tang JL, Zhan SY, Ernst E. Review of randomised controlled trials of traditional Chinese medicine. BMJ (1999) 319:160–61.
9 Schulz KF, Chalmers I, Hayes RJ, Altman D. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA (1995) 273:408–12.
10 Egger M, Davey Smith G. Meta-analysis: bias in location and selection of studies. BMJ (1998) 316:61–66.
11 Dickersin K, Manheimer E, Wieland S, Robinson KA, Lefebvre C, McDonald S. Development of the Cochrane Collaboration's CENTRAL Register of controlled clinical trials. Eval Health Prof (2002) 25:38–64.
12 Jüni P, Altman DG, Egger M. Assessing the quality of controlled clinical trials. BMJ (2001) 323:42–46.
13 Hasselblad V, Hedges LV. Meta-analysis of screening and diagnostic tests. Psychol Bull (1995) 117:167–78.[CrossRef][Web of Science][Medline]
14 Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ (2003) 327:557–60.
15 Sterne JAC, Egger M. Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol (2001) 54:1046–55.[CrossRef][Web of Science][Medline]
16 Thompson SG, Sharp SJ. Explaining heterogeneity in meta-analysis: A comparison of methods. Stat Med (1999) 18:2693–708.[CrossRef][Web of Science][Medline]
17 Sterne JAC, Egger M, Davey-Smith G. Investigating and dealing with publication and other biases in meta-analysis. BMJ (2001) 323:101–05.
18 Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med (2002) 21:1539–58.[CrossRef][Web of Science][Medline]
19 Ioannidis JP, Lau J. Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas. JAMA (2001) 285:437–43.
20 Huwiler-Müntener K, Jüni P, Junker C, Egger M. Quality of reporting of randomized trials as a measure of methodological quality. JAMA (2002) 287:2801–04.
21 Shang A, Huwiler-Muntener K, Nartey L, et al. Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy. Lancet (2005) 366:726–32.[CrossRef][Web of Science][Medline]
22 Vickers A, Goyal N, Harland R, Rees R. Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials (1998) 19:159–66.[CrossRef][Web of Science][Medline]
23 Liu JP, McIntosh H, Lin H. Chinese medicinal herbs for asymptomatic carriers of hepatitis B virus infection. Cochrane Database Syst Rev (2003) CD002231.
24 Ernst E. Herbal medicines: where is the evidence? BMJ (2000) 321:395–96.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||