IJE Advance Access published online on September 17, 2007
International Journal of Epidemiology, doi:10.1093/ije/dym168
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Editorial |
Quality in epidemiological research: should we be submitting papers before we have the results and submitting more hypothesis-generating research?
MRC Centre for Causal Analyses in Translational Epidemiology & Department of Social Medicine, University of Bristol, UK.
Canynge Hall, Whiteladies Road, Bristol, BS8 4SH, UK. E-mail: d.a.lawlor{at}bristol.ac.uk
Accepted 19 July 2007
All epidemiological studies, including observational epidemiology, randomized controlled trials and genetic epidemiology studies, have potential limitations. Even when studies are conducted to the highest possible quality, residual confounding might explain associations found in observational epidemiology,1 randomized trials might not be generalizable to the target population (or sub-groups),2 and genetic association studies might be affected by population stratification.3 In reality the situation is worse since there is ample evidence that many observational studies,4,5 randomized controlled trials6,7 and genetic association studies8–10 are poorly conducted or reported, often in ways that can have profound effects on causal understanding and, as a consequence, detrimental effects on the development and use of effective interventions for disease prevention and treatment.
One attempt to improve this situation is the generation of a number of guidelines for reporting different study types. These guidelines include ‘CONsolidation Standards Of Reporting Trials’ (CONSORT) [http://www.consort-statement.org], ‘QUality Of Reporting Of Meta-analyses’ (QUOROM) [http://www.consort-statement.org/QUOROM.pdf] and ‘STrengthening the Reporting of OBservational studies in Epidemiology’ (STROBE) [http://www.strobe-statement.org/]. Many influential journals have signed up to these guidelines, meaning that reports must conform to them in order to be published.
One potential source of bias that can affect all study types and that reporting guidelines are unlikely to influence is that of publication bias. Indeed, better reporting of studies might result in greater publication bias if reviewers and editors find it easier to see some ‘legitimate’ reason for rejecting a study with results that do not accord with their vision of the world. Further, evidence suggests that studies are considerably less likely to be published if they report no association (usually defined on the basis of a large P-value) than those that report an association with a small P-value, particularly if the direction of association is as conventional wisdom would expect it to be.11
Trial registers are an important advance in trying to minimize publication bias for randomized controlled trials. However, for observational studies, including genetic association studies, this approach will not work. Cohort studies have an ever-increasing number of variables and, rightly, are a source for efficiently investigating many different hypotheses to those originally planned. Requirements to register every hypothesis that anyone thought of doing in a cohort study would potentially stifle scientific advance and such a requirement could be a barrier to the rapid replication and publication of potentially important findings (see for example, the recent discovery of the association of a new genetic variant with body mass index and its subsequent rapid replication in a number of independent studies.12 The requirement to register this analysis in each individual study would have stifled timely collaborative research).
A possible solution would be to change the journal submission and review process to one where studies are initially submitted and reviewed without results: a ‘prior to results submission’ policy. Submitted papers would have an introduction giving a clear description of why the study is important, what it aims to add to existing knowledge and the specific objectives and an appropriate methodological description conforming to any relevant reporting guidelines, including a very clear description of the statistical methods. I would also suggest that in the Methods section of the paper, the authors list their study's strengths and limitations with respect to addressing their specific research objectives and adding to scientific understanding.
Editors and reviewers would then assess the paper on the criteria that we ought to be assessing any paper on, namely how important the research objectives are (in light of what research already exists on the particular topic) and the quality of the study methods, including proposed statistical approaches. Once a paper had been accepted the author would be asked to return it with the results included.
Some journals do operate a similar policy for randomized controlled trials. For example, the Lancet will review trial protocols written prior to starting the trial (and so clearly without results) and provide some commitment to reviewing and publishing the trial findings:
The Lancet will assess protocols of randomised trials and systematic reviews and will publish on our website summaries of those protocols that survive review. It is best to submit your protocol before the start of the trial; all trials should be registered. We also make a commitment to seek peer review of the paper reporting the primary clinical data of those protocols that we publish (see Lancet 2001; 357: 1819-20), and therefore encourage submission of such papers to The Lancet.’ (see,http://www.thelancet.com/authors/lancet/authorinfo#protocol)
However, the question remains as to whether a similar policy could work with observational epidemiology, where the potential for publication bias is greater, the ability to detect it less, but the potential adverse consequences in terms of distorting causal understanding and health policy equal to that of randomized controlled trials. When I have discussed the idea of ‘prior to results submission’ for observational epidemiology (including genetic epidemiology) with colleagues, responses are varied, but a number of valid objections have been raised.
Journals should only publish ‘important nulls’
The implication here is that whether a null finding is important or not can only be determined once we know that the result is null. To a large extent this is at the root of publication bias.11 In theory, as reviewers and editors we should be able to decide whether a particular research question is important irrespective of what the results might show. Whilst for the large part I think this is true, it is also true that the impact (importance) of results is likely to be greatest when the priors were low. Thus, reviewers could reject a paper with a low prior probability of revealing anything important and in doing so impede the timely publication of important results. This can be thought of as how we deal with ‘chance’ or unexpected findings.
How would this approach deal with ‘chance’ findings in research?
Scientific research has to have room for chance or unexpected findings, and as discussed earlier such findings might be rejected more readily under a ‘prior to results submission’ policy. That said, the concern that a ‘prior to results submission’ might result in delayed or lack of publication of important but unlikely findings has to be balanced against current practice in observational epidemiology, which tends to result in the publication of false-positive findings and the presentation of hypothesis generating research as hypothesis testing research.
A pertinent example of an important but unexpected finding is provided in the current issue of the IJE as a ‘classic’ reprint.13,14 Kune, Kune and Watson explored the association of a large number of chronic illnesses, operations and medications with colorectal cancer in a case control study of 727 cases.13,14 They found independent associations of hypertension, heart disease, chronic arthritis and use of aspirin (all protective) with colorectal cancer. The authors were completely honest about the nature of their study—‘... partly as an exploratory step to stimulate the creation of new hypotheses of colorectal cancer aetiology.’ and noted that ‘The authors had no a priori hypotheses regarding colorectal cancer and these factors.’ With respect to aspirin they commented ‘This finding, whatever the mechanism may be, has potential significance in colorectal cancer chemoprevention and merits early confirmation.’ The finding with respect to aspirin was novel; it had not been previously reported in any publications.
Subsequent research, described in detail in this issue by Gabriel Kune,15 provided strong evidence that this ‘chance’ association was likely to be causal. This evidence includes the long-term follow-up of two large randomized controlled trials that were initially conducted to explore the effect of aspirin in the primary prevention of cardiovascular disease,16,17 which show that aspirin is effective in the long-term primary prevention of colorectal cancer. A systematic review of these trials concludes that ‘Use of 300mg or more of aspirin a day for about 5 years is effective in primary prevention of colorectal cancer in randomised controlled trials, with a latency of about 10 years, which is consistent with findings from observational studies.’18
An important question is whether the paper by Kune, Kune and Watson would have ever been published at all, or in a timely manner, if it had been submitted without results, but with an introduction and methods similar to those of the published paper. It is hard to see how reviewers and editors would have been convinced by the ‘we are going to explore the association of a large number of diseases, operations and medications with colorectal cancer’ without knowing that any of the results were ‘interesting’. However, if this paper had not been published it is unlikely that further work would have been undertaken to determine the long-term protective effect of aspirin on colorectal cancer—a potentially important preventive measure may have been missed.
If journals were to start a ‘prior to results submission’ policy for observational epidemiology, then it would be important to have sections of major journals that deal with ‘hypothesis generating’ papers. Papers submitted to this section would be clearly described as presenting unexpected findings and/or as hypothesis generating and would be submitted with their results. Even without a change to ‘prior to results submission’ policy, journals should promote ‘hypothesis generating’ papers so that investigators do not feel that they have to falsely dress-up such papers as hypothesis testing. There are specific journals dedicated to hypothesis generating research, but I think it would be valuable to have sections in major journals for hypothesis generating papers. The British Medical Journal did at one time have a section called ‘Research Pointers’ that encouraged publication of hypothesis generating papers (see for example,19–21). Unfortunately, ‘Research Pointers’ no longer features in the ‘Type of article’ list provided for authors by the British Medical Journal. But in order to encourage all investigators to see the importance of being honest about the nature of such papers, and in recognition of their future potential, it would be valuable to see major journals encouraging such submissions.
Health journals are newspapers and therefore interested in news—they will always want to report the earthquake that happened and not all the places without earthquakes
This point is more than just being about making journals an interesting read, and relates to the potential importance of unexpected findings as discussed earlier. A solution might be ‘prior to results submission’ for a substantial number of papers and regular publication of ‘hypothesis generating’ papers that are given appropriate prominence but clearly identified as needing further replication.
How can we trust peer reviewers to decide what is important to publish?
This is a potential problem for the peer review process in general. It shouldn’t be any greater or less with the system of ‘prior to results submission’. It requires journal editors to select associate editors and reviewers for a paper who are knowledgeable about the research area that the paper deals with. The advantage of not having access to the study's results at the time of review is that such reviewers will not be influenced by whether the results of a new study concord with their own results or their own beliefs about a particular association.
Where does the discussion section of the paper come into this?
I think a discussion of the study strengths and limitations (currently included in the discussion section of most papers) should go in the Methods section and be submitted in the ‘prior to results submission’. The discussion of how to interpret the results, including what they mean for future clinical, public health and research practice could, with this approach, become truly discursive. The paper could be published without a discussion section but linked to an interactive web-based discussion page, where the authors could place their initial interpretation of the findings, and how they think they should be used, and others with relevant knowledge and/or interest in the area could also contribute to this discussion. ‘Others’ would include the general public and patient groups as well as researchers and health professionals. To some extent the British Medical Journal ‘rapid responses’, and similar systems in other journals, currently provide this facility; though in its current form it does not replace the authors own discussion.
One disadvantage to this approach to the discussion is that policy makers, clinicians and others might want to have the discussion together with the main paper at some stage relatively soon after publication of the paper. As a solution to this the journal could agree to combine the two at regular intervals (say 2-monthly) up to 12 months after the initial publication of the paper and update the web-based full version of the paper with the added discussion. At 12 months there would remain a link to the webpage discussion board of the paper. A further issue that is less easy to resolve is that this approach excludes anyone without access to the web contributing to, or accessing, the discussion. This is a potential problem not only for this suggested approach of ‘prior to results submission’ policy with a general web-based discussion, but also for open publication more generally, which is by and large attained via the world-wide-web. Finally, dealing with the discussion in this way would require extra resources that many journals would not be able to find.
Good analyses are pre-specified, but also iterative; ‘prior to results submission’ is too didactic
In observational epidemiology, we are all familiar with discussing (as a team of investigators) initial results from a pre-planned analysis and then deciding upon an additional set of analyses. That said, for most work it should be possible, and indeed desirable, to pre-specify the main analysis protocol, including (with justification) what factors will be included as potential confounders and potential mediators in multivariable models, how these covariables will be included in models, how models will be developed and what sub-group analyses will be undertaken. Any changes to this pre-specified analysis, either as a result of the initial review process or after seeing the initial results, should be highlighted as being ‘iterative’ and the justification for these additional analyses provided.
This only deals with publication bias that results from editors and reviewers not that which results from authors
This is true and of course most authors will do the analyses and have the results at the time that they submit their ‘prior to results submission’. The authors could undertake lots of analyses (particularly within large cohort studies22) and decide only to submit papers for those that they think are interesting; no one would have any way of checking this (either with the current system or a ‘prior to results submission’ policy). They could set out with a particular hypothesis, find a null result and decide not to submit for publication or they could write the introduction and methods of a paper to fit the results that they find (for example finding an a priori reason for a sub-group analyses that has really shown up post hoc). It is unlikely that there will ever be a single solution to dealing with all sources of publication bias in observational epidemiology. The ‘prior to results submission’ approach and promotion of hypothesis generation papers would give a message to everyone involved in research that the key criteria on which any research project is judged by peers is the importance of the question and an honest description of the methods used to address this. Perhaps with time such an approach would make us all a little more objective about our results and encourage us to have clearer thoughts about the importance of our research objectives and how best to address these.
This is a 2-stage review process and so will slow things down and require extra resources that most journals do not have
The issue of lack of resources for medical journals and the burden on reviewers is very real, and any process that resulted in a need for extra resources or that might increase reviewer burden as my suggestions would is unlikely to succed until the process of peer reviewing is fully valued and appropriately resourced.
Some concluding thoughts
Trial registers, reporting guidelines and major journals refusing to publish trials that are not registered, or papers that do not conform to reporting guidelines, probably seemed radical 20 years ago and will have had opposition. I am not convinced that ‘prior to results submission’ and the promotion of hypothesis-generating papers is any more radical. It is another way of trying to improve research quality, and in observational epidemiology we do need to think of radical ways of reducing publication bias and improving the quality of our research. That said my idea for ‘prior ro results submission’ is not new. I am grateful to lain chalmers for pointing me in the direction of a publication by Walster and Cleary that appeared 37 years ago and suggested the same approach for social sciences research23. It is a brilliant paper, one all epidemiologists should read, but it does not seem to have had any impact on editorial policy.
Acknowledgements
The ideas in this editorial have been discussed with a number of people who have given me important insights. These include (in alphabetical order of first name): Abigail Fraser (University of Bristol), Andy Ness (University of Bristol), Anne-Marie Nybo-Andersen (National Institute of Health, Copenhagen), George Davey Smith (University of Bristol), Mary Shaw (University of Bristol), Mia Madsen (National Institute of Health, Copenhagen), Thorkild Sorensen (Institute of Preventive Medicine at Copenhagen University hospital). In addition, Abigail Fraser (University of Bristol), George Davey Smith (University of Bristol) and Mary Shaw (University of Bristol) provided useful comments on an earlier draft of the editorial. The comments in this editorial are those of the author and not necessarily anyone mentioned in these acknowledgements. I thank Ian Chaucher's for introducing me to reference 23.
Conflict of interest: None declared.
References
1 Davey Smith G, Phillips AN. Confounding in epidemiological studies: why ‘independent’ effects may not be all they seem. Br Med J (1992) 305:757–59.[ISI][Medline]
2 Michels KB. The women's health initiative–curse or blessing? Int J Epidemiol (2006) 35:814–16.
3 Campbell CD, Ogburn EL, Lunetta KL, et al. Demonstrating stratification in a European American population. Nat Genet (2005) 37:868–72.[CrossRef][ISI][Medline]
4 Pocock SJ, Collier TJ, Dandreo KJ, et al. Issues in the reporting of epidemiological studies: a survey of recent practice. Br Med J (2004) 329:883.
5 Grimes DA, Schulz KF. Bias and causal associations in observational research. Lancet (2002) 359:248–52.[CrossRef][ISI][Medline]
6 Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA (1995) 273:408–12.[Abstract]
7 Thornley B, Adams C. Content and quality of 2000 controlled trials in schizophrenia over 50 years. Br Med J (1998) 317:1181–84.
8 Cardon LR, Bell JI. Association study designs for complex diseases. Nat Rev Genet (2001) 2:91–99.[CrossRef][ISI][Medline]
9 Colhoun HM, McKeigue PM, Davey Smith G. Problems of reporting genetic associations with complex outcomes: can we avoid being swamped by spurious findings? Lancet (2003) 361:865–72.[CrossRef][ISI][Medline]
10 Tabor HK, Risch NJ, Myers RM. Opinion: candidate-gene approaches for studying complex genetic traits: practical considerations. Nat Rev Genet (2002) 3:391–97.[ISI][Medline]
11 Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. Br Med J (1997) 315:640–45.
12 Frayling TM, Timpson NJ, Weedon MN, et al. A common variant in the FTO gene is associated with body mass index and predisposes to childhood and adult obesity. Science (2007) , Epub doi: 10.1126/science.1141634.
13 Kune GA, Kune S, Watson LF. Colorectal cancer risk, chronic illnesses, operations, and medications: case control results from the Melbourne Colorectal Cancer Study. Cancer Res (1988) 48:4399–404.
14 Kune GA, Kune S, Watson LF. Colorectal cancer risk, chronic illnesses, operations, and medications: case control results from the Melbourne Colorectal Cancer Study (Historical Reprint). Int J Epidemiol (2007).
15 Kune GA. Commentary: aspirin and cancer prevention. Int J Epidemiol (2007).
16 Peto R, Gray R, Collins R, et al. Randomised trial of prophylactic daily aspirin in British male doctors. Br Med J (1988) 296:313–16.[ISI][Medline]
17 Farrell B, Godwin J, Richards S, Warlow C. The United Kingdom transient ischaemic attack (UK-TIA) aspirin trial: final results. J Neurol Neurosurg Psychiatry (1991) 54:1044–54.[Abstract]
18 Flossmann E, Rothwell PM. Effect of aspirin on long-term risk of colorectal cancer: consistent evidence from randomised and observational studies. Lancet (2007) 369:1603–13.[CrossRef][ISI][Medline]
19 Roy S, Hill AV, Knox K, Griffiths D, Crook D. Research pointers: association of common genetic variant with susceptibility to invasive pneumococcal disease. Br Med J (2002) 324:1369.
20 Cullington HE. Light eye colour linked to deafness after meningitis. Br Med J (2001) 322:587.
21 Charles D, Ness AR, Campbell D, Davey Smith G, Hall MH. Taking folate in pregnancy and risk of maternal breast cancer. Br Med J (2004) 329:1375–76.
22 Davey Smith G, Lawlor DA, Harbord R, Timpson N, Day I, Ebrahim S. Clustered environments and randomized genes: a fundamental distinction between conventional and genetic epidemiology. PLoS Med (2007) . (In press).
23 Walster GW, Cleary A. A proposal for a new editorial policy in social sciences. The American Statistican (1970) 24:16–19.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  Erratum Int. J. Epidemiol., December 1, 2007; 36(6): 1371 - 1371. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||