Loading...
 
Toggle Health Problems and D

Using natural experimental studies to guide public health action, not RCTs – Nov 2019

Using natural experimental studies to guide public health action: turning the evidence-based medicine paradigm on its head

J Epidemiol Community Health, doi:10.1136/jech-2019- 213085
Additional material is published online only. To view please visit the journal online (http://dx.doi.org/10.1136/ jech-2019-21 3085).

VitaminDWiki

Some European countries added Vitamin D for a while, but then reduced it ==> problems noticed years later
Examples

Natural Experiment PRO
>100X more participants
Wide diversity of participants
>10X longer duration
Natural Experiment CON
Often 10X to 100X lower dose than RCT


Non-Vitamin D examples

When some US states permitted abortion ==> less crime 20 years later
When some US states legalized Marijuana ==> less crime in a few years

 Download the PDF from VitaminDWiki
David Ogilvie david.ogilvie at mrc-epid.cam. ac.uk ,1 Jean Adams,1 Adrian Bauman,2 Edward W. Gregg,3 Jenna Panter,1 Karen R. Siegel,4 Nicholas J. Wareham,1 Martin White1
1MRC Epidemiology Unit and Centre for Diet and Activity Research (CEDAR), University of Cambridge, Cambridge, UK
2Charles Perkins Centre and Prevention Research Collaboration, University of Sydney, Sydney, New South Wales, Australia
3School of Public Health, Imperial College, London, UK
4National Center for Chronic Disease Prevention and Health Promotion, Centers for Disease Control and Prevention, Atlanta, Georgia, USA

Despite smaller effect sizes, interventions delivered at population level to prevent non-communicable diseases generally have greater reach, impact and equity than those delivered to high-risk groups. Nevertheless, how to shift population behaviour patterns in this way remains one of the greatest uncertainties for research and policy. Evidence about behaviour change interventions that are easier to evaluate tends to overshadow that for population-wide and system-wide approaches that generate and sustain healthier behaviours. Population health interventions are often implemented as natural experiments, which makes their evaluation more complex and unpredictable than a typical randomised controlled trial (RCT). We discuss the growing importance of evaluating natural experiments and their distinctive contribution to the evidence for public health policy.
We contrast the established evidence-based practice pathway, in which RCTs generate 'definitive' evidence for particular interventions, with a practice-based evidence pathway in which evaluation can help adjust the compass bearing of existing policy. We propose that intervention studies should focus on reducing critical uncertainties, that non-randomised study designs should be embraced rather than tolerated and that a more nuanced approach to appraising the utility of diverse types of evidence is required. The complex evidence needed to guide public health action is not necessarily the same as that which is needed to provide an unbiased effect size estimate. The practice-based evidence pathway is neither inferior nor merely the best available when all else fails. It is often the only way to generate meaningful evidence to address critical questions about investing in population health interventions.

Clipped from PDF

Non-randomised study designs should be embraced rather than tolerated

Although natural experimental study designs have important theoretical underpinnings in common with the RCT, their worth does not reside solely in the extent to which they emulate an RCT design. Dunning proposes three criteria for assessing the utility of natural experimental studies.31 The first is that the allocation of an intervention can be treated ‘as if it were random, although not within a planned RCT. Although randomisation eliminates important sources of potential confounding, an expectation that intervention studies should entail a comparable allocation process (such as a lottery) would further entrench existing evaluative biases because many interventions relevant to public health are never likely to fulfil this criterion.22 This may be because randomisation is impractical (eg, new transport infrastructure is built in particular places for particular reasons) or politically unpalatable (if, eg, it is seen as withholding a service from certain areas or groups).12 Furthermore, intervention studies that ‘fail’ this criterion may pass with flying colours on Dunning’s other two criteria for utility. One of these relates to the relevance of the intervention to current, real- world policy questions. A key advantage of natural experimental studies is that they ‘do not interfere in the natural data generation process’,32 and thereby largely avoid the problems of ‘artificial and less directly informative’ inferences from effects observed in experimental studies in more controlled settings.33 The other criterion relates to the plausibility of the causal inference.25 Here again, a natural experimental study may be ‘more likely to generate causal evidence that applies to intervention implementation in real life’,34 particularly if it elicits evidence of how an intervention achieves its effects.27

Of course, this may appear to sit uneasily within a research funding system based on a biomedical paradigm that privileges the RCT above all other methods for establishing effective- ness.35 But randomisation does not necessarily hold the key to unlocking questions about public health action.25 Nor does the proliferation of epidemiological studies that link environmental exposures with health behaviours in a statistically robust way but are incapable of testing whether altering the former influences the latter.21 36 If a given method or study design is chosen for its alignment with the applied research question and executed in a rigorous and transparent way, it is likely to contribute important evidence even though (and perhaps because) it falls into the implicitly disparaging category of ‘non-randomised’ studies.35

A more thoughtful approach to appraising the utility of evidence

This is not to deny that many non-randomised studies do have major limitations and are reported in ways that lack rigour or transparency. For example, systematic reviews of studies linking changes in the built environment with changes in diet, physical activity and adiposity have noted multiple potential sources of bias and that ‘studies with weaker designs were more likely to report associations in the positive direction’.23 37 In addition to all the issues that complicate the practice and interpretation of trials, in a natural experimental study close attention needs to be paid to understanding exactly what exposure to an intervention consists of; how an intervention comes to be assigned to some people, groups or areas and not others; finding a valid basis for estimating the counterfactual, such as by using a meaningful control group or a graded measure of intervention exposure; selecting and interpreting the adjustment for appropriate covariates to minimise the risk of confounding; and interpreting complex patterns within the outcomes, which may include divergent and potentially inequitable responses between subgroups, dose-response relationships and comparisons with multiple controls.18 38 39

We have well-established, and continually developing, catechisms for assessing the internal validity of intervention studies, and groups of studies, in health research.40 However, we lack clear consensus on the relative importance or interpretation of different aspects of internal validity in natural experimental studies, and therefore on how to make constructive use of an evidence base that fits poorly into existing appraisal systems.6 For example, current tools for assessing risk of bias appear predicated on a preference for studies that resemble an RCT as closely as possible.22 23 They tend to downplay or ignore the importance of ‘greater qualitative appraisal (and) theoretical and statistical knowledge’,32 and of what different quantitative and qualitative components of single or multiple studies might contribute in combination to a growing body of overall, more generalisable causal inference.31 33 In particular, we lack consensus on ‘how good is good enough’—which partly depends, of course, on the answer to the question ‘good enough for what?’ The complex evidence needed to guide public health action is not necessarily the same as that which is needed to provide an unbiased estimate of an effect size.

Conclusion

We are more likely to halt the rise in the global prevalence of non-communicable diseases by taking and evaluating new, more ambitious or radical actions to address the underlying causes than by merely applying existing preventive approaches—even if these are effective—with greater intensity. Even apparently simple questions about effectiveness in this arena cannot be answered without action—although based on the best available evidence at the time—that necessarily precedes evaluation. The practice-based evidence pathway can be regarded as an essential, and currently under-resourced and undervalued, complement to the more established evidence-based practice pathway. It is neither inferior nor merely the best available when all else fails. On the contrary, it is often the only way to generate meaningful evidence to address critical questions about investing in population health interventions.
The two pathways for generating evidence described in this paper do not represent mutually exclusive approaches. Some policy and practice innovations could and should be evaluated in RCTs, and many more would benefit from more planned evaluation using a wider range of study designs. Nevertheless, the public health research community and those who fund and publish their work have key roles to play in supporting the development and credibility of researchers in this field, and the more thoughtful conduct, appraisal and synthesis of natural experimental studies, to populate critical missing pieces of the evidence base to support more effective public health action.


Created by admin. Last Modification: Thursday November 21, 2019 23:16:01 GMT-0000 by admin. (Version 5)