ReviewMethods to assess intended effects of drug treatment in observational studies are reviewed
Introduction
In the evaluation of intended effects of drug therapies, well-conducted randomized controlled trials (RCTs) have been widely accepted as the scientific standard [1]. The key component of RCTs is the randomization procedure, which allows us to focus on only the outcome variable or variables in the different treatment groups in assessing an unbiased treatment effect. Because adequate randomization will assure that treatment groups will differ on all known and unknown prognostic factors only by chance, probability theory can easily be used in making inferences about the treatment effect in the population under study (confidence intervals, significance). Proper randomization should remove all kinds of potential selection bias, such as physician preference for giving the new treatment to selected patients or patient preference for one of the treatments in the trial [2], [3]. Randomization does not assure equality on all prognostic factors in the treatment groups, especially with small sample sizes, but it assures confidence intervals and P-values to be valid by using probability theory [4].
There are settings where a randomized comparison of treatments may not be feasible due to ethical, economic or other constraints [5]. Also, RCTs usually exclude particular groups of patients (because of age, other drug usage or noncompliance); are mostly conducted under strict, protocol-driven conditions; and are generally of shorter duration than the period that drugs are used in clinical practice [6], [7]. Thus, RCTs typically provide evidence of what can be achieved with treatments under the controlled conditions in selected groups of patients for a defined period of treatment.
The main alternatives are observational studies. Their validity for assessing intended effects of therapies has long been debated and remains controversial [8], [9], [10]. The recent example of the potential cardiovascular risk reducing effects of hormone replacement therapy (HRT) illustrates this controversy [11]. Most observational studies indicated that HRT reduces the risk of cardiovascular disease, whereas RCTs demonstrated that HRT increases cardiovascular risk [12]. The main criticism of observational studies is the absence of a randomized assignment of treatments, with the result that uncontrolled confounding by unknown, unmeasured, or inadequately measured covariates may provide an alternative explanation for the treatment effect [13], [14].
Along with these criticisms, many different methods have been proposed in the literature to assess treatment effects in observational studies. With all these methods, the main objective is to deal with the potential bias caused by the nonrandomized assignment of treatments, a problem also known as confounding [15].
Here we review existing methods that seek to achieve valid and feasible assessment of treatment effects in observational studies.
Section snippets
Design for observational studies
A first group of method of dealing with potential bias following from nonrandomized observational studies is to narrow the treatment and/or control group in order to create more comparable groups on one or more measured characteristics. This can be done by selection of subjects or by choosing a specific study design. These methods can also be seen as only a first step in removing bias, in which case further reduction of bias has to be attained by means of data-analytical techniques.
Data-analytical techniques
Another group of bias reducing methods are the data-analytical techniques, which can be divided into model-based techniques (regression-like methods) and methods without underlying model assumptions (stratification and matching).
Validations and sensitivity analyses
Horwitz et al. [54] proposed to validate observational studies by constructing a cohort of subjects in clinical practice that is restricted by the inclusion criteria of RCTs. Similarity in estimated treatment effects from the observational studies and the RCTs would provide empirical evidence for the validity of the observational method. Although this may be correct in specific situations [17], [55], it does not provide evidence for the validity of observational methods for the evaluation of
Summary and discussion
Although randomized clinical trials remain the gold standard in the assessment of intended effects of drugs, observational studies may provide important information on effectiveness under everyday circumstances and in subgroups not previously studied in RCTs. The main defect in these studies is the incomparability of groups, giving a possible alternative explanation for any treatment effect found. Thus, focus in such studies is directed toward adjustment for confounding effects of covariates.
References (63)
- et al.
Allocation concealment in randomised trials: defending against deciphering
Lancet
(2002) - et al.
Reliable assessment of the effects of treatment on mortality and major morbidity, II: observational studies
Lancet
(2001) - et al.
Evidence from randomised trials on the long-term effects of hormone replacement therapy
Lancet
(2002) - et al.
Randomized versus historical controls for clinical trials
Am J Med
(1982) - et al.
Randomised trials comparing chemotherapy regimens for advanced non-small cell lung cancer: biases and evolution over time
Eur J Cancer
(2003) - et al.
A case–control evaluation of treatment efficacy: the example of magnesium sulfate prophylaxis against eclampsia in patients with preeclampsia
J Clin Epidemiol
(1997) - et al.
A nested case–control study of the effectiveness of screening for prostate cancer: research design
J Clin Epidemiol
(2001) - et al.
Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates
J Clin Epidemiol
(1995) - et al.
A simulation study of the number of events per variable in logistic regression analysis
J Clin Epidemiol
(1996) - et al.
Effect of lipid-lowering therapy on early mortality after acute coronary syndromes: an observational study
Lancet
(2001)
Uses of ecologic studies in the assessment of intended treatment effects
J Clin Epidemiol
Developing improved observational methods for evaluating therapeutic effectiveness
Am J Med
Comparison of predictions based on observational data with the results of randomized controlled clinical trials of coronary artery bypass surgery
J Am Coll Cardiol
Fundamentals of clinical trials
Why transition from alternation to randomisation in clinical trials was made
BMJ
The value of randomization and control in clinical trials
Stat Med
Current problems and future challenges in randomized clinical trials
Circulation
The exclusion of the elderly and women from clinical trials in acute myocardial infarction
JAMA
Cardiovascular drugs: discrepancies in demographics between pre- and post-registration use
Eur J Clin Pharmacol
Methods in health services research. Interpreting the evidence: choosing between randomised and non-randomised studies
BMJ
Randomized, controlled trials, observational studies, and the hierarchy of research designs
N Engl J Med
Understanding the divergent data on postmenopausal hormone therapy
N Engl J Med
Case–control study, meta-analysis, and bouillabaisse: putting the calcium antagonist scare into context
Ann Intern Med
Confounding and indication for treatment in evaluation of drug treatment for hypertension
BMJ
Observational studies
Prospective cohort study to determine if trial efficacy of anticoagulation for stroke prevention in atrial fibrillation translates into clinical effectiveness
BMJ
Is drug treatment of hypertension in clinical practice as effective as in randomized controlled trials with regard to the reduction of the incidence of stroke?
Epidemiology
Identifying confounding by indication through blinded prospective review
Am J Epidemiol
The risk of myocardial infarction associated with antihypertensive drug therapies
JAMA
Antihypertensive drug therapies and the risk of ischemic stroke
Arch Intern Med
The case–crossover design: a method for studying transient effects on the risk of acute events
Am J Epidemiol
Cited by (156)
Advances in methodologies of negative controls: a scoping review
2024, Journal of Clinical EpidemiologyPredictors of seizure freedom, response and retention after 12 months of treatment with eslicarbazepine acetate: A post-hoc analysis of the Euro-Esli study
2021, Epilepsy ResearchCitation Excerpt :Real-world studies complement evidence from clinical trials by elucidating an ASM’s effectiveness when used under everyday clinical practice conditions (Ben-Menachem, 2005; Arzimanoglou et al., 2010). A major limitation of real-world observational studies is the presence of confounding factors and one of the main strategies to try and control the impact of confounding factors is the use of regression models (Pourhoseingholi et al., 2012; Klungel et al., 2004). Information on the patient characteristics associated with the likelihood of treatment response is particularly useful when considering an ASM as a potential treatment option; however, evidence for clinical predictors of response and retention with ASMs is currently limited (Bonnett et al., 2012; Sarkis et al., 2017).
Conditional Financing of Drugs in the Netherlands: Past, Present, and Future—Results From Stakeholder Interviews
2019, Value in HealthCitation Excerpt :Furthermore, numerous articles refer to the methodological difficulties associated with analyzing RWD and using RWE in decision making.24–26 From a methodological perspective, many advances have been made in the analysis of RWD, both alone or in combination with randomized controlled trial data.27–29 Moreover, recent guidelines issued by the combined efforts of the International Society for Pharmacoeconomics and Outcomes Research and the International Society for Pharmacoepidemiology provide an example of clear guidance on good procedures for the conduct and reporting of real-world studies to increase decision makers' confidence in RWE.30,31
Confounding adjustment in observational studies on cardiothoracic interventions: a systematic review of methodological practice
2023, European Journal of Cardio-thoracic Surgery