Review
Methods to assess intended effects of drug treatment in observational studies are reviewed

https://doi.org/10.1016/j.jclinepi.2004.03.011Get rights and content

Abstract

Background and objective

To review methods that seek to adjust for confounding in observational studies when assessing intended drug effects.

Methods

We reviewed the statistical, economical and medical literature on the development, comparison and use of methods adjusting for confounding.

Results

In addition to standard statistical techniques of (logistic) regression and Cox proportional hazards regression, alternative methods have been proposed to adjust for confounding in observational studies. A first group of methods focus on the main problem of nonrandomization by balancing treatment groups on observed covariates: selection, matching, stratification, multivariate confounder score, and propensity score methods, of which the latter can be combined with stratification or various matching methods. Another group of methods look for variables to be used like randomization in order to adjust also for unobserved covariates: instrumental variable methods, two-stage least squares, and grouped-treatment approach. Identifying these variables is difficult, however, and assumptions are strong. Sensitivity analyses are useful tools in assessing the robustness and plausibility of the estimated treatment effects to variations in assumptions about unmeasured confounders.

Conclusion

In most studies regression-like techniques are routinely used for adjustment for confounding, although alternative methods are available. More complete empirical evaluations comparing these methods in different situations are needed.

Introduction

In the evaluation of intended effects of drug therapies, well-conducted randomized controlled trials (RCTs) have been widely accepted as the scientific standard [1]. The key component of RCTs is the randomization procedure, which allows us to focus on only the outcome variable or variables in the different treatment groups in assessing an unbiased treatment effect. Because adequate randomization will assure that treatment groups will differ on all known and unknown prognostic factors only by chance, probability theory can easily be used in making inferences about the treatment effect in the population under study (confidence intervals, significance). Proper randomization should remove all kinds of potential selection bias, such as physician preference for giving the new treatment to selected patients or patient preference for one of the treatments in the trial [2], [3]. Randomization does not assure equality on all prognostic factors in the treatment groups, especially with small sample sizes, but it assures confidence intervals and P-values to be valid by using probability theory [4].

There are settings where a randomized comparison of treatments may not be feasible due to ethical, economic or other constraints [5]. Also, RCTs usually exclude particular groups of patients (because of age, other drug usage or noncompliance); are mostly conducted under strict, protocol-driven conditions; and are generally of shorter duration than the period that drugs are used in clinical practice [6], [7]. Thus, RCTs typically provide evidence of what can be achieved with treatments under the controlled conditions in selected groups of patients for a defined period of treatment.

The main alternatives are observational studies. Their validity for assessing intended effects of therapies has long been debated and remains controversial [8], [9], [10]. The recent example of the potential cardiovascular risk reducing effects of hormone replacement therapy (HRT) illustrates this controversy [11]. Most observational studies indicated that HRT reduces the risk of cardiovascular disease, whereas RCTs demonstrated that HRT increases cardiovascular risk [12]. The main criticism of observational studies is the absence of a randomized assignment of treatments, with the result that uncontrolled confounding by unknown, unmeasured, or inadequately measured covariates may provide an alternative explanation for the treatment effect [13], [14].

Along with these criticisms, many different methods have been proposed in the literature to assess treatment effects in observational studies. With all these methods, the main objective is to deal with the potential bias caused by the nonrandomized assignment of treatments, a problem also known as confounding [15].

Here we review existing methods that seek to achieve valid and feasible assessment of treatment effects in observational studies.

Section snippets

Design for observational studies

A first group of method of dealing with potential bias following from nonrandomized observational studies is to narrow the treatment and/or control group in order to create more comparable groups on one or more measured characteristics. This can be done by selection of subjects or by choosing a specific study design. These methods can also be seen as only a first step in removing bias, in which case further reduction of bias has to be attained by means of data-analytical techniques.

Data-analytical techniques

Another group of bias reducing methods are the data-analytical techniques, which can be divided into model-based techniques (regression-like methods) and methods without underlying model assumptions (stratification and matching).

Validations and sensitivity analyses

Horwitz et al. [54] proposed to validate observational studies by constructing a cohort of subjects in clinical practice that is restricted by the inclusion criteria of RCTs. Similarity in estimated treatment effects from the observational studies and the RCTs would provide empirical evidence for the validity of the observational method. Although this may be correct in specific situations [17], [55], it does not provide evidence for the validity of observational methods for the evaluation of

Summary and discussion

Although randomized clinical trials remain the gold standard in the assessment of intended effects of drugs, observational studies may provide important information on effectiveness under everyday circumstances and in subgroups not previously studied in RCTs. The main defect in these studies is the incomparability of groups, giving a possible alternative explanation for any treatment effect found. Thus, focus in such studies is directed toward adjustment for confounding effects of covariates.

References (63)

  • S.W. Wen et al.

    Uses of ecologic studies in the assessment of intended treatment effects

    J Clin Epidemiol

    (1999)
  • R.I. Horwitz et al.

    Developing improved observational methods for evaluating therapeutic effectiveness

    Am J Med

    (1990)
  • M.A. Hlatky et al.

    Comparison of predictions based on observational data with the results of randomized controlled clinical trials of coronary artery bypass surgery

    J Am Coll Cardiol

    (1988)
  • L.M. Friedman et al.

    Fundamentals of clinical trials

    (1996)
  • I. Chalmers

    Why transition from alternation to randomisation in clinical trials was made

    BMJ

    (1999)
  • P. Urbach

    The value of randomization and control in clinical trials

    Stat Med

    (1993)
  • A.R. Feinstein

    Current problems and future challenges in randomized clinical trials

    Circulation

    (1984)
  • J.H. Gurwitz et al.

    The exclusion of the elderly and women from clinical trials in acute myocardial infarction

    JAMA

    (1992)
  • N.F. Wieringa et al.

    Cardiovascular drugs: discrepancies in demographics between pre- and post-registration use

    Eur J Clin Pharmacol

    (1999)
  • M. McKee et al.

    Methods in health services research. Interpreting the evidence: choosing between randomised and non-randomised studies

    BMJ

    (1999)
  • J. Concato et al.

    Randomized, controlled trials, observational studies, and the hierarchy of research designs

    N Engl J Med

    (2000)
  • F. Grodstein et al.

    Understanding the divergent data on postmenopausal hormone therapy

    N Engl J Med

    (2003)
  • F.H. Messerli

    Case–control study, meta-analysis, and bouillabaisse: putting the calcium antagonist scare into context

    Ann Intern Med

    (1995)
  • D.E. Grobbee et al.

    Confounding and indication for treatment in evaluation of drug treatment for hypertension

    BMJ

    (1997)
  • P.R. Rosenbaum

    Observational studies

    (2002)
  • L. Kalra et al.

    Prospective cohort study to determine if trial efficacy of anticoagulation for stroke prevention in atrial fibrillation translates into clinical effectiveness

    BMJ

    (2000)
  • O.H. Klungel et al.

    Is drug treatment of hypertension in clinical practice as effective as in randomized controlled trials with regard to the reduction of the incidence of stroke?

    Epidemiology

    (2001)
  • S.C. Johnston

    Identifying confounding by indication through blinded prospective review

    Am J Epidemiol

    (2001)
  • B.M. Psaty et al.

    The risk of myocardial infarction associated with antihypertensive drug therapies

    JAMA

    (1995)
  • O.H. Klungel et al.

    Antihypertensive drug therapies and the risk of ischemic stroke

    Arch Intern Med

    (2001)
  • M. Maclure

    The case–crossover design: a method for studying transient effects on the risk of acute events

    Am J Epidemiol

    (1991)
  • Cited by (156)

    • Predictors of seizure freedom, response and retention after 12 months of treatment with eslicarbazepine acetate: A post-hoc analysis of the Euro-Esli study

      2021, Epilepsy Research
      Citation Excerpt :

      Real-world studies complement evidence from clinical trials by elucidating an ASM’s effectiveness when used under everyday clinical practice conditions (Ben-Menachem, 2005; Arzimanoglou et al., 2010). A major limitation of real-world observational studies is the presence of confounding factors and one of the main strategies to try and control the impact of confounding factors is the use of regression models (Pourhoseingholi et al., 2012; Klungel et al., 2004). Information on the patient characteristics associated with the likelihood of treatment response is particularly useful when considering an ASM as a potential treatment option; however, evidence for clinical predictors of response and retention with ASMs is currently limited (Bonnett et al., 2012; Sarkis et al., 2017).

    • Conditional Financing of Drugs in the Netherlands: Past, Present, and Future—Results From Stakeholder Interviews

      2019, Value in Health
      Citation Excerpt :

      Furthermore, numerous articles refer to the methodological difficulties associated with analyzing RWD and using RWE in decision making.24–26 From a methodological perspective, many advances have been made in the analysis of RWD, both alone or in combination with randomized controlled trial data.27–29 Moreover, recent guidelines issued by the combined efforts of the International Society for Pharmacoeconomics and Outcomes Research and the International Society for Pharmacoepidemiology provide an example of clear guidance on good procedures for the conduct and reporting of real-world studies to increase decision makers' confidence in RWE.30,31

    View all citing articles on Scopus
    View full text