Framework for Assessing the Reproducibility of Observational Comparative Effectiveness Research: Direct Oral Anticoagulants and Ischemic or Hemorrhagic Events Case Study

Background: Evidence based medicine often must rely on a single study addressing a clinical question with little consideration of how those results generalize to different populations, estimators, and assumptions. Further, observational studies are fraught with confounding due to the lack of true randomization. Therefore, a framework is needed for a systematic assessment of the reliability of evidence against several different study and population dimensions.

Objectives: Apply a systematic framework to assess the reliability of a recently published observational study on the association of rivaroxaban and apixaban with major ischemic or hemorrhagic events in patients with atrial fibrillation (Ray et al).

Methods: The Ray et al. study was first replicated in IQVIA Open Claims data as our closest equivalent of the original Medicare data, as a new-user active-comparator cohort design. Then, the generalizability of the replication was evaluated in the other data sources: IQVIA Hospital charge master, Department of Veterans Affairs (VA), Stanford and Columbia data converted to the OMOP CDM. Cox proportional hazards models were used to estimate the hazard ratio (HR) and the 95% confidence interval (CI). Confounding was accounted for using 1:1 large-scale propensity score matching. The robustness of association was assessed through a series of sensitivity analyses: 1) change in the definitions of exposure and outcomes, 2) empirical calibration of the effect estimates (negative control outcomes), 3) change in time-at-risk and confounder adjustment techniques.

Results: Almost 1M patients with atrial fibrillation were new users of either drug between 2013 and 2018 (Columbia 2,736, Hospitals 15,497, Open Claims 868,619, Stanford 2,853, VA 77,365), with similar comorbidities and age distributions as those in Ray. Study diagnostics showed high overlap between rivaroxaban and apixaban patients with respect to their baseline characteristics and no notable imbalance after propensity score matching. The HRs were in the same direction with Ray’s and CIs were overlapping for all estimated outcomes, suggesting a successful replication of study findings: hemorrhagic stroke 1.17 (1.00-1.37), ischemic stroke 1.11 (1.08, 1.13), intracranial bleed 1.03 (0.99, 1.08) and extracranial bleed 1.25 (1.23, 1.27). Sensitivity analyses had similar results. The negative controls did not reveal substantial residual systematic error. Effect sizes were in the same direction for most outcomes and the CIs overlapped with Ray’s across sites, indicating generalizability of the results.

Conclusions: Our reproducibility framework confirmed the findings of the Ray et al study and proved to be an effective tool for increasing the robustness of published observational research.