Background: Adverse event (AE) signal detection traditionally performed via analysis of spontaneous reporting systems has important limitations. Use of longitudinal real-world data (RWD) for signal detection as a component of comprehensive safety surveillance offers the opportunity to improve and complement current approaches.
Objectives: To assess the performance of US longitudinal claims data and FDA AE Reporting System (FAERS) data against a gold standard using different methods for hypothesis-free signal detection.
Methods: The case drug for this pilot study was injectable insulin glargine (Lantus). PharMetrics claims data (October 2016 - June 2020) was used to run signal detection using TreeScan, a publicly available software enabling automated tree-based scan statistics on hierarchical data. A new-user cohort of Lantus patients was derived in PharMetrics, and three specific designs were applied: a self-controlled 90-day pre/post design, a 90-day self-controlled Tree-based temporal scan, and a 1:1 propensity-score (PS) matched design. For the PS matched design, new Lantus patients were matched to new users of Neutral Protamine Hagedorn (NPH) insulin. For signal detection using FAERS, 3 different metrics and thresholds were used: the EBGM ≥4.0, EB05 ≥ 2.0 and the composite metrics with PRR ≥ 2 and N ≥ 3 and PRR CHISQ ≥ 4. For signal detection using PharMetrics and TreeScan, a signaling threshold p< 0.05 was used. For both approaches, the gold standard was the list of known AEs derived from the Lantus product label. A random sample of 200 negative control AEs not reported with Lantus in FAERS (representing a total of 293 gold standard events) and 200 ICD codes in PharMetrics were selected (representing a total of 241 events). Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for each approach were calculated.
Results: Among the 3 designs used for signal detection in PharMetrics, the best performance was reached using the Tree-temporal scan analysis. Three gold standard events were detected (sensitivity: 7.3%; specificity: 99.0%; PPV: 60.0%; NPV: 83.9%) at p< 0.05. For the others designs, no gold-standard events were detected. In the analysis using FAERS data, sensitivity varied between 25-38%, specificity: 92.5%-97.5%, PPV: 68-82%, and NPV: 73.5%-75.2%.
Conclusions: The performances of 3 different approaches to signal detection using TreeScan on PharMetrics and EBGM and PRR on FAERS data against a reference standard provided varying results. This variation may be explained by fundamental differences in the type of data used as well as key disparities in the designs of the approaches.
