Using Negative Control Outcomes to Identify Unmeasured Confounding in a Self-Controlled Risk Interval Study Evaluating the COVID-19 Vaccine Myocarditis Safety Signal

Background: The self-controlled risk interval (SCRI) design lent itself well to the rapid safety evaluation of COVID-19 vaccines, but limited post-vaccination follow-up necessitated a pre-vaccination control window. This increases the risk of time-varying confounding especially for later doses. Negative control outcomes (NCOs) are often used to assess unmeasured confounding, but they are rarely applied to SCRIs.

Objectives: Analyse two NCOs to assess unmeasured confounding in a COVID-19 vaccine safety SCRI on myocarditis.

Methods: We used primary and secondary healthcare data from Italian (ARS), Spanish (BIFAP, FISABIO, SIDIAP) and British (CPRD) datasources (DS). Exposed cases with ≥1 year of follow-up before 1 September 2020 (study start) were eligible. Data were extracted from study start to the end of data availability (December 2021 (ARS) - April 2022 (BIFAP)). Exposures were doses 1/2 of the Pfizer (PF), Moderna (MO), AstraZeneca (AZ), and Janssen (JJ) COVID-19 vaccines. Outcomes were myocarditis (MC), otitis externa (OE; NCO), and non-congenital valvular heart disease (VHD; NCO). These NCOs were chosen because their risk factors were similar to MC except for the association with COVID-19 vaccines.
The SCRI used a 60-day control window starting 90 days before dose 1 and dose-specific 28-day risk windows. Analyses were stratified by vaccine brand, adjusted for calendar time in 30-day periods, and pooled across DS with random effects meta-analysis. We expect a null association between exposure and NCOs and interpret deviations from this as potential unmeasured confounding.

Results: We included 461 MC cases (29% women), 40,732 OE cases (58% women), and 17,187 VHD cases (51% women). MC risk increased significantly after dose 2 of PF (IRR = 1.85, 95%CI 1.32-2.60, I2 = 0%) and MO (IRR = 2.39, 95%CI 1.40-4.10, I2 = 0%). Vaccination did not affect OE risk with little heterogeneity between DS, except for JJ (IRR = 1.26, 95%CI 0.99-1.60, I2 = 42%). We also observed heterogeneity for AZ dose 1 (I2 = 69%) because of an isolated finding in ARS (IRR = 4.80, 95%CI 1.62-14.2). VHD risk decreased after vaccination but there was considerable heterogeneity between DS (I2 = 0% for JJ to I2 =93% for PF dose 2). The strongest effects were seen in the DS relying entirely (ARS) or partially (FISABIO) on hospital records for diagnoses.

Conclusions: The VHD results suggest confounding towards the null, underestimating MC risk, but the null finding for OE does not support this. We interpret this as VHD being a poor NCO; it could be a cardiac complication of COVID-19 and thus vaccines protect against it. The degree of unmeasured confounding also varies geographically and across DS. NCOs are a useful tool to assess weakensses in COVID-19 vaccine safety SCRIs.