How to underpower your real-world data study: examples and ways to avoid them using quantitative bias analysis

Background: We often apply sample size and power calculations to real-world data (RWD) in the same manner as trials, but without the operational control as present in trials, the results are subject to greater information bias. Some sources of bias may have little impact on the effect estimate, but they will result in greater uncertainty and potentially in failure to reach desired power if not accounted for while planning. This is especially true for small target populations or rare events.

Objectives: To identify sources of information bias in RWD that result in loss of power, to illustrate the impact in a simple example with survival analysis, and to demonstrate the use of quantitative bias analysis (QBA) to estimate the bias-adjusted sample size.

Methods: We reviewed literature to identify sources of information bias in RWD studies. Selecting one example, we present the approach and findings of applying QBA in sample size calculations in the presence of 5%, 20%, and 30% false-negative rate (outcome misclassification) in both exposed and unexposed in a 1:1 matched survival analysis to achieve 90% power (5% type 1 error). Assuming a hazard ratio of 0.3 and an event rate of 0.004 per person-month among the unexposed, we estimated sample size ignoring the outcome misclassification, then estimated bias-adjusted sample sizes. We repeated the exercise assuming a false-negative rate differential with regard to exposure of 20% and 10% in the exposed and unexposed groups respectively, and vice versa.

Results: Application of QBA showed that in the presence of outcome misclassification reaching 90% power for the matched design required larger sample sizes. Moreover, a simple intuitive increase by the proportion of misclassification would be insufficient. Compared to the estimated sample size required without consideration of misclassification, non-differential outcome misclassification of 5%, 20% and 30% were seen to require an increase in sample size of 5.2%, 25% and 42.9%, respectively, to achieve the same level of power. For outcome misclassification differential with respect to the exposure, higher levels of misclassification in the study arm with fewer events was associated with greater loss of power. Outcome misclassification of 20% in the exposed and 10% in the unexposed required 21.5% increase in sample size, whereas the converse required a 14% increase to achieve 90% power.

Conclusions: Loss of power associated with non-differential outcome misclassification can be substantial. QBA provides a useful framework to quantify the impact of information bias on the effect estimate in the analysis, but also to incorporate estimated bias into sample size calculations. This can be easily extended to other types of information bias.