C. C. Chrystoja1,2, N. Baxter1,4, C. Bell1,5, G. Tomlinson1,2, D. Urbach1,2,3 1University of Toronto,Institute Of Health Policy, Management And Evaluation,Toronto, Ontario, Canada 2University Health Network,Toronto, Ontario, Canada 3Women’s College Hospital,Toronto, Ontario, Canada 4St. Micheal’s Hospital,Toronto, Ontario, Canada 5Mount Sinai Hospital,Toronto, ON, Canada
Introduction: Poor-quality studies have frequently led to the adoption by mainstream medicine of surgical techniques and devices that were, at a later date, found to be faulty and devastating in their adverse outcomes for patients. There are many examples where clinicians, healthcare decision-makers, and regulators were unaware of the true risks of emergent medical technologies because no high-quality randomised studies evaluated the risks of the procedure and existing non-randomised studies were systematically biased. Our objective was to determine the extent to which study attributes associated with bias influence effect estimates in non-randomised studies.
Methods: We selected three case examples in different clinical areas, with representation of surgical procedures, medical devices, and drug therapy, including: (1) off-pump versus on-pump coronary artery bypass grafting, (2) mesh-augmented versus native tissue pelvic organ prolapse repair, and (3) hormone replacement therapy for postmenopausal women. MEDLINE was searched to identify non-randomised comparative studies and related systematic reviews, whose references were examined to identify additional studies. Study attributes were extracted using the INFORM tool, a quality assessment instrument we developed based on a conceptual framework of bias, review of existing instruments, consultation with experts, and an interactive process piloted with users of varying epidemiological backgrounds. Fixed-effects meta-analyses were separately performed for each outcome of interest: reintervention and stroke (for case #1), mesh erosion and incontinence (for case #2), and all-cause death and cardiovascular events (for case #3). Meta-regression was used to explore the effect of study attributes on outcomes.
Results: A total of 96 non-randomised studies were identified for case #1, 30 studies for case #2, and 39 studies for case #3. In case #1, 22 studies with 85,542 participants evaluated reintervention as an outcome and 82 studies with 698,776 participants assessed stroke. Sixteen studies with 2,154 participants evaluated mesh erosion in case #2 and 13 studies with 7,409 participants examined incontinence. In case #3, 12 studies with 514,872 participants looked at all-cause death and 17 studies with 262,329 participants assessed cardiovascular events. See Figure 1 for details.
Conclusion: Studies that did not match in the design phase showed a greater treatment benefit for off-pump coronary artery bypass grafting and hormone replacement therapy compared to studies that used a systematic approach. However, the other study attributes were associated with different direction and magnitude of treatment effects.
