I’m increasingly drawn to the concept of what is ‘correct’ in relation to systematic reviews and meta-analyses.  If we’re wanting to demonstrate the worth of a ‘rapid’ reviews we need to compare it to a ‘gold standard’ ‘correct’ answer.  There does not appear to be such a thing.

I’m particularly drawn to the 2014 Deschartres paper that:

“…compare treatment outcomes estimated by meta-analysis of all trials and several alternative analytic strategies: single most precise trial (ie, trial with the narrowest confidence interval), meta-analysis restricted to the 25% largest trials, limit meta-analysis (a meta-analysis model adjusted for small-study effect), and meta-analysis restricted to trials at low overall risk of bias.”

So, same question, different samples of trials (one of which was ‘all’ trials) and different results.  The authors conclude:

“Estimation of treatment outcomes in meta-analyses differs depending on the strategy used. This instability in findings can result in major alterations in the conclusions derived from the analysis and underlines the need for systematic sensitivity analyses.”

They further give practical considerations (heuristics?):

“Estimation of treatment outcomes in meta-analyses differs depending on the strategy used. This instability in findings can result in major alterations in the conclusions derived from the analysis and underlines the need for systematic sensitivity analyses.”

An important topic and some interesting papers:

• McMaster Premium LiteratUre Service (PLUS) performed well for identifying new studies for updated Cochrane reviews Hemens BJ, Haynes RB. J Clin Epidemiol. 2012 Jan;65(1):62-72.e1
• The Impact of Study Size on Meta-analyses: Examination of Underpowered Studies in Cochrane Reviews. Turner RM et al. PLoS ONE 2013. 8(3): e59202
• A pragmatic strategy for the review of clinical evidence Sagliocca L et al. J Eval Clin Pract. 2013 Aug;19(4):689-96.
• Influence of trial sample size on treatment effect estimates: meta-epidemiological study. Dechartres A et al. BMJ 2013;346:f2304
• Association Between Analytic Strategy and Estimates of Treatment Outcomes in Meta-analyses. Dechartres A et al. JAMA. 2014;312(6):623-630.
• How systematic reviews cause research waste. Roberts I et al. Correspondance. The Lancet. Vol 386 October 17, 2015
• In defence of reviews of small trials: underpinning the generation of evidence to inform practice[editorial]. Handoll HHG et al. Cochrane Database of Systematic Reviews 2015;(11)