At the heart of evidence synthesis is a fundamental question which seems unresolved and rarely articulated: Of the evidence produced in trials, for a given intervention, how much do you need to produce a fit-for-purpose evidence review?

In my more naive years I assumed systematic reviews got all the evidence. Alas, that is clearly not the case. Therefore, if its not all the evidence it’s a sample. Alas, this sample has been characterised (by no less than Glasziou and Chalmers) as a biased subsample (full quote: “Reviewers trying to summarize all the research addressing a particular question are limited by access only to a biased subsample of what has been done.”) This is based on the knowledge that 50% of trials are never published (and likely missed by most systematic reviews).

Another related point is that, as well as not finding all the trials, the systematic reviewers don’t get all the data (typically, just the data captured in the 6-12 page journal article). The problem with this is well described and has – again – be described, this time in a blog by TranspariMED “Access to information on medicines under threat in Europe “..which reports “a German team of experts found that CSRs provided over twice as much information on patient relevant outcomes than all other publicly available sources combined.” Previously the distinction between journal articles and fuller, clinical study reports, has been described and can be massive (see Clinical study reports of randomised controlled trials: an exploratory review of previously confidential industry reports)

I’ve tried to capture this ‘evidence capture’ graphically:

I can understand how this situation has come about. In the early days of the systematic review world (thinking about the time Cochrane started) awareness of such issues as reporting bias, clinical study reports was a fraction of what it is now. At the time it might well have been thought that a search of a number of all databases would capture ALL the evidence (a slight aside, earlier Cochrane definitions of a systematic reviews included reference to capturing ALL the trial evidence – this has now been watered down considerably to “...attempts to identify, appraise and synthesize all the empirical evidence“).

Alas, we know more now. Hence the question of sampling becomes important. So, to repeat:

For an evidence synthesis how much of the evidence produced in trials is needed?

Get too much and it’s arguable wasteful (wasted resource gold-plating a review)

Get too little and it’s wasteful (waster resource producing a pointless review).

Paraphrasing Goldilocks, how much evidence synthesis is “just right”?