Database selection in systematic reviews: an insight through clinical neurology
Vasser M et al. Health Info Libr J, 34: 156–164.
Unfortunately, it’s behind a paywall, so here’s the abstract:
Failure to perform a comprehensive search when designing a systematic review (SR) can lead to bias, reducing the validity of review’s conclusions.
We examined the frequency and choice of databases used by reviewers in clinical neurology.
Ninety-five SRs and/or meta-analyses were located across five prominent neurology journals between 2008 and 2014. Methods sections were reviewed, and all bibliographic databases were coded.
On average, 2.59 databases were used in SR searches. Seven reviews included an information specialist, and these reviews reported a greater number of information sources used during the search process. Thirty-nine databases were reported across studies. PubMed/MEDLINE® and EMBASE were cited most frequently.
Searching too few databases may reduce the validity and generalisability of SR results. We found that the majority of systematic reviewers in clinical neurology do not search an adequate number of databases, which may yield a biased sample of primary studies and, thus, may influence the accuracy of summary effects.
Systematic reviewers should aim to search a sufficient number of databases to minimise selection bias. Additionally, systematic reviewers should include information specialists in designing SR methodology, as this may improve systematic review quality.
I’d love to see the evidence behind some of the statements. For instance, what is the evidence that the majority of SRs do not search an adequate number of databases? That presumes there is an adequate number – have I missed something?
Also, people tend to look at increasing the number of databases (or any other methodological burden) and easily throw it out there with a disregard for the potential harm. One could easily increase the cost for no gain (or marginal gain) – ethical or unethical?
Bias is bad, but waste is also bad. Doing one review with small bias, is that ‘better’ than ten reviews with a bit more bias?
No idea of the answer or how you’d answer it. But in the absence of evidence the prevailing, faith-based, response gets adopted by the uncritical. Surely, science is about scepticism – that works both ways.