Appraisal of: "Arber M, Glanville J, Isojarvi J, Baragula E, Edwards M, Shaw A, Wood H. Which databases should be used to identify studies for systematic reviews of economic evaluations? Int J Technol Assess Health Care. 2018 Jan;34(6):547-54."

Reviewer(s): 
David Kaunelis
Kelly Farrah
Short description: 

NHS EED and HEED, two key databases for retrieving health economics information, have closed. Based on this changed landscape, this article assesses which databases are now the best sources of information for retrieving economic evaluations to inform systematic reviews. The authors built a quasi-gold standard database of 351 records compiled from 46 systematic reviews of economic evaluations. Nine databases were searched for each record. Embase had the highest yield (89%), followed by Scopus (84%) and MEDLINE and PubMed (both 81%). The HTA database identified the highest number of unique citations (13/351). Embase also uniquely identified two conference abstracts, an important consideration if this type of material is eligible for inclusion in a review. All nine database combined retrieved 337/351 (96%) records. Searching a combination of Embase, the HTA database and either PubMed or MEDLINE identified 95% of the quasi-gold standard records (333/351). The authors concluded that searching additional database outside the core group may be inefficient because of limited incremental yield. Searchers should not rely on PubMed or MEDLINE alone. Searching a multi-disciplinary database may also be useful, especially for non-clinical or public health topics. The authors conclude that searchers should focus on developing suitable search strategies in these key databases to ensure high sensitivity and adequate precision. Supplementary search techniques such as grey literature searching may be more efficient than searching a larger number of databases, as 14/351 (4%) citations were not identified in any of the databases.

MEDLINE search strategies reported in source systematic reviews were also assessed. 10/29 (34.5%) of re-run search strategies missed at least one of the included records found in MEDLINE (with 25 citations missed in total). Weaknesses in the population or intervention concepts, rather than the economics concept, were identified as negatively impacting search retrieval.

Limitations stated by the author(s): 

Candidate reviews were screened by one reviewer only, unless a clear inclusion decision could not be made.

Researchers may have access to more than the nine databases chosen for this review. Other databases were not tested.

The quasi-gold standard database used in this project was compiled from reports commissioned or carried out by a single entity (NICE) and comprised 351 economic evaluations. A larger reference set that included evaluations from more than one entity would have increased generalizability of the findings. Reports focused on clinical medicine or public health in the context of the United Kingdom, so may be less generalizable for other topics such as mental health or health management, or for healthcare in low- and middle-income countries.

The quality of the searches conducted by the systematic reviews from which the quasi-gold standard database was harvested was not assessed. It was assumed that the searches conducted were of sufficient quality to provide a representative set of economic evaluations. Weaknesses in these search strategies could have failed to retrieve relevant economic evaluations, perhaps reducing the representativeness of the quasi-gold standard.

“Efficient” was defined as the fewest number of databases that could be combined to retrieve that largest number of citations. Other types of efficiency such as the time needed to search a database or the number of irrelevant citations retrieved were not considered. Many databases are available on multiple platforms which can impact retrieval and efficiency.

For the evaluation of MEDLINE searches, the sensitivity of each of review’s search strategy in MEDLINE was used as a proxy for search quality. There are other methods available to assess the quality of searches, such as the PRESS checklist, or how the search strategy performs when translated to other databases.

Limitations stated by the reviewer(s): 
While not a particular limitation of this study, the authors do note that the HTA Database is not currently being updated, which may hamper retrieval of recently published studies from the producers of HTA. Until the database is updated, searchers should identify this material by searching websites of individual HTA agencies, or through a general web search. The authors do not mention which databases were searched in the reports used to generate the quasi-gold standard. Since all economic evaluations were drawn from a single organization (NICE), it is possible that they share similar search methodologies, and, thus may search a standard list of databases. If not all of the nine databases were searched for these reports, it may have biased estimates of database yield.
Study Type: 
Single study
Supplemental publications to the study: 
Which information sources should be used to identify studies for systematic reviews of economic evaluations in healthcare?
Tags:
  • Share this