Databases

Scope: provide the database name(s); add to records where a database rather than a website or interface is discussed

Appraisal of: Thielen FW, Van Mastrigt G, Burgers LT, Bramer WM, Majoie H, Evers S, Kleijnen J. How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3). Exp

Short description: 

This article is the second in a three-part series on how to prepare a systematic review of economic evaluations. It provides a good overview of the literature on how to select relevant databases and develop a search strategy for retrieving economic evaluations. While the main target audience of the article is developers of clinical practice guidelines, the process described is helpful for systematic review researchers as well as those undertaking health technology assessments. The authors identify four steps in the search process: (1) selecting relevant databases (basic, specific and optional); (2) developing a comprehensive search strategy; (3) performing the searches (including documentation), and (4) selecting the relevant studies. The authors note the recent discontinuation of two health economics databases (HEED and NHS EED), which has resulted in an increased reliance on the use of search filters designed to capture economic evaluations. Also noted are unsettled issues, such as the lack of consensus on how many and which databases should be searched, as well a lack of uniform guidance on the methodology of developing a sound search strategy. Validated search filters and automated processes may help to overcome problems created by the lack of health economics-specific databases.

Limitations stated by the author(s): 

No limitations stated by the study authors.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Review

Appraisal of: Wood H, Arber M, Glanville JM. Systematic reviews of economic evaluations: how extensive are their searches? Int J Technol Assess Health Care. 2017 Mar 27:1-7.

Reviewer(s): 
Short description: 

This study evaluated the search methodology of recent systematic reviews of economic evaluations. A sample of 42 reviews identified through a MEDLINE search was analyzed. The analysis included: databases searched (general & specialist), health technology assessment sources searched, and supplementary search techniques used. The search approaches used in the systematic reviews were compared to two current recommendations: 1) the minimum search resources from the National Institute for Health and Care Excellence (NICE) economic search requirements for single technology appraisals; 2) the resources recommended in the costs and economic evaluation chapter of SuRe Info. A majority (55%) of the reviews did not meet either the NICE or SuRe Info recommendations. The data collection was hindered by lack of clarity and errors in search methodology reporting within the systematic reviews. It is likely that current recommendations on searching for economic evaluations will change due to the recent closure of two specialized databases (NHS EED & HEED). 

Limitations stated by the author(s): 

The sample of systematic reviews was pragmatic and limited to papers which were freely available or available through the authors' subscription access. Additionally, the sample was limited to English language systematic reviews only. Reviews were determined to be systematic reviews if the review authors identified them as such; however, it is possible that the review authors may have mislabeled narrative reviews as systematic.  Because of lack of clarity in search methodology reporting, reviews that vaguely mentioned searching "The Cochrane Library" were assumed to have searched all databases contained by this resource, which may have overestimated the use of the NHS EED and HTA databases.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Single study

Appraisal of: "De Cock E, Cosmatos I, Kirsch E. Use of databases for health resource utilization and cost analyses in EU-5: results from a focused literature review [abstract and poster]. Value Health 2016;19(3):A80-A81"

Short description: 
This conference abstract describes the use of databases to identify health resource utilization (HRU).  The authors searched PubMed/MEDLINE and Embase for secondary care, health resources/utilisation, databases, and EU-5 (Great-Britain (GB), France, Germany, Italy, and Spain) published between 2011 and 2015. 146 abstracts were initially retrieved and 48 were eligible with settings in GB, Italy, Germany, France, and Spain. The type of database differed by country. The most frequently listed resource items were inpatient stays, specialist visits and pharmacotherapy. The use of databases depends on each country's health system.
 
Limitations stated by the author(s): 

The abstract was probably too brief for the authors to be able to explore limitations.

Limitations stated by the reviewer(s): 
The available information is limited because this was presented at a conference. The focus was on the EU5 countries and limited to five years, but in the absence of other published studies on the identification of resource use data, the full report is awaited with interest.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Preston, L., et al. "Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking." Syst Rev 2015 4: 82.

Reviewer(s): 
Short description: 

A convenience sample of nine Health Technology Assessment (HTA) systematic reviews of diagnostic test accuracy, with 302 included citations, was analysed to determine the number and proportion of included citations that were indexed in and retrieved from MEDLINE and EMBASE. An assessment was also made of the number and proportion of citations not retrieved from these databases but that could have been identified from the reference lists of included citations.

287/302 (95 %) of the included citations were indexed across MEDLINE and EMBASE. The reviews’ searches of MEDLINE and EMBASE accounted for 85 % of the included citations (256/302). Of the forty-six (15 %) included citations not retrieved by the published searches, 24 (8 %) could be found in the reference lists of included citations. Only 22/302 (7 %) of the included citations were not found. 

Limitations stated by the author(s): 

This study used a small, non-random sample of diagnostic test accuracy systematic reviews. This was done for reasons of pragmatism: first, because the authors had full access to the search strategies and reference databases of these reviews and, second, because of the exploratory nature of this project. We also assumed that the vast majority of the included citations in the reviews were located through screening of titles, abstracts and full papers.

We have also assumed, because the number of studies missed by operating the proposed MEDLINE, EMBASE and reference tracking strategy is so small that the findings of the systematic reviews would not have been greatly affected by their omission. However, this is uncertain and can only be assessed statistically by excluding those particular studies from the many analyses reported in the reviews, although, as noted above, most of these reviews conducted narrative synthesis. Such an analysis is a major task to undertake retrospectively and has therefore not been completed in this exploratory study. Future work should test the findings of this small study in a larger, preferably prospective sample of systematic reviews from multiple institutions. If possible, statistical analysis should also be undertaken to quantify fully the impact of omitting any data from studies that might otherwise be missed.

Limitations stated by the reviewer(s): 
No further limitations were identified.
Study Type: 
Single study

Appraisal of: Pitt C, Goodman C, Hanson K. Economic evaluation in global perspective: a bibliometric analysis of the recent literature. Health Econ. 2016 Feb;25 Suppl 1:9-28.

Reviewer(s): 
Short description: 

A bibliometric analysis of full economic evaluations of health interventions was conducted. A search of 14 databases identified 2844 economic evaluations published between 1 January 2012 and 3 May 2014. Analyses included: the sensitivity and specificity of searches in all databases, the distribution of the publications between countries, regions and health areas studied, as well as authors' country and institutional affiliations, journal type, language, and type of economic evaluation. The Scopus search identified the largest number of economic evaluations (85%), followed by NHS EED (80%), and MEDLINE (79%). EconLit captured only 1% of the publications identified. The majority of economic evaluations were published in biomedical journals (74%), addressed high-income countries (83%), and had at least one author based in a high-income country (91%). The health areas studied correlated more closely with the burden of diseases in high income countries compared to low-and middle-income countries. 

Limitations stated by the author(s): 

The NHS EED database is no longer being updated as of March 2015. Wiley HEED ceased to be available at the end of 2014. The authors assigned one point per institution per article in the analysis of institutional affiliation, which gives more weight to articles from multiple institutions. A single article could be classified as belonging to multiple health areas, which could be considered double-counting.

Limitations stated by the reviewer(s): 
The search strategies used to identify economic evaluations were unique strategies created for this study and were not established search filters, so they do not reflect current practice of information specialists. Because the search strategies were very specific, they may have missed relevant studies which were not explicit about being an economic evaluation. Additionally, variations in the search strategies between databases may have affected sensitivity and precision of retrieval. For example, in the cross-disciplinary databases strategies included a concept to limit retrieval to health topics. As such, the results presented might not fully represent the content of the databases. Because of these factors, it may be difficult to draw meaningful conclusions based on the data presented in this article.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Sampson M, Shojania KG, McGowan J, Daniel R, Rader T, Iansavichene AE et al. Surveillance search techniques identified the need to update systematic reviews. J Clin Epidemiol 2008; 61(8): 755-762.

Short description: 

The article reports on literature surveillance methods to identify new evidence eligible for updating systematic reviews. Five surveillance search approaches are tested in the context of identifying studies that would signal major or invalidating new evidence for existing systematic reviews of health care interventions.

Searches were tested in a cohort of 77 systematic reviews. No one method yielded consistently high recall of relevant new evidence, so combinations of the strategies were examined. A search algorithm based on PubMed’s related article search in combination with a subject searching using clinical queries was the most effective combination, retrieving all relevant new records in 68 cases. Citing RCT searching found unique material in only 2 cases (recall 37%).

Surveillance for emerging evidence that signals the need to update systematic reviews is feasible using a combination of subject searching and searching based on the PubMed’s related article function.

Limitations stated by the author(s): 

The retrospective observational design used here has certain limitations including absence of control over data collectiondin the case of the CENTRAL database, where indexing dates of records are not recorded; this limited our ability to determine when new evidence would have been available to reviewers. Our assessment of the point at which a review was in need of update could have been influenced by our access to subsequent confirmatory evidence, although any resulting bias would influence each search approach equally.

Limitations stated by the reviewer(s): 
No additional limitations were detected by the authors.
Study Type: 
Single study

Appraisal of: Waffenschmidt S, Janzen T, Hausner E, Kaiser T. Simple search techniques in PubMed are potentially suitable for evaluating the completeness of systematic reviews. J Clin Epidemiol. 2013; 66(6):660-665.

Short description: 

The aim was to determine whether PubMed's Related Citations (RelCits) and/or a simple-structured Boolean search (SSBS) are efficient and reliable search techniques to assess the completeness of an evidence base consisting of published randomized controlled trials (RCTs). A retrospective analysis of 330 PubMed citations included as relevant in 19 systematic reviews (SRs) of drugs was undertaken, with the proportion of relevant citations identified by both RelCits and SSBS determined. It was concluded that combining the first 20 RelCits and an SSBS in PubMed is a suitable method to assess the completeness of an evidence base of published RCTs.

Limitations stated by the author(s): 
  • Limited generalizability of results because of the development of a gold standard on the basis of SRs completed by IQWiG.
  • Questionable if the techniques also work with new citations not yet indexed as well as citations without an abstract.
  • Applicability of the findings is currently limited to the search for RCTs; further research is needed to test the applicability to non-RCTs.
  • Only PubMed citations were analyzed.
Limitations stated by the reviewer(s): 
No additional limitations were detected by the reviewers.
Study Type: 
Single study
Tags:

Appraisal of: Sampson M. Updating searches for systematic reviews [dissertation]. Department of Information Studies: University of Aberystwyth; 2009.

Short description: 

This thesis examines methods for updating searches for systematic reviews of healthcare interventions on the base of 77 systematic reviews. The Related Article search showed the highest recall. Relative performance of the test searches was consistent regardless of whether the intervention was a drug, device or procedure. All searches showed variability across clinical areas, but Related Articles RCT showed the most consistency. The pairing of Related Article RCT and Clinical Query gave excellent recall of new relevant material.

Limitations stated by the author(s): 

Screening method, generalization of results on precision (cohort was only 10 AHRQ Evidence Reports), the selection of the sample may have implications for generalizability.

Limitations stated by the reviewer(s): 
No additional limitations were detected by the reviewers.
Study Type: 
Single study

Appraisal of: O'Leary N, Tiernan E, Walsh D, Lucey N, Kirkova J, Davis MP. The pitfalls of a systematic MEDLINE review in palliative medicine: symptom assessment instruments. Am J Hosp Palliat Care. 2007; 24(3): 181-184.

Short description: 

The authors describe the usefulness of the related articles link in PubMed as part of a systematic MEDLINE review. Of 1181 citations identified, through a PubMed search, 10 articles met the indusion criteria. as did a further 21 articles identified through hand-searching the references of the 10 articles. The PubMed related articles link of the 10 articles yielded 15% of all reports finally included in the systematic review. The authors suggest that it is a useful tool in PubMed for revietrieving complex evidence.

Limitations stated by the author(s): 

No limitations stated by the authors.

Limitations stated by the reviewer(s): 
Different numbers are described in the result section: references that met the inclusion criteria and references finally included in the systematic review. This selection process is not clearly documented and it is not clear how many references finally included in the systematic review are identified through which source.
Study Type: 
Single study

Appraisal of: Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 2005; 331(7524): 1064-1065.

Short description: 

To describe where papers come from in a systematic review of complex evidence. Only 30% of sources were obtained from the protocol defined at the outset of the study (that is, from the database and hand searches). Fifty one per cent were identified by “snowballing” (such as scanning reference lists (44%) and citation tracking (7%)), and 24% by personal knowledge or personal contacts.

Limitations stated by the author(s): 

No limitations were stated by the authors.

Limitations stated by the reviewer(s): 
The article assesses the total contribution of the different sources but it is not possible to identify the additional references which could only be found by the additional search techniques. Besides that it is not mentioned if perhaps the original Boolean search was inadequate as the searches and the results are not discussed in detail. The generalizability of the results is also questionable because the authors results are based only on one review (topic “diffusion of service-level innovations in healthcare organizations”).
Study Type: 
Single study
Syndicate content