Databases

Scope: provide the database name(s); add to records where a database rather than a website or interface is discussed

Appraisal of: "Arber M, Glanville J, Isojarvi J, Baragula E, Edwards M, Shaw A, Wood H. Which databases should be used to identify studies for systematic reviews of economic evaluations? Int J Technol Assess Health Care. 2018 Jan;34(6):547-54."

Reviewer(s): 
Short description: 

NHS EED and HEED, two key databases for retrieving health economics information, have closed. Based on this changed landscape, this article assesses which databases are now the best sources of information for retrieving economic evaluations to inform systematic reviews. The authors built a quasi-gold standard database of 351 records compiled from 46 systematic reviews of economic evaluations. Nine databases were searched for each record. Embase had the highest yield (89%), followed by Scopus (84%) and MEDLINE and PubMed (both 81%). The HTA database identified the highest number of unique citations (13/351). Embase also uniquely identified two conference abstracts, an important consideration if this type of material is eligible for inclusion in a review. All nine database combined retrieved 337/351 (96%) records. Searching a combination of Embase, the HTA database and either PubMed or MEDLINE identified 95% of the quasi-gold standard records (333/351). The authors concluded that searching additional database outside the core group may be inefficient because of limited incremental yield. Searchers should not rely on PubMed or MEDLINE alone. Searching a multi-disciplinary database may also be useful, especially for non-clinical or public health topics. The authors conclude that searchers should focus on developing suitable search strategies in these key databases to ensure high sensitivity and adequate precision. Supplementary search techniques such as grey literature searching may be more efficient than searching a larger number of databases, as 14/351 (4%) citations were not identified in any of the databases.

MEDLINE search strategies reported in source systematic reviews were also assessed. 10/29 (34.5%) of re-run search strategies missed at least one of the included records found in MEDLINE (with 25 citations missed in total). Weaknesses in the population or intervention concepts, rather than the economics concept, were identified as negatively impacting search retrieval.

Limitations stated by the author(s): 

Candidate reviews were screened by one reviewer only, unless a clear inclusion decision could not be made.

Researchers may have access to more than the nine databases chosen for this review. Other databases were not tested.

The quasi-gold standard database used in this project was compiled from reports commissioned or carried out by a single entity (NICE) and comprised 351 economic evaluations. A larger reference set that included evaluations from more than one entity would have increased generalizability of the findings. Reports focused on clinical medicine or public health in the context of the United Kingdom, so may be less generalizable for other topics such as mental health or health management, or for healthcare in low- and middle-income countries.

The quality of the searches conducted by the systematic reviews from which the quasi-gold standard database was harvested was not assessed. It was assumed that the searches conducted were of sufficient quality to provide a representative set of economic evaluations. Weaknesses in these search strategies could have failed to retrieve relevant economic evaluations, perhaps reducing the representativeness of the quasi-gold standard.

“Efficient” was defined as the fewest number of databases that could be combined to retrieve that largest number of citations. Other types of efficiency such as the time needed to search a database or the number of irrelevant citations retrieved were not considered. Many databases are available on multiple platforms which can impact retrieval and efficiency.

For the evaluation of MEDLINE searches, the sensitivity of each of review’s search strategy in MEDLINE was used as a proxy for search quality. There are other methods available to assess the quality of searches, such as the PRESS checklist, or how the search strategy performs when translated to other databases.

Limitations stated by the reviewer(s): 
While not a particular limitation of this study, the authors do note that the HTA Database is not currently being updated, which may hamper retrieval of recently published studies from the producers of HTA. Until the database is updated, searchers should identify this material by searching websites of individual HTA agencies, or through a general web search. The authors do not mention which databases were searched in the reports used to generate the quasi-gold standard. Since all economic evaluations were drawn from a single organization (NICE), it is possible that they share similar search methodologies, and, thus may search a standard list of databases. If not all of the nine databases were searched for these reports, it may have biased estimates of database yield.
Study Type: 
Single study

Appraisal of: "Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev, 5(1). doi:10.1186/s13643-016-0249-x"

Reviewer(s): 
Short description: 

This review provides a systematic overview of the available published evidence of searching methods to inform qualitative evidence synthesis (QES). The author sought to assess and identify:

1) the current state of knowledge in relation to searching for qualitative evidence

2) the robustness of the evidence base

3) research gaps and future priorities.

The studies were obtained from the Reference Manager database of the Cochrane Qualitative Methods Group’s study register, of which the author is responsible for updating and maintaining. Supplementary citation searches via Google Scholar was also carried out for 15 key papers. 113 studies were assessed for inclusion. Quality assessment of the included studies was not deemed feasible due to a large proportion of the included studies providing only narrative findings, the lack of a common appraisal instrument and the high levels of heterogeneity across the remaining studies.

The evidence underpinning systematic approaches to searching for qualitative evidence is classified and summarized within one or more of eight headings/ “7 S structure/ framework”: overviews and methodological guidelines, sampling, sources, structured questions, search procedures, search strategies and filters, supplementary strategies and standards. The author summarizes the available evidence and key issues within each section and makes recommendations for further empirical research. Table 7 breaks down the key starting principles in reference to the “7S structure” of searching to inform qualitative evidence synthesis to inform future guidance and Table 8 provides an overview of research priorities.

The review concludes that there is a lack of empirical data to inform information retrieval for QES and that the strength of the evidence is weak and largely based on personal/ professional experience and case studies. Advances have been made in reporting QES, however, validated standards are lacking. 

Limitations stated by the author(s): 

The author states that in order for studies to be included, the references needed to include terms specifically related to searching or retrieval in their titles or abstracts, cite a number of key texts, or be referred to from previously identified items. The full-text of all papers reporting QES were not examined. There is a possibility that potentially relevant reviews reporting emerging information retrieval methods that were not reported in the title or abstract were missed. However, these risks are offset by the sensitive search approach and the currency and comprehensiveness of the Cochrane Qualitative Methods Group study register. The author also notes that some papers were excluded as they did not distinguish between qualitative and quantitative approaches, which could potentially be useful for mixed methods reviews.

Limitations stated by the reviewer(s): 
1. This was a single reviewer methodology review so judgements on eligibility and interpretations of potential significance and contribution of individual studies may not be consistent and/or reproducible. [Study Selection Bias] 2. Although as a methodology review there is no formal requirement to follow PRISMA reporting standards this review may have benefited from more complete and transparent reporting. [Incomplete Reporting Bias] 3. As the reviewer was author on a high proportion of included studies this may have consciously impacted on study identification and subconsciously on study selection and interpretation. [Citation Bias; Observer Bias]
Study Type: 
Single study

Appraisal of: "Yao X, Vella E, Brouwers M. How to conduct a high-quality systematic review on diagnostic research topics. Surg Oncol. 2018 Mar;27(1):70-75."

Reviewer(s): 
Short description: 

Authors suggest six steps are necessary for conducting high-quality systematic reviews on diagnostic topics around the objective, research question, searching for literature, assessing risk of bias, analyzing, and interpreting data. The review aims to instruct researchers on diagnostic methodology and recent updates in the field.

Rather than designing a study around the elements of PICO, PIRO is recommended in which “I” represents index test, and “R” represents reference standard.

The literature search for diagnostic topics can begin with using an existing systematic review that answers the research question, and is of satisfactory quality. The search date of that study can be used as a starting point to search forward in MEDLINE, Embase, Cochrane Database of Systematic Reviews, and PROSPERO. A grey literature search is also recommended.

When no appropriate systematic reviews are available, a search of primary studies will need to be conducted. Databases to be used in this case include MEDLINE, Embase, and Cochrane Library databases, as well as related conference proceedings. It is preferred that language limits are not used and that a grey literature search is carried out if deemed appropriate by researchers. The search should include Medical Subject Headings terms and keywords representing the diagnostic test and target condition. Adding terms to capture diagnostic accuracy or study design is not recommended, as indexing of this concept has not been proven to be dependable.

This is a second article in a series. First article: Yao X, Vella E. How to conduct a high-quality original study on a diagnostic research topic. Surg Oncol. 2017;26(3):305-9.

 

Limitations stated by the author(s): 

Authors noted that in order to keep the article brief, they did not fully explore the concepts of how to design strategies for different databases or peer review.

Limitations stated by the reviewer(s): 
Authors made general observations about the quality of diagnostic studies needing improvement without expanding on any evidence to support the claim.
Study Type: 
Review

Appraisal of: "Dakin H, Abel L, Burns R, Yang Y. Review and critical appraisal of studies mapping from quality of life or clinical measures to EQ-5D: an online database and application of the MAPS statement. Health Qual Life Outcomes. 2018;16:202."

Short description: 

This paper presents an updated systematic review of studies reporting mapping algorithms from clinical or patient reported outcome measures to EQ-5D-3L or EQ-5D-5L, as well as an assessment of how far mapping studies conform to the MAPS reporting statement.

The review identified 144 mapping studies reporting 190 algorithms mapping from 110 different instruments onto EQ-5D. The 15/17 studies published in 2016 had low adherence to the MAPS checklist.

The mapping studies can be accessed in the Health Economics Research Centre (HERC) Database of Mapping Studies, available at https://www.herc.ox.ac.uk/downloads/herc-database-of-mapping-studies. This database was created in 2013 and is focused on studies mapping to EQ-5D.

Limitations stated by the author(s): 

The review did not include a search of EMBASE. Single reviewer data extraction was undertaken. Only 2016 studies were assessed against the MAPS criteria.

Limitations stated by the reviewer(s): 
The authors do not describe how frequently they update the database.
Study Type: 
Single study

Appraisal of: "Arber M, Wood H, Isojarvi J, Glanville J. Which information sources should be used to identify studies for systematic reviews of economic evaluations in healthcare? Value Health. 2017 Oct/Nov;20(9):A738. Abstract PRM46."

Reviewer(s): 
Short description: 

NHS EED and HEED, two key databases for retrieving health economics information, have closed. Based on this changed landscape, this abstract assesses which database are now the best sources of information for retrieving economic evaluations for models and systematic reviews. The authors built a quasi-gold standard database of 351 records compiled from 46 systematic reviews of economic evaluations. Nine databases were searched for each record. Embase had the highest yield (0.89), followed by Scopus (0.84) and MEDLINE and PubMed (both 0.81). The HTA database identified the highest number of unique citations (13/351). All nine database combined retrieved 337/351 (0.96) records. The authors conclude that for most systematic reviews, Embase, the HTA database and either PubMed or MEDLINE are likely sufficient to identify economic evaluations found in bibliographic databases. Searching a multi-disciplinary database may also be useful, especially in non-clinical topics. Supplementary search techniques may be more efficient than searching a larger number of databases.

MEDLINE search strategies reported in source systematic reviews were also assessed. 10/29 (34.5%) of re-run search strategies missed at least one of the included records found in MEDLINE (with 25 citations missed in total). Weaknesses in the population or intervention concepts, rather than the economics concept, were identified as negatively impacting search retrieval.

Limitations stated by the author(s): 

None stated.

Limitations stated by the reviewer(s): 
Since the results were published only in abstract form, very limited information was presented on methods, results and conclusions.
Study Type: 
Single study

Appraisal of: "Ara R, Brazier J, Peasgood T, Paisley S. The identification, review and synthesis of health state utility values from the literature. Pharmacoeconomics. 2017;35:43–55."

Short description: 

This paper is a guidance document providing an overview of how health state utility values can be identified, reviewed and synthesised when conducting systematic reviews. The paper includes a case study of a review in osteoporosis-related conditions.

In terms of study identification, the authors note that a range of study designs could be relevant and that a number of instruments could be required including condition-specific preference based measures or generic preference based measures. They recommend that a variety of resources and methods should be used to identify studies. As well as electronic databases, searchers should also look at reference lists, conduct key author and citation searches and contact experts.

The authors recommend caution in using filters too early in the search process, to avoid missing potentially relevant studies. The authors also note the absence of dedicated subject headings within MeSH and EMTREE, and that although general subject headings such as ‘Quality of life’ will yield relevant studies, they are likely to demonstrate poor precision. Free text terms should be included in searches and are categorised as general terms (such as QALY), instrument specific terms (such as EQ-5D) and terms describing methods of utility elicitation such as standard gamble.

Limitations stated by the author(s): 

The authors discuss the issues of searching for utility studies including current limitations. They do not discuss limitations of their review and do not provide details of their methods in this publication.

Limitations stated by the reviewer(s): 
The overview and recommendations in relation to searching do not seem controversial.
Study Type: 
Single study

Appraisal of: "Boeker M., Vach W, Motschall E.. Time-dependent migration of citations through PubMed and OvidSP subsets: a study on a series of simultaneous PubMed and OvidSP searches. Stud Health Technol Inform. 2013;192:1196."

Reviewer(s): 
Short description: 

The aim of this study was to compare PubMed and OvidSP in terms of the timeliness of the different interfaces. The authors conclude OvidSP is updated within a period of two days.

Limitations stated by the author(s): 

No limits were discussed by the authors.

Limitations stated by the reviewer(s): 
This is a conference abstract so not enough detail is presented to ascertain the limitations.
Study Type: 
Single study

Appraisal of: "Fortier KJ, Kiss N, Tongbram V. What is the optimal search engine for results from Embase and MEDLINE: Ovid or Embase.com? Value Health 2013:16(3):A25."

Reviewer(s): 
Short description: 

The aim of this study was to compare OVID and Embase.com in terms of the results of searches and ease of searching. Using several searches the authors investigated differences in results and why any dispartieis exist. Ease of use was rates on a scale of 0 to 5. Although differences were found there were few if any relevant articles missed by either search interface. The authors conclude that there is no notable difference between OVID and Embase.com but Embase.com was rated easier to use overalll.

Limitations stated by the author(s): 

No limits were discussed by the authors.

Limitations stated by the reviewer(s): 
This is a conference abstract so not enough detail is presented to ascertain the limitations.

Appraisal of: Shannon M. Sullivan, Kate Tsiplova & Wendy J. Ungar. A scoping review of pediatric economic evaluation 1980-2014: do trends over time reflect changing priorities in evaluation methods and childhood disease? Expert Rev Pharmacoecon Outcomes R

Short description: 

This is a scoping review of the volume, economic analytic techniques and therapeutic areas encompassed by the pediatric economic evaluation studies published between 1980 and 2014.  The review uses records from the Paediatric Economic Database Evaluation (PEDE) database of pediatric economic evaluations to explore trends in analytical techniques, target age groups, interventions, outcomes and disease classifications. The paper describes the eligibility criteria for PEDE. PEDE is populated through searches of  MEDLINE, CINAHL, EMBASE, International Pharmaceutical Abstracts (IPA), EconLit, the Cochrane Library, NHS EED and ERIC, as well as searches of over 73 websites of HTA agencies and research groups. PEDE is updated annually.

Limitations stated by the author(s): 

Studies might have been missed for inclusion in PEDE if adolescents were included in adult populations.  Non-English language papers are not eligible unless they have an English abstract.

Limitations stated by the reviewer(s): 
No other limitations were detected by the reviewers.

Appraisal of: Thielen FW, Van Mastrigt G, Burgers LT, Bramer WM, Majoie H, Evers S, Kleijnen J. How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3). Exp

Short description: 

This article is the second in a three-part series on how to prepare a systematic review of economic evaluations. It provides a good overview of the literature on how to select relevant databases and develop a search strategy for retrieving economic evaluations. While the main target audience of the article is developers of clinical practice guidelines, the process described is helpful for systematic review researchers as well as those undertaking health technology assessments. The authors identify four steps in the search process: (1) selecting relevant databases (basic, specific and optional); (2) developing a comprehensive search strategy; (3) performing the searches (including documentation), and (4) selecting the relevant studies. The authors note the recent discontinuation of two health economics databases (HEED and NHS EED), which has resulted in an increased reliance on the use of search filters designed to capture economic evaluations. Also noted are unsettled issues, such as the lack of consensus on how many and which databases should be searched, as well a lack of uniform guidance on the methodology of developing a sound search strategy. Validated search filters and automated processes may help to overcome problems created by the lack of health economics-specific databases.

Limitations stated by the author(s): 

No limitations stated by the study authors.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Review
Syndicate content