Embase

Appraisal of: "Harboe I, Desser A, Nordheim L, Glanville J. PD85: Testing search filters to retrieve economic evaluations in Embase. Int J Technol Assess Health Care. 2018;34(S1):158."

Reviewer(s): 
Short description: 

The authors tested a self-created Ovid Embase cost-effectiveness analysis (CEA) search filter, compared to six other published filters designed to identify economic evaluations in Embase. In this filter validation study, sensitivity, precision and specificity were measured against a gold standard set of economic evaluations derived from the National Health Service Economic Evaluation Database (NHS EED). The set was populated with NHS EED citations published in 2008-2013 that were also available in Ovid Embase (n = 2,198). Testing of the CEA filter resulted in a sensitivity of 0.899 and precision of 0.029. No filter achieved the stated objective of achieving a sensitivity of at least 0.90 with a precision of 0.10, and specificity of at least 0.95, although one tested filter was close to achieving that objective. The authors conclude that it is challenging to develop economic search filters that balance sensitivity and precision. Choice of filter should be based on the researcher’s acceptable levels of performance.

Limitations stated by the author(s): 

None stated.

Limitations stated by the reviewer(s): 
The gold standard set of economic evaluations was derived entirely from the NHS EED database which was itself updated by searches done in various databases. The filters used in these search updates would control the contents of both the database and the gold standard set used here, which could potentially have skewed the performance of the tested filters. This is only a conference abstract with limited details of the study methods, results and conclusions. Further details can be found in the lead author’s thesis (see Supplemental publications to the study, below).
Study Type: 
Single study
Related Chapters: 

Based on: Harboe I. Testing the best performing methodological search filters to retrieve health economic evaluations in Embase: a filter validation study [master’s thesis].

Appraisal of: "Gurung P, Makineli S, Spijker R, Leeflang MMG. The EMTREE term "Diagnostic Test Accuracy Study" retrieved less than half of the diagnostic accuracy studies in EMBASE. J Clin Epidemiol. 2020;126:116-21."

Reviewer(s): 
Short description: 

The authors tested the retrieval of the EMTREE terms ‘diagnostic test accuracy study’ (checktag), ‘diagnostic test accuracy’ and ‘sensitivity and specificity’.  Two sets of records were obtained by randomly selecting records from Embase and screening them to identify reports of diagnostic test accuracy studies (DTAs).  The second set, which tested the recall of all three terms, comprised 2000 records added in 2016. Following exclusions and deduplication, 1223 records were screened to identify DTA studies. 33 of the records were considered to be DTA studies, but only 14 were labelled with the EMTREE checktag (sensitivity=42.4%). Of the non-DTA studies, 6 were labeled with the EMTREE checktag. 9 studies were assigned the term ‘diagnostic test accuracy’ (8 accurately, giving a sensitivity of 24.2%) and 30 studies were assigned the term ‘sensitivity and specificity’ (21 accurately, giving a sensitivity of 63.6%). Using all three terms gave a sensitivity of 72.7%.

The EMTREE checktag has only 42.4% sensitivity and its use alone risks missing more than half of the relevant studies. Using all three terms together might provide adequate sensitivity for a scoping search only.  Imaging studies were particularly lacking in terms of accurate assignment of  indexing terms.

The Embase indexing is not currently reliable enough to use for identifying DTA studies for systematic review purposes and new techniques such as machine learning may have increasing potential to identify studies.

 
Limitations stated by the author(s): 

The authors acknowledged that the two sets of records might not reflect a real review situation where the search would usually not limit to DTA studies and subject terms would also be used. Also, records that were excluded by two assessors were not all checked in detail.

 
Limitations stated by the reviewer(s): 
No further limitations noted.
Study Type: 
Single study

Appraisal of: "Li L, Smith HE, Atun R, Tudor Car L. Search strategies to identify observational studies in MEDLINE and Embase. Cochrane Database Systematic Reviews 2019, Issue 3. Art. No.: MR000041. DOI:10.1002/14651858.MR000041.pub2”

Reviewer(s): 
Short description: 

The objective of this review was to assess the sensitivity and precision of a search strategy with a methodological filter to identify observational studies in MEDLINE and Embase.

In April 2018 the Authors searched Medline, Embase, CINAHL, the Cochrane Library, Google Scholar, Open Grey and scanned reference list for articles.

The two studies included tested 18 filters in total which used different approaches.

Using search filters for identifying observational studies for systematic reviews  is not recommended and further research should be undertaken.

Limitations stated by the author(s): 
  • Focus of these search strategies was different: one study focused on retrieval of observational studies of surgical interventions and the other on non-randomised studies
  • Studies were published 10 years before the date of this systematic review and indexing of observational studies has changed in the meantime
  • Authors were unsure of the reliability of the reported performance data due to the lack of external validation for most search filters and methodological limitations
Limitations stated by the reviewer(s): 
• The review included only two studies • Heterogeneity between studies • Search was not updated before publication and therefore may have missed more recent publications
Study Type: 
Review

Appraisal of: "Yao X, Vella E, Brouwers M. How to conduct a high-quality systematic review on diagnostic research topics. Surg Oncol. 2018 Mar;27(1):70-75."

Reviewer(s): 
Short description: 

Authors suggest six steps are necessary for conducting high-quality systematic reviews on diagnostic topics around the objective, research question, searching for literature, assessing risk of bias, analyzing, and interpreting data. The review aims to instruct researchers on diagnostic methodology and recent updates in the field.

Rather than designing a study around the elements of PICO, PIRO is recommended in which “I” represents index test, and “R” represents reference standard.

The literature search for diagnostic topics can begin with using an existing systematic review that answers the research question, and is of satisfactory quality. The search date of that study can be used as a starting point to search forward in MEDLINE, Embase, Cochrane Database of Systematic Reviews, and PROSPERO. A grey literature search is also recommended.

When no appropriate systematic reviews are available, a search of primary studies will need to be conducted. Databases to be used in this case include MEDLINE, Embase, and Cochrane Library databases, as well as related conference proceedings. It is preferred that language limits are not used and that a grey literature search is carried out if deemed appropriate by researchers. The search should include Medical Subject Headings terms and keywords representing the diagnostic test and target condition. Adding terms to capture diagnostic accuracy or study design is not recommended, as indexing of this concept has not been proven to be dependable.

This is a second article in a series. First article: Yao X, Vella E. How to conduct a high-quality original study on a diagnostic research topic. Surg Oncol. 2017;26(3):305-9.

 

Limitations stated by the author(s): 

Authors noted that in order to keep the article brief, they did not fully explore the concepts of how to design strategies for different databases or peer review.

Limitations stated by the reviewer(s): 
Authors made general observations about the quality of diagnostic studies needing improvement without expanding on any evidence to support the claim.
Study Type: 
Review

Appraisal of: "Fortier KJ, Kiss N, Tongbram V. What is the optimal search engine for results from Embase and MEDLINE: Ovid or Embase.com? Value Health 2013:16(3):A25."

Reviewer(s): 
Short description: 

The aim of this study was to compare OVID and Embase.com in terms of the results of searches and ease of searching. Using several searches the authors investigated differences in results and why any dispartieis exist. Ease of use was rates on a scale of 0 to 5. Although differences were found there were few if any relevant articles missed by either search interface. The authors conclude that there is no notable difference between OVID and Embase.com but Embase.com was rated easier to use overalll.

Limitations stated by the author(s): 

No limits were discussed by the authors.

Limitations stated by the reviewer(s): 
This is a conference abstract so not enough detail is presented to ascertain the limitations.

Appraisal of: Preston, L., et al. "Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking." Syst Rev 2015 4: 82.

Reviewer(s): 
Short description: 

A convenience sample of nine Health Technology Assessment (HTA) systematic reviews of diagnostic test accuracy, with 302 included citations, was analysed to determine the number and proportion of included citations that were indexed in and retrieved from MEDLINE and EMBASE. An assessment was also made of the number and proportion of citations not retrieved from these databases but that could have been identified from the reference lists of included citations.

287/302 (95 %) of the included citations were indexed across MEDLINE and EMBASE. The reviews’ searches of MEDLINE and EMBASE accounted for 85 % of the included citations (256/302). Of the forty-six (15 %) included citations not retrieved by the published searches, 24 (8 %) could be found in the reference lists of included citations. Only 22/302 (7 %) of the included citations were not found. 

Limitations stated by the author(s): 

This study used a small, non-random sample of diagnostic test accuracy systematic reviews. This was done for reasons of pragmatism: first, because the authors had full access to the search strategies and reference databases of these reviews and, second, because of the exploratory nature of this project. We also assumed that the vast majority of the included citations in the reviews were located through screening of titles, abstracts and full papers.

We have also assumed, because the number of studies missed by operating the proposed MEDLINE, EMBASE and reference tracking strategy is so small that the findings of the systematic reviews would not have been greatly affected by their omission. However, this is uncertain and can only be assessed statistically by excluding those particular studies from the many analyses reported in the reviews, although, as noted above, most of these reviews conducted narrative synthesis. Such an analysis is a major task to undertake retrospectively and has therefore not been completed in this exploratory study. Future work should test the findings of this small study in a larger, preferably prospective sample of systematic reviews from multiple institutions. If possible, statistical analysis should also be undertaken to quantify fully the impact of omitting any data from studies that might otherwise be missed.

Limitations stated by the reviewer(s): 
No further limitations were identified.
Study Type: 
Single study

Appraisal of: "Preston L, Carroll C, Gardois P, Paisley S, Kaltenthaler E. Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking."

Short description: 

This exploratory study analysed a convenience sample of nine Health Technology Assessments (HTA) reviews of diagnostic test accuracy (DTA) to identify which of the included citations were indexed on MEDLINE or EMBASE; to identify the number and proportion of citations that were retrieved by the search strategies with or without the addition of checking the reference lists of included studies.                     

Of the 302 included citations included in the nine reviews, 287 (95%) were indexed in MEDLINE and EMBASE. The reviews searches accounted for 256 (85%) of the included citations and a further 24 (8%) could be identified from the reference lists of included citations. 7% of the citations (22/302) were not found by searching or reference checking.

The proportion of citations identified using both searches of MEDLINE, and EMBASE and checking the reference lists resulted in the identification of 280/302 (93%) included studies. The authors suggest that there might be a case for restricting systematic review searches  of DTA studies to MEDLINE, EMBASE and the reference lists of included studies. 

Limitations stated by the author(s): 

The study’s pragmatic, convenience sample of a small number of published DTA reviews was intended as an exploratory analysis only. The  authors did not conduct an analysis to assess the impact of excluding the unidentified studies on the conclusions of the HTAs, so the results of the study are uncertain. 

Limitations stated by the reviewer(s): 
Six of the nine studies in the sample used a published filter for diagnostic studies, which is not recommended by Cochrane. This may have impacted the sensitivity of the searches and the sample not be representative of other DTA reviews.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Beynon R, Leeflang MM, McDonald S, Eisinga A, Mitchell RL, Whiting P, Glanville JM. Search strategies to identify diagnostic accuracy studies in MEDLINE and EMBASE. Cochrane Database Syst Rev 2013, Issue 9.

Reviewer(s): 
Short description: 

This review assessed the performance (sensitivity, specificity and precision) of 70 filters (described in 19 studies) designed to identify diagnostic test accuracy studies for use with either the MEDLINE or Embase database. The filters were appraised for potential bias in terms of use of a diagnostic test accuracy search strategy; choice of gold standard; method of validation. The ISSG Search Filter Appraisal Checklist was used to assess quality. None of the filters tested met the pre-specified levels of sensitivity > 90% and precision >10%. Using methodological search filters as the sole method of identifying studies for systematic reviews of diagnostic test accuracy is not recommended.

Limitations stated by the author(s): 

A search for unpublished search filters was not conducted. The authors acknowledged the possibility of bias introduced by the studies that evaluated filters but did not replicate them in the paper. 

Limitations stated by the reviewer(s): 
No limitations have been identified by the reviewers.
Study Type: 
Review
Related Chapters: 

Appraisal of: Golder S, Wright K, Rodgers M. Failure or success of search strategies to identify adverse effects of medical devices: a feasibility study using a systematic review. Syst Rev 2014;3:113.

Short description: 

A systematic review of the safety of recombinant human bone morphogenetic protein-2 (rhBMP-2) for spinal fusion was used as a case study to investigate whether adverse effects terms were prevalent in the bibliographic records of the included studies. The aim of the study was to assess whether the development of a search filter for medical devices would be feasible.

Out of the 82 publications (49 studies) included in the systematic review, 51 were indexed in MEDLINE and 55 in Embase. The results showed that 94% of the MEDLINE records and 95% of the Embase records contained at least one adverse effects related term. However, the records contained a wide range of adverse effects terms, and no single term was able to retrieve the majority of the included publications. In addition, the most successful search terms were different from search terms shown to be most successful when searching for adverse drug effects. The study concluded that the development of adverse effects search filters for medical devices is feasible, but that such filters should be developed using a large set of relevant records in order to identify the optimal combination of search terms.

Limitations stated by the author(s): 

The main limitation to this study is that because only one systematic review was used as a case study, the generalisability of the study results is limited. This case study is also of a medical device with a pharmaceutical component, while a case study of a different kind of a medical device might have given different results.

In addition, unlike many other HTAs and systematic review authors, the study authors were able to obtain unpublished data directly from the manufacturer. They also included an unusually high number of conference abstracts and multiple publications for the same study in their review. This might have influenced the findings of this study.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Single study

Comments from the authors:

Appraisal of: Golder S, Wright K, Rodgers M. The contribution of different information sources to identify adverse effects of a medical device: a case study using a systematic review of spinal fusion. Int J Technol Assess Health Care. 2014;30(14):1-7.

Short description: 

The included studies from a systematic review of the safety of recombinant human bone morphogenetic protein-2 (rhBMP-2) for spinal fusion were used for analysis. For each study, it was investigated in whichc sources they were available and where they were identified. If a study was available on a database but not retrieved by the original search strategy, the bibliographic record was examined to determine why it was not retrieved. The sensitivity, precision, and numbers needed to read for searches in each of the databases was calculated, as well as sensitivity*precision. The minimum combination of sources required to identify all the included publications using the original search strategies used was recorded.

The minimum combination of sources to identify all the publications was Science Citation Index (SCI), Embase, CENTRAL and either MEDLINE or PubMed, in addition to reference checking, contacting authors and using automated current awareness service.

The highest precision was achieved in CENTRAL, followed by ToxFile and MEDLINE. The highest sensitivity*precision was achieved by MEDLINE, followed by CENTRAL and Science Citation Index.

Limitations stated by the author(s): 

The main limitation of this study is that relatively few sources were searched in the original systematic review, meaning that only a few resources could be compared for their relative value in providing relevant data.

Also, as the analysis is only based on one case study systematic review, the generalisability of the results is limited.

In addition, the authors were able to obtain unpublished data directly from the manufacturer. They also included an unusually high number of conference abstracts and multiple publications for the same study in their review. This might have influenced the findings of this study.

This case study is also of a medical device with a pharmaceutical component, while a case study of a different kind of a medical device might have given different results.

 

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Single study
Syndicate content