MEDLINE

Appraisal of: “Glanville J, KotasE, Featherstone R, Dooley G. Which are the most sensitive search filters to identify randomized controlled trials in MEDLINE? J Med Libr Assoc 2020,108(4):556-563”

Reviewer(s): 
Short description: 

Search filters exist to help with the standardised and consistent retrieval of a pre-specified group of studies from bibliographic databases. The Cochrane RCT search filters are an example of such a tool, these were previously tested and adapted in 2008, and were then checked in this 2020 study for their current performance in retrieving appropriate study records.

In 2019, the authors tested current Cochrane RCT search filters alongside 36 other known RCT search filters for sensitivity, specificity, and precision by searching within a gold standard set of RCTs from 2016. The gold standard set of RCT abstracts was derived from the Cochrane CENTRAL Cochrane Central Register of Controlled Trials) database which uses Medline as one of its sources, the set was then combined with each individual search filter.

The authors found that the current Cochrane RCT search filters performed well against other search filters as it has a relatively high sensitivity and a slightly better precision relative to other search filters. If a RCT filter with a balance of sensitivity and precision is required, the authors suggest using the Nwosu et al. (1998) search filter.

Limitations stated by the author(s): 

The authors describe the following limitations of this study:

  • The search was conducted on a set of 2016 records with (the then current) 2019 indexing, so the authors were unable to ascertain how the search filters may have performed under actual 2016 conditions with those indexing terms.
  • Some records in CENTRAL may not be reports of controlled clinical trials due to errors in indexing and/or classification, but the impact of their small number within such a large dataset was felt to be minimal.
  • Relative precision is a pragmatic measure of fitness and the use of CENTRAL as a gold standard is the best approximation of RCTs within Medline. There may be some slight discrepancy in reported sensitivity presented in this paper with when re-run under other circumstances.
Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Single study
Related Chapters: 

Appraisal of: "Li L, Smith HE, Atun R, Tudor Car L. Search strategies to identify observational studies in MEDLINE and Embase. Cochrane Database Systematic Reviews 2019, Issue 3. Art. No.: MR000041. DOI:10.1002/14651858.MR000041.pub2”

Reviewer(s): 
Short description: 

The objective of this review was to assess the sensitivity and precision of a search strategy with a methodological filter to identify observational studies in MEDLINE and Embase.

In April 2018 the Authors searched Medline, Embase, CINAHL, the Cochrane Library, Google Scholar, Open Grey and scanned reference list for articles.

The two studies included tested 18 filters in total which used different approaches.

Using search filters for identifying observational studies for systematic reviews  is not recommended and further research should be undertaken.

Limitations stated by the author(s): 
  • Focus of these search strategies was different: one study focused on retrieval of observational studies of surgical interventions and the other on non-randomised studies
  • Studies were published 10 years before the date of this systematic review and indexing of observational studies has changed in the meantime
  • Authors were unsure of the reliability of the reported performance data due to the lack of external validation for most search filters and methodological limitations
Limitations stated by the reviewer(s): 
• The review included only two studies • Heterogeneity between studies • Search was not updated before publication and therefore may have missed more recent publications
Study Type: 
Review

Appraisal of: "Yao X, Vella E, Brouwers M. How to conduct a high-quality systematic review on diagnostic research topics. Surg Oncol. 2018 Mar;27(1):70-75."

Reviewer(s): 
Short description: 

Authors suggest six steps are necessary for conducting high-quality systematic reviews on diagnostic topics around the objective, research question, searching for literature, assessing risk of bias, analyzing, and interpreting data. The review aims to instruct researchers on diagnostic methodology and recent updates in the field.

Rather than designing a study around the elements of PICO, PIRO is recommended in which “I” represents index test, and “R” represents reference standard.

The literature search for diagnostic topics can begin with using an existing systematic review that answers the research question, and is of satisfactory quality. The search date of that study can be used as a starting point to search forward in MEDLINE, Embase, Cochrane Database of Systematic Reviews, and PROSPERO. A grey literature search is also recommended.

When no appropriate systematic reviews are available, a search of primary studies will need to be conducted. Databases to be used in this case include MEDLINE, Embase, and Cochrane Library databases, as well as related conference proceedings. It is preferred that language limits are not used and that a grey literature search is carried out if deemed appropriate by researchers. The search should include Medical Subject Headings terms and keywords representing the diagnostic test and target condition. Adding terms to capture diagnostic accuracy or study design is not recommended, as indexing of this concept has not been proven to be dependable.

This is a second article in a series. First article: Yao X, Vella E. How to conduct a high-quality original study on a diagnostic research topic. Surg Oncol. 2017;26(3):305-9.

 

Limitations stated by the author(s): 

Authors noted that in order to keep the article brief, they did not fully explore the concepts of how to design strategies for different databases or peer review.

Limitations stated by the reviewer(s): 
Authors made general observations about the quality of diagnostic studies needing improvement without expanding on any evidence to support the claim.
Study Type: 
Review

Appraisal of: "Arber M, Garcia S, Veale T, Edwards M, Shaw A, Glanville JM. Performance of Ovid MEDLINE search filters to identify health state utility studies. Int J Technol Assess Health Care. 2017 Jan;33(4);472-80."

Reviewer(s): 
Short description: 

The retrieval of studies that report health state utility values (HSUVs) is an important aspect of information retrieval for HTA and economic model production.  This study first assessed three MEDLINE search filters designed by the York Health Economic Consortium (YHEC) to identify studies reporting HSUVs. The relative recall method was used to test the sensitivity of each filter. Three quasi gold standard (QSG) sets of relevant studies were compiled from reviews of studies reporting HSUVs. The first QSG (consisting of 294 records) was used to assess the performance of the three initial filters. The best performing of the three filters was further developed using the second QSG (139 records). Ultimately, three final search filters were validated using the third QSG (139 records). The first final search filter is sensitivity maximizing, with 95% sensitivity and a number needed to read (NNR) of 842. The second filter balances sensitivity and precision, and has a 92% sensitivity with an NNR of 502. The third filter is precision maximizing, with 88% sensitivity and an NNR of 383. Real world volume of retrieved records was also tested to illustrate the impact of using the three filters for three example health conditions. Having a range of sensitivity and precision options allows researchers to choose filters based on their search requirements. The authors believe that these are the first validated filters for retrieving HSURs. Search strategies for all three final filters are presented in the article.

Limitations stated by the author(s): 

The robustness of the relative recall method depends on the quality of the searches within each systematic review. Despite passing a quality checklist, some of the systematic reviews employed less than optimal search methods, which may have negatively affected study identification.

Also, a larger quasi-gold standard set may have improved the generalizability of the study findings. During filter development, several terms (utility loss, disutility, short form, SF-12) were removed to increase precision. However, during final validation, it was found that inclusion of these search terms would have retrieved an additional three records, revealing the difficulty in making filter decisions based on a small number of relevant records. These removed terms could be added to the filters to maximize sensitivity. Similarly, some terms added to the filters resulted from an analysis of a very small number of records. These revisions were made to increase sensitivity, but the missed records retrieved through these additions were outliers, and less likely to be representative of the wider body of relevant results. The addition of these terms may increase search result size, but potentially have minimal impact on the total number of relevant results retrieved.

The initial filters created by YHEC were developed through pragmatic and traditional subjective methods, rather than using objective filter design methods. Despite the pragmatic development, these strategies performed well in testing.

The final filters are designed to find reviews, cost-utility analyses and utility elicitation studies that report HSUVs, as well utility mapping studies. Since the filters do not distinguish between these different types of research, precision for research seeking only utility elicitation studies, for example, will be lower than the results reported in the article.

 

 

Limitations stated by the reviewer(s): 
These filters have been validated only for Ovid MEDLINE searches. Translations of these filters to other databases may not produce search results with similar sensitivity or NNR.
Study Type: 
Single study

Appraisal of: "Sampson M. Complementary approaches to searching MEDLINE may be sufficient for updating existing systematic reviews. J Clin Epidemiol 2016; 78: 108-115."

Short description: 

The aim of the article was to test whether the combined approach of a focused Boolean search paired with a second search using the similar articles feature of PubMed or support vector machine (SVM) can yield high recall with reasonable precision.

The general approach of a Boolean plus a ranking search is effective in MEDLINE retrieval for systematic reviews. Very high levels of identification of relevant MEDLINE records, with adequate precision, are possible using a focused Boolean search complemented by a document similarity or ranking method.

Limitations stated by the author(s): 

There are two limitations to our proposed strategy. Other databases should be searched in the unusual event that numerous studies, representing more than a small proportion of the total N, are not included in MEDLINE. Second, when it is important to find articles too new to be indexed by MEDLINE, systematic reviewers may wish to conduct a simple PubMed search limited to the nonindexed subsets.

Limitations stated by the reviewer(s): 
It is not possible to reproduce or use the support vector machine functionality.
Study Type: 
Single study

Appraisal of: "Boeker M., Vach W, Motschall E.. Time-dependent migration of citations through PubMed and OvidSP subsets: a study on a series of simultaneous PubMed and OvidSP searches. Stud Health Technol Inform. 2013;192:1196."

Reviewer(s): 
Short description: 

The aim of this study was to compare PubMed and OvidSP in terms of the timeliness of the different interfaces. The authors conclude OvidSP is updated within a period of two days.

Limitations stated by the author(s): 

No limits were discussed by the authors.

Limitations stated by the reviewer(s): 
This is a conference abstract so not enough detail is presented to ascertain the limitations.
Study Type: 
Single study

Appraisal of: "Fortier KJ, Kiss N, Tongbram V. What is the optimal search engine for results from Embase and MEDLINE: Ovid or Embase.com? Value Health 2013:16(3):A25."

Reviewer(s): 
Short description: 

The aim of this study was to compare OVID and Embase.com in terms of the results of searches and ease of searching. Using several searches the authors investigated differences in results and why any dispartieis exist. Ease of use was rates on a scale of 0 to 5. Although differences were found there were few if any relevant articles missed by either search interface. The authors conclude that there is no notable difference between OVID and Embase.com but Embase.com was rated easier to use overalll.

Limitations stated by the author(s): 

No limits were discussed by the authors.

Limitations stated by the reviewer(s): 
This is a conference abstract so not enough detail is presented to ascertain the limitations.

Appraisal of: Preston, L., et al. "Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking." Syst Rev 2015 4: 82.

Reviewer(s): 
Short description: 

A convenience sample of nine Health Technology Assessment (HTA) systematic reviews of diagnostic test accuracy, with 302 included citations, was analysed to determine the number and proportion of included citations that were indexed in and retrieved from MEDLINE and EMBASE. An assessment was also made of the number and proportion of citations not retrieved from these databases but that could have been identified from the reference lists of included citations.

287/302 (95 %) of the included citations were indexed across MEDLINE and EMBASE. The reviews’ searches of MEDLINE and EMBASE accounted for 85 % of the included citations (256/302). Of the forty-six (15 %) included citations not retrieved by the published searches, 24 (8 %) could be found in the reference lists of included citations. Only 22/302 (7 %) of the included citations were not found. 

Limitations stated by the author(s): 

This study used a small, non-random sample of diagnostic test accuracy systematic reviews. This was done for reasons of pragmatism: first, because the authors had full access to the search strategies and reference databases of these reviews and, second, because of the exploratory nature of this project. We also assumed that the vast majority of the included citations in the reviews were located through screening of titles, abstracts and full papers.

We have also assumed, because the number of studies missed by operating the proposed MEDLINE, EMBASE and reference tracking strategy is so small that the findings of the systematic reviews would not have been greatly affected by their omission. However, this is uncertain and can only be assessed statistically by excluding those particular studies from the many analyses reported in the reviews, although, as noted above, most of these reviews conducted narrative synthesis. Such an analysis is a major task to undertake retrospectively and has therefore not been completed in this exploratory study. Future work should test the findings of this small study in a larger, preferably prospective sample of systematic reviews from multiple institutions. If possible, statistical analysis should also be undertaken to quantify fully the impact of omitting any data from studies that might otherwise be missed.

Limitations stated by the reviewer(s): 
No further limitations were identified.
Study Type: 
Single study

Appraisal of: "Preston L, Carroll C, Gardois P, Paisley S, Kaltenthaler E. Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking."

Short description: 

This exploratory study analysed a convenience sample of nine Health Technology Assessments (HTA) reviews of diagnostic test accuracy (DTA) to identify which of the included citations were indexed on MEDLINE or EMBASE; to identify the number and proportion of citations that were retrieved by the search strategies with or without the addition of checking the reference lists of included studies.                     

Of the 302 included citations included in the nine reviews, 287 (95%) were indexed in MEDLINE and EMBASE. The reviews searches accounted for 256 (85%) of the included citations and a further 24 (8%) could be identified from the reference lists of included citations. 7% of the citations (22/302) were not found by searching or reference checking.

The proportion of citations identified using both searches of MEDLINE, and EMBASE and checking the reference lists resulted in the identification of 280/302 (93%) included studies. The authors suggest that there might be a case for restricting systematic review searches  of DTA studies to MEDLINE, EMBASE and the reference lists of included studies. 

Limitations stated by the author(s): 

The study’s pragmatic, convenience sample of a small number of published DTA reviews was intended as an exploratory analysis only. The  authors did not conduct an analysis to assess the impact of excluding the unidentified studies on the conclusions of the HTAs, so the results of the study are uncertain. 

Limitations stated by the reviewer(s): 
Six of the nine studies in the sample used a published filter for diagnostic studies, which is not recommended by Cochrane. This may have impacted the sensitivity of the searches and the sample not be representative of other DTA reviews.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Pohl, S. Boolean and Ranked Information Retrieval for Biomedical Systematic Reviewing [thesis]. Department of Computer Science and Software Engineering: University of Melbourne; 2012.

Short description: 

The aim of the study was to devise and examine ranked retrieval methods with regard to their applicability to the search tasks involved in the preparation and update of systematic reviews. A ranked retrieval alternative in the form of the p-norm extended Boolean retrieval (EBR) model was shown to provide advantages over the current Boolean approach, but to also satisfy important task specific requirements. Using this retrieval method, a greater number of relevant studies than with the current Boolean approach can be identified for inclusion in systematic reviews, reducing the dependence on potentially biased secondary search methods. The method is close to current practice and hence requires minimal changes to the current process, which should facilitate its adoption.

Limitations stated by the author(s): 

"One caveat with this experimental setup is that the queries could possibly be biased by knowledge about the documents in the test set. Hence, absolute performance on the test set might not be reliable, but this setup still allows for relative comparison of the approach because both the optimized queries as well as the baseline would have this advantage." (p. 154)

Limitations stated by the reviewer(s): 
No additional limitations were detected by the reviewers.
Study Type: 
Single study
Supplemental publications to the study: 
Syndicate content