Search filters

Scope: use this with methods papers on search filter design, comparison of search filter performance

Appraisal of: "Arber M, Garcia S, Veale T, Edwards M, Shaw A, Glanville JM. Performance of Ovid MEDLINE search filters to identify health state utility studies. Int J Technol Assess Health Care. 2017 Jan;33(4);472-80."

Reviewer(s): 
Short description: 

The retrieval of studies that report health state utility values (HSUVs) is an important aspect of information retrieval for HTA and economic model production.  This study first assessed three MEDLINE search filters designed by the York Health Economic Consortium (YHEC) to identify studies reporting HSUVs. The relative recall method was used to test the sensitivity of each filter. Three quasi gold standard (QSG) sets of relevant studies were compiled from reviews of studies reporting HSUVs. The first QSG (consisting of 294 records) was used to assess the performance of the three initial filters. The best performing of the three filters was further developed using the second QSG (139 records). Ultimately, three final search filters were validated using the third QSG (139 records). The first final search filter is sensitivity maximizing, with 95% sensitivity and a number needed to read (NNR) of 842. The second filter balances sensitivity and precision, and has a 92% sensitivity with an NNR of 502. The third filter is precision maximizing, with 88% sensitivity and an NNR of 383. Real world volume of retrieved records was also tested to illustrate the impact of using the three filters for three example health conditions. Having a range of sensitivity and precision options allows researchers to choose filters based on their search requirements. The authors believe that these are the first validated filters for retrieving HSURs. Search strategies for all three final filters are presented in the article.

Limitations stated by the author(s): 

The robustness of the relative recall method depends on the quality of the searches within each systematic review. Despite passing a quality checklist, some of the systematic reviews employed less than optimal search methods, which may have negatively affected study identification.

Also, a larger quasi-gold standard set may have improved the generalizability of the study findings. During filter development, several terms (utility loss, disutility, short form, SF-12) were removed to increase precision. However, during final validation, it was found that inclusion of these search terms would have retrieved an additional three records, revealing the difficulty in making filter decisions based on a small number of relevant records. These removed terms could be added to the filters to maximize sensitivity. Similarly, some terms added to the filters resulted from an analysis of a very small number of records. These revisions were made to increase sensitivity, but the missed records retrieved through these additions were outliers, and less likely to be representative of the wider body of relevant results. The addition of these terms may increase search result size, but potentially have minimal impact on the total number of relevant results retrieved.

The initial filters created by YHEC were developed through pragmatic and traditional subjective methods, rather than using objective filter design methods. Despite the pragmatic development, these strategies performed well in testing.

The final filters are designed to find reviews, cost-utility analyses and utility elicitation studies that report HSUVs, as well utility mapping studies. Since the filters do not distinguish between these different types of research, precision for research seeking only utility elicitation studies, for example, will be lower than the results reported in the article.

 

 

Limitations stated by the reviewer(s): 
These filters have been validated only for Ovid MEDLINE searches. Translations of these filters to other databases may not produce search results with similar sensitivity or NNR.
Study Type: 
Single study

Appraisal of: "Preston L, Carroll C, Gardois P, Paisley S, Kaltenthaler E. Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking."

Short description: 

This exploratory study analysed a convenience sample of nine Health Technology Assessments (HTA) reviews of diagnostic test accuracy (DTA) to identify which of the included citations were indexed on MEDLINE or EMBASE; to identify the number and proportion of citations that were retrieved by the search strategies with or without the addition of checking the reference lists of included studies.                     

Of the 302 included citations included in the nine reviews, 287 (95%) were indexed in MEDLINE and EMBASE. The reviews searches accounted for 256 (85%) of the included citations and a further 24 (8%) could be identified from the reference lists of included citations. 7% of the citations (22/302) were not found by searching or reference checking.

The proportion of citations identified using both searches of MEDLINE, and EMBASE and checking the reference lists resulted in the identification of 280/302 (93%) included studies. The authors suggest that there might be a case for restricting systematic review searches  of DTA studies to MEDLINE, EMBASE and the reference lists of included studies. 

Limitations stated by the author(s): 

The study’s pragmatic, convenience sample of a small number of published DTA reviews was intended as an exploratory analysis only. The  authors did not conduct an analysis to assess the impact of excluding the unidentified studies on the conclusions of the HTAs, so the results of the study are uncertain. 

Limitations stated by the reviewer(s): 
Six of the nine studies in the sample used a published filter for diagnostic studies, which is not recommended by Cochrane. This may have impacted the sensitivity of the searches and the sample not be representative of other DTA reviews.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 2007; (2): MR000001.

Short description: 

Aim of the Cochrane review was to compare the technique of handsearching with electronic searching to identify RCT.

Thirty-four studies were included. Handsearching identified between 92% to 100% of the total number of reports of randomized trials found in the various comparisons in this review. Searching MEDLINE retrieved 55%, EMBASE 49% and PyscINFO 67%. The retrieval rate of the electronic database varied depending on the complexity of the search. The Cochrane Highly Sensitive Search Strategy (HSSS) identified 80% of the total number of reports of randomized trials.

Conclusion of the authors is, that hand searching has a valuable role in identifying reports of RCT for inclusion in SR of health care interventions.

Particularly important in identifying trials reported as abstracts, letters and those published in languages other than English.

Limitations stated by the author(s): 

In this review, no attempts were made to assess the importance of the trials missed by either handsearching or electronic searching. There is evidence to suggest that only around half of all trials reported as abstracts are subsequently published in full and that published trials may show a larger treatment effect than ’grey’ trials (for example those published as conference abstracts). Further research is needed to assess the importance of those trials missed by either method of searching.

Limitations stated by the reviewer(s): 
Main problem is, that the review is outdated. The last search was in 2002. Handsearch was mainly compared to “old” 3 phase Highly Sensitive Search Strategy filter.
Study Type: 
Review

Appraisal of: Mattioli S, Farioli A, Cooke RMT, Baldasseroni A, Ruotsalainen J, Placidi D, et al. Hidden effectiveness? Results of hand-searching Italian language journals for occupational health interventions. Occup Environ Med 2012;69(7):522-524.

Short description: 

The study compared the yield of hand-searching with optimised electronic search strategies in retrieving occupational health (OH) intervention studies published in a language other than English.

The authors systematically hand-searched and screened reports of OH intervention studies published in Italian in peer-reviewed scientific journals between 1990 and 2008. The Cochrane Occupational Safety and Health Review Groups (OSHRG) most sensitive search string retrieved all 16 articles published in the Italian language journals that were indexed in MEDLINE.

These findings suggest that a sensitive electronic search strategy may be able to find most of the OH interventions published in languages other than English that are indexed in MEDLINE.

Limitations stated by the author(s): 

No limitations were stated by the authors.

Limitations stated by the reviewer(s): 
The authors hand searched only 3 MEDLINE-indexed journals.
Study Type: 
Single study

Appraisal of: Arber M, Garcia S, Veale T, Edwards M, Shaw A, Glanville J. Sensitivity of a search filter designed to identify studies reporting health state utility values [poster].

Reviewer(s): 
Short description: 

The retrieval of studies that report health state utility values (HSUVs) is an important aspect of information retrieval for HTA and economic model production.  The York Health Economic Consortium (YHEC) created and tested three search filters designed to identify studies reporting HSUVs. The relative recall method was used to test the sensitivity of each filter. A quasi-gold standard set of 293 studies was compiled by identifying a sample of systematic reviews of studies reporting HSUVs. All three filters tested retrieved the same 267 citations from the quasi-gold standard (0.91 sensitivity). Filter 3 had the highest precision at 0.003. The 26 studies missed by the filters did not explicitly refer to HSUVs within the citations. Inclusion of the “quality of life” subject heading would have retrieved 20 of those studies, but also increased overall results by 116%. Further analysis of the 26 records missed by the search filters is planned to try to improve sensitivity without a significant loss in precision. The search strategy for the most precise search filter is included in the poster.

Limitations stated by the author(s): 

The retrieval of studies that report health state utility values (HSUVs) is an important aspect of information retrieval for HTA and economic model production.  The York Health Economic Consortium (YHEC) created and tested three search filters designed to identify studies reporting HSUVs. The relative recall method was used to test the sensitivity of each filter. A quasi-gold standard set of 293 studies was compiled by identifying a sample of systematic reviews of studies reporting HSUVs. All three filters tested retrieved the same 267 citations from the quasi-gold standard (0.91 sensitivity). Filter 3 had the highest precision at 0.003. The 26 studies missed by the filters did not explicitly refer to HSUVs within the citations. Inclusion of the “quality of life” subject heading would have retrieved 20 of those studies, but also increased overall results by 116%. Further analysis of the 26 records missed by the search filters is planned to try to improve sensitivity without a significant loss in precision. The search strategy for the most precise search filter is included in the poster.

Limitations stated by the reviewer(s): 
Ovid MEDLINE was the only database searched for this study.
Study Type: 
Single study

Appraisal of: Frazier JJ, Stein CD, Tseytlin E, Bekhuis T. Building a gold standard to construct search filters: a case study with biomarkers for oral cancer. J Med Libr Assoc. 2015;103(1):22-30.

Reviewer(s): 
Short description: 

This paper describes the process involved in the development of a gold standard dataset in oral squamous carcinoma (OSCC) against which future filters for prognostic biomarkers could be tested. Given the lack of filters for biomarkers in OSCC, the authors set out to produce a set of key studies that a filter developed for use with other databases should retrieve. They designed a MEDLINE search strategy to retrieve prognostic studies using the prognostic filter developed by the Haynes group. An annotating system was used to review the references retrieved by the search. Two reviewers independently assessed the studies and a calibration process for the results is described. The agreement about relevance across reviewers was measured by Cohen’s kappa statistic, achieving a high level of agreement. This system for developing gold standards is likely to be generalizable to other subject domains.

Limitations stated by the author(s): 

Studies were screened using only the title and abstract, and a small percentage of retrieved studies didn’t include an abstract.

Searches were only conducted in MEDLINE.

Limitations stated by the reviewer(s): 
The search strategy used to identify studies of prognostic biomarkers for OSCC incorporated an existing search filter for prognostic studies without any discussion or reference to if, and how, this had been validated. The authors limited to studies indexed with the MeSH Humans tag. A better approach would have been to carefully exclude animal studies. The searches were conducted on a small 5 year segment of MEDLINE which may not be appropriate for other topic areas
Study Type: 
Single study

Appraisal of: Sampson M, Zhang L, Morrison A, Barrowman NJ, Clifford TJ, Platt RW, et al. An alternative to the hand searching gold standard: validating methodological search filters using relative recall. BMC Med Res Methodol. 2006;6(33)

Reviewer(s): 
Short description: 

This paper explores the relative recall technique to develop a gold standard for use in evaluating search filters. Filters or ‘hedges’ are search strategies used in conjunction with a topic search  to focus on those records which are the focus of the filter. Validating a filter requires the existence of a ‘gold standard’ of records known to be of the study type required. However, developing a gold standard by handsearching can be time consuming. The authors identified 105 Cochrane Reviews that had used the Cochrane Highly Sensitive Search Strategy to identify studies.  The included studies in those reviews were gathered to form a gold standard. Three variants of the Highly Sensitive Search Strategy (HSSS) filter were tested in terms of their ability to retrieve gold standard records in MEDLINE.  Sensitivity ranging from 0.98 to 0.91 was achieved. 

Limitations stated by the author(s): 

The feasibility of this method will depend on the number of available systematic reviews using the methodology of interest.  The authors acknowledge criticisms of sensitivity (recall) and precisions as measures of search performance.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Single study

Appraisal of: Beynon R, Leeflang MM, McDonald S, Eisinga A, Mitchell RL, Whiting P, Glanville JM. Search strategies to identify diagnostic accuracy studies in MEDLINE and EMBASE. Cochrane Database Syst Rev 2013, Issue 9.

Reviewer(s): 
Short description: 

This review assessed the performance (sensitivity, specificity and precision) of 70 filters (described in 19 studies) designed to identify diagnostic test accuracy studies for use with either the MEDLINE or Embase database. The filters were appraised for potential bias in terms of use of a diagnostic test accuracy search strategy; choice of gold standard; method of validation. The ISSG Search Filter Appraisal Checklist was used to assess quality. None of the filters tested met the pre-specified levels of sensitivity > 90% and precision >10%. Using methodological search filters as the sole method of identifying studies for systematic reviews of diagnostic test accuracy is not recommended.

Limitations stated by the author(s): 

A search for unpublished search filters was not conducted. The authors acknowledged the possibility of bias introduced by the studies that evaluated filters but did not replicate them in the paper. 

Limitations stated by the reviewer(s): 
No limitations have been identified by the reviewers.
Study Type: 
Review
Related Chapters: 

Appraisal of: Golder S, Wright K, Rodgers M. Failure or success of search strategies to identify adverse effects of medical devices: a feasibility study using a systematic review. Syst Rev 2014;3:113.

Short description: 

A systematic review of the safety of recombinant human bone morphogenetic protein-2 (rhBMP-2) for spinal fusion was used as a case study to investigate whether adverse effects terms were prevalent in the bibliographic records of the included studies. The aim of the study was to assess whether the development of a search filter for medical devices would be feasible.

Out of the 82 publications (49 studies) included in the systematic review, 51 were indexed in MEDLINE and 55 in Embase. The results showed that 94% of the MEDLINE records and 95% of the Embase records contained at least one adverse effects related term. However, the records contained a wide range of adverse effects terms, and no single term was able to retrieve the majority of the included publications. In addition, the most successful search terms were different from search terms shown to be most successful when searching for adverse drug effects. The study concluded that the development of adverse effects search filters for medical devices is feasible, but that such filters should be developed using a large set of relevant records in order to identify the optimal combination of search terms.

Limitations stated by the author(s): 

The main limitation to this study is that because only one systematic review was used as a case study, the generalisability of the study results is limited. This case study is also of a medical device with a pharmaceutical component, while a case study of a different kind of a medical device might have given different results.

In addition, unlike many other HTAs and systematic review authors, the study authors were able to obtain unpublished data directly from the manufacturer. They also included an unusually high number of conference abstracts and multiple publications for the same study in their review. This might have influenced the findings of this study.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Single study

Comments from the authors:

Appraisal of: Beale S, Duffy S, Glanville J, Lefebvre C, Wright D. McCool R. Varley D, Boachie C, Fraser C, Harbour J et al. Choosing and using methodological search filters: searchers' views. Health Info Libr J. 2014;31(2):133-47.

Reviewer(s): 
Short description: 

The paper reports on a survey of information specialists working in, or for, UK NICE and a questionnaire advertised to a wider audience via a range of email lists, that explored searchers’ use of filters, how they select information filters and what information would help them to make those selections. It found that search filters are mainly used to reduce large results sets. Less technical information about filters would help information specialists to making choices as would ratings, and more information about how the filter was validated and who created it.

Limitations stated by the author(s): 

No limitations stated.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Single study
Related Chapters: 
Syndicate content