Search filters

Scope: use this with methods papers on search filter design, comparison of search filter performance

Appraisal of: Beale S, Duffy S, Glanville J, Lefebvre C, Wright D. McCool R. Varley D, Boachie C, Fraser C, Harbour J et al. Choosing and using methodological search filters: searchers' views. Health Info Libr J. 2014;31(2):133-47.

Reviewer(s): 
Short description: 

The paper reports on a survey of information specialists working in, or for, UK NICE and a questionnaire advertised to a wider audience via a range of email lists, that explored searchers’ use of filters, how they select information filters and what information would help them to make those selections. It found that search filters are mainly used to reduce large results sets. Less technical information about filters would help information specialists to making choices as would ratings, and more information about how the filter was validated and who created it.

Limitations stated by the author(s): 

No limitations stated.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Petrova M, Sutcliffe P, Fulford KW, Dale J. Search terms and a validated brief search filter to retrieve publications on health-related values in Medline: A word frequency analysis study. J Am Med Inform Assoc 2012; 19(3): 479–488.

Short description: 

The aim of the article is to present search terms (keywords) and a “brief” search filter to identify information on health-related values in Medline developed by using word frequency analysis. The purpose of the project is to support broad-scoping searches by “a brief search filter (≤20 lines) of high precision (≥67%) and acceptable sensitivity (≥67%) which can be used […] in generic medical and health databases (e.g., Medline, Embase, or Cinahl) and across a range of topics (e.g., health conditions, health settings, health interactions, etc.)”.

The article provides extensive information on the background as on health-related values defined by the authors and on word frequency analyses. The application of this analysis in the conditions diabetes, dementia, schizophrenia and obesity resulted in a small overlap of relevant MeSH and free-text search terms (9 of 124 different MeSH terms (7.3%) and 4 of 144 different words (2.8%) were identified in all 4 conditions among the 50 “best” MeSH respective words in each condition). Applying the developed search filter in hypersensitivity and dentistry resulted in (authors defined) sensitivity of 70.1% (hypersensitivity) and 47.1% (dentistry), and precision of 63.6% (hypersensitivity) and 82.6% (dentistry). 

Limitations stated by the author(s): 
  • A number of ‘subjective’ additions to the objectively derived filter improved sensitivity and precision.
  • The brief values filter can be applied productively to other topics in Medline, but there is instability in its performance.
  • The study covers a limited period, limited number of topics, a single database, and a single approach to compiling its corpus of citations.
  • The study lacked well-articulated criteria for distinguishing cases (of values publications) from non-cases. The criteria used may have been overly inclusive.
  • This study did not have the capacity for both theoretical and empirical work of adequate depth.
  • The reported frequencies for text words did not necessarily reflect the number of abstracts in which a word appears.
  • Data cleaning was performed on text words that had a 100% precision, but not for the remainder.
  • Limit to the very field of designing objective search strategies on the basis of word frequency analysis: The field may be lagging behind advances in medical informatics. Some of the methods applied in this study may need to be superseded, including methods that have improved on standard practices of objective search strategy design.

 

 

Limitations stated by the reviewer(s): 
The theoretical concept of values defined by the authors is not fully clear and different from those used in ethical analysis in HTA. Besides the many limitations already stated by the authors there are some significant methodological shortcomings, and the results don’t fulfill the needs of systematic reviews in HTA regarding sensitivity in a sufficient manner. The proposed search filter is clearly not this universal solution as claimed in the authors’ objectives. As well the results indicate that the method of word frequency analysis itself fails in the field of values. The shortcomings in detail: The glossary definitions are not always identical with the common ones in Information Retrieval. Authors’ objective: No reasoning for the line number (and no meaning on the length of each line) and sensitivity and precision. The validation, in particular the external validation, applied by the authors does not fulfill criteria of an evidence-based validation. Building-up the derivation dataset is not reported in a transparent and reproducible manner. As the conditions were just defined by one MeSH term each the sensitivity of the results may be low. Calculating sensitivity of the search filter by using these datasets seems to be questionable (and a self-fulfilling prophecy).
Study Type: 
Single study

Appraisal of: Jenkins, M. Evaluation of methodological search filters—a review. Health Information and Libraries Journal, 21: 148-163

Reviewer(s): 
Short description: 

This is an overview of approaches to methodological search filter development that identified   four stages in the process – search term selection, identification of a gold standard, evaluation, and validation. One of the key findings is the lack of consistency in how search filters are tested and validated. The project also highlighted the need for filter developers to improve on their reporting and to offer more guidance to information specialists and clinicians in using search filters. An appraisals checklist, based on the results, of the review is presented.

Limitations stated by the author(s): 

The author briefly refers to 1) the implications for search filters if searching full text sources rather than bibliographic databases and 2) the lack of research into the impact of search filters on the time, cost and impact on the results of the reviews and their use among clinicians.

Limitations stated by the reviewer(s): 
The author stated that the appraisals checklist presented in the paper requires refinement after consulting with experts. It is not clear whether this was ever completed.
Study Type: 
Review
Related Chapters: 

Appraisal of: Bak G, Mierzwinski-Urban M, Fitzsimmons H, Morrison A, Maden-Jenkins M. A pragmatic critical appraisal instrument for search filters: introducing the CADTH CAI. Health Info Libr J. 2009 Sep;26(3):211-9

Reviewer(s): 
Short description: 

This paper describes the search for a pragmatic critical appraisal instrument (CAI) that experienced systematic review searchers can use in the selection of robust search filters. A focused search of the literature identified a single candidate - the critical appraisal tool developed by Michelle Jenkins. Testing of this showed that its use requires an in-depth understanding of search filter methodology and statistics that not all information specialists may possess. In light of these findings, the authors substantially revised the tool to consist of 6 domains and 10 closed questions. Instructions about how to use the tool are provided. It features a key question about whether the filter being appraised is up to date.

Limitations stated by the author(s): 

The authors acknowledged that the CADTH CAI is a pragmatic tool for use by information specialists and suggested that research methodologists would be better served using the original, more sophisticated, appraisal tool developed by Jenkins. Even less experienced searchers with little knowledge of controlled vocabularies and search filter methodologies could potentially find the CADTH CAI difficult to use. They also highlighted the lack of research demonstrating whether the most relevant studies are included in the smaller results sets obtained when a filter is incorporated into the search strategy.

Limitations stated by the reviewer(s): 
No additional limitations were detected by the reviewers.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Glanville J, Bayliss S, Booth A, Dundar Y, Fernandes H, Fleeman N D, Foster L, Fraser C, Fry-Smith A, Golder S, Lefebvre C, Miller C, Paisley S, Payne L, Price A and Welch K. So many filters, so little time: the development of a search

Reviewer(s): 
Short description: 

Study describes the development and testing of a tool used to assess the quality of search filters designed to retrieve records for studies with a specific research design. The resulting tool comprises a search filter appraisal checklist that reports both filter design methods and search performance i.e. sensitivity and precision. As well as helping information specialists select the most appropriate filter, it can also be used by researchers developing search filters by indicating what aspects of filter development should be reported to ensure that the methodology can be successfully evaluated.

Limitations stated by the author(s): 

The performance of the checklist has not been assessed using independent assessors and a range of filters.

Subsequent to the development of the issg tool, CADTH produced a critical appraisal and ranking tool. No comparison between the two has been conducted to date.

The issg checklist has restricted its focus to search filters designed to retrieve studies with a specific research type (e.g. RCT or diagnostic test accuracy study). There has not yet been any exploration of whether any of the processes could also be applied to other search filters.

Limitations stated by the reviewer(s): 
Since 2008 when this project was undertaken there has been increased interest in the use of text mining techniques as an alternative method of identifying and retrieving relevant information from large databases. Although still at the research stage currently, if these methods become well established the need for search filters will decrease over time.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Kaunelis D. When everything is too much: development of a CADTH narrow economic search filter. Poster presented at: 2011 CADTH Symposium. 2011 Apr 3-5; Vancouver, BC.

Reviewer(s): 
Short description: 
This poster reported an investigation of the performance of filters to find a gold standard set of economic evaluations with a focus on maximizing the precision of the filter. The best performing filter was further adapted and then tested for performance on the same gold standard and using a series of queries. The ‘CADTH narrow search filter’ found approximately 75% of the gold standard records with better precision than other filters, which tended to maximize sensitivity. The author suggested that this filter is useful for rapid reviews, to identify critiques of models, and to carry out high level reviews of an intervention or reviews of an indication. The filter mostly comprised title and abstract terms and should therefore be usable across a range of databases. It is available for Ovid and for PubMed interfaces.
Limitations stated by the author(s): 
No limitations were identified.
Limitations stated by the reviewer(s): 
The filter was developed and tested in the same gold standard and so will tend to over-perform. The performance of the filter should be tested in other gold standards to see whether performance from the test set is maintained. The sensitivity and precision of the search filters was not provided.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Papaioannou D, Brazier J, Paisley S. Systematic searching and selection of health state utility values from the literature. Value Health 2013;16(4):686-695.

Reviewer(s): 
Short description: 
Health state utility values (HSUVs) are used in populating decision-analytic models in health technology assessments. This article presented guidance on how to systematically search the published literature for HSUVs to ensure the use of unbiased HSUV data within cost-effectiveness analyses. Issues in defining the scope of the review were first examined. The authors then presented an extensive list of relevant search terms (including MeSH terms) that can be used in HSUV searches. Also covered were which electronic databases to search, as well as the use of supplementary search techniques, including reference list checking and contact with experts. The article concluded by discussing how to review and select studies that contain HSUV data, how to assess the quality of included studies, and how to perform HSUV data extraction.
Limitations stated by the author(s): 
None stated.
Limitations stated by the reviewer(s): 
The authors used an osteoporosis case study to demonstrate their methods. These approaches may not be generalizable to other health states. The authors acknowledged issues in using sensitive vs. precise search strategies in electronic databases for HSUV data, but they didn’t explicitly state search techniques for adapting a sensitive search into a precise one. Also, the search filter presented in the supplemental materials has not been validated.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Glanville J, Paisley S. Identifying economic evaluations for health technology assessment. Int J Technol Assess Health Care 2010;26(4):436-440.

Reviewer(s): 
Short description: 
This study surveys the current evidence base for search methods and resources for retrieving economic evaluations in the context of health technology assessment (HTA) work. To find economic evaluations, specific economic evaluation databases and larger biomedical databases must be searched to obtain the most comprehensive sets of results. In addition to a list of key databases, strategies for searching both types of databases are presented, along with other general database search tips. Evidence on the performance of several economic evaluation search filters is summarized, with a call for further evidence-generating research in this area.
Limitations stated by the author(s): 
The authors state that there is a lack of evidence supporting best practice in retrieving economic evaluations, such as efficient grey literature searching, the development of search filters, and “comparative yield” of different search approaches and resources.
Limitations stated by the reviewer(s): 
Non-database sources appear to have been surveyed in the study, but the authors do not elaborate on any identified.
Study Type: 
Review
Related Chapters: 

Appraisal of: Glanville J, Paisley S. Chapter 7: Searching for evidence for cost-effectiveness decisions. In: Shemilt I, Mugford M, Marsh K, Donaldson C. (editors). Evidence-based decisions and economics: health care, social welfare, education and crimin

Reviewer(s): 
Short description: 
This textbook chapter summarizes the available published evidence on the approaches and methods for literature searching in support of both economic evaluations and decision models in health care. For economic evaluations, a range of database resources and search filter designs are presented. For decision models, the inappropriateness of the traditional PICO search approach is discussed. The authors provide an outline of the unique types of information needs and resources required for these models. Systematic approaches and transparency are emphasized, and an understanding of the development and purpose of both economic evaluations and decision models is encouraged.
Limitations stated by the author(s): 
The chapter authors focus their analysis to health care related economic evaluations and decision models only. Searches on other peripheral topics in the health economics area, such as burden of illness studies, health resources use, etc. are deferred.
Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewer.
Study Type: 
Review
Related Chapters: 

Appraisal of: McKinlay RJ, Wilczynski NL, Haynes RB, Hedges Team. Optimal search strategies for detecting cost and economic studies in EMBASE. BMC Health Serv Res 2006;6:67.

Short description: 
This study reported the development of a search strategy for cost and economics studies in Embase. The gold standard was created by handsearching 55 journals for the year 2000. The authors provide strategies to find studies about costs and about economic studies broadly defined. A cost strategy with a sensitivity of 100% and precision of 8.6% is presented along with an economic studies strategy of 100% with 1.4 % precision. Best compromise strategies are offered for costs (sensitivity 98.4% and precision 18.2%) and economics (sensitivity 96.8% and precision 4.3%)
Limitations stated by the author(s): 
Precision will be lower when searching the entire Embase database. Multivariate statistical techniques might yield better performing strategies but authors did test a logistic regression approach.
Limitations stated by the reviewer(s): 
55 journals were chosen for frequency of yield and these may be different to the wider range of journals which publish cost and economics studies. The handsearched records are for one year only. High impact factor journal were chosen and these may be different to other journals perhaps in terms of encouraging better study reporting. The authors have a broad definition of economics which means their strategy may not be optimized to find economic evaluations.
Study Type: 
Single study
Related Chapters: 
Syndicate content