Search strategy development

Author(s): 
Last revised: 
2019-03-25

Introduction

The Cochrane Information Retrieval Methods Group have published an evidence-based chapter on search methods for the Cochrane Handbook (1), which provides the basis for this summary alongside guidance produced by the Centre for Reviews and Dissemination (2) and the Agency for Healthcare Research and Quality (AHRQ) (3). 

The revised and updated searching chapter of the Cochrane Handbook is in preparation. To avoid duplication of effort with the development of the Cochrane Handbook, appraisals have not been prepared for studies in this chapter. Once the revised Cochrane Handbook is available it will be used to update this chapter.

Sensitivity and precision

In order to retrieve as many studies relevant to the review as possible, and to compensate for the limitations of information source records and indexing, search strategy development and construction for systematic reviews (SRs) conventionally aims for sensitivity (1). Increasing the sensitivity of a search increases the possibility of identifying all relevant studies, but also tends to reduce precision because the number of irrelevant results is increased (1, 2). Sampson et al examined a cross section of 94 health related SRs that reported the flow of bibliographic records through the review process and found that search precision of approximately 3% was typical (4). The number of results retrieved, which therefore must be screened against eligibility criteria, has implications for the resources required to conduct a SR. This trade-off between sensitivity and precision should be acknowledged and discussed with the wider review team, and an appropriate balance sought within the context of the resources available.

The emphasis on search strategy sensitivity over precision typically reflects the context of SRs of quantitative research on clinical interventions. This emphasis may not be the same in searches developed for different purposes within the health technology assessment (HTA) context.  In the context of qualitative SRs or qualitative evidence syntheses for example, there is discussion as to whether these type of reviews share the same need as SRs of quantitative research for ‘comprehensive’, ‘exhaustive’ bibliographic database searches (5). Guidance from the Cochrane Qualitative and Implementation Methods Group recommends that search procedures in the context of qualitative evidence synthesis should generally privilege specificity over sensitivity (6).  Similarly, in the context of conducting a search to inform an ‘evidence-map’ (where an overview of the extent, nature and characteristics of a research area is of interest) research has indicated that less sensitive searches may be appropriate.  In a study which compared a ‘highly sensitive’ search strategy with a ‘highly specific’ search strategy for an evidence-mapping exercise on diabetes and driving to inform clinical guidance development, the authors reported that the results of the ‘highly specific’ search would have been sufficient for answering the research question (7). The authors concluded that using highly specific instead of sensitive search strategies is “fully adequate for evidence maps with the aim of covering mainly the breadth rather than depth of a research spectrum”.  Recent research has also suggested that the conventional approach to search methodology – with its focus on sensitive searches of bibliographic databases as the primary method of study identification – may not be optimal for some SRs on complex topics, or in areas other than clinical health.  In the context of a SR to evaluate the health benefits of environmental enhancement and conservation activities, Cooper et al (8) compared an approach led by searches of bibliographic databases with an approach led by supplementary search methods.  The authors found that extensive bibliographic database searching was of limited value in terms of contribution to synthesis, but that grey literature searching was valuable and identified studies that made unique contributions to both the quantitative and qualitative synthesis. The authors concluded that the approach led by supplementary search methods (where the primary methods of study identification were grey literature searching and contacting experts, supplemented by bibliographic database searches which emphasised precision over sensitivity) was valid when compared with the conventional approach.

Structuring the search

The Cochrane Handbook suggests a search strategy should be structured around the main concepts being examined by the review.  For reviews of interventions, this can be expressed using PICO (Patient (or Participant or Population), Intervention, Comparison and Outcome).  It is usually seen as undesirable to include all elements of the PICO in the search strategy as some concepts are often poorly described or non-existent in the title and abstract of a database record or the assigned indexed terms. For reviews of many interventions a search may reasonably be comprised of the population, intervention, and a study design filter (1) if appropriate. A validated search filter is recommended where one exists for the concept of interest (3). In some topic areas, for example complex interventions, where many of the concepts are particularly ill-defined, it may be preferable to use a broader search strategy (such as searching only for the population or intervention) and increase the resources allocated to sifting records (2). 

Alternatives to the PICO framework have also been evaluated for searches in some fields; examples include the SPIDER tool to structure searches for qualitative and mixed methods research (9) and the BeHEMoTh tool to structure searches for theory (10). In a structured methodological review on searching for qualitative research, Booth lists 11 different notations for use in this context (including PICO, SPIDER and BeHEMoTh), but states that, as with quantitative reviews, there is little empirical data to support the merits of question formulation (5). In a SR published in 2018, Eriksen and Frandsen investigated whether the use of the PICO model as a search strategy tool affected the quality of a literature search (11). The authors found only three studies which assessed the effect of using the PICO model versus other available models or unguided searching. The authors concluded that no solid conclusions could be drawn about the effect of using the PICO model on the quality of the literature search.

Selecting search terms

The Cochrane Handbook recommends that in order to identify as many relevant records as possible, search strategies should combine subject headings selected from the database’s controlled vocabulary or thesaurus (with appropriate “explosions”) and a wide range of free-text terms (1). The choice of free-text terms should include consideration of synonyms, related terms and variant spellings. 

Methods for identifying search terms have traditionally included techniques such as checking the bibliographic records of known relevant studies, consulting topic experts and scanning database subject indexing guides (3), but alternative methods have also been proposed.

Bramer et al evaluated a structured approach where thesaurus terms and synonyms for title / abstract searching were collected from the Emtree thesaurus, combined into a search strategy, and then tested for completeness using an ‘optimization method’ (12). This method involved identifying articles indexed with identified Emtree thesaurus terms but which did not include the synonyms already used in the search strategy in their title or abstract. Relevant terms from the titles and abstracts of these records were then added to the search strategy, and their added value was evaluated in discussion with the researcher who had requested the search. Further optimisation was done by reversing this process: looking for new thesaurus terms in articles where the titles and/or abstracts contained one of the identified synonyms but lacked the thesaurus terms already identified.  The authors concluded that the method creates opportunities for faster development of SR search strategies that find more relevant studies than other methods with equivalent search precision.

Text mining is a rapidly developing tool with potential application in a range of tasks associated with the production of SRs, including the identification of search terms (2). AHRQ published a review on the use of text-mining tools as an emerging methodology within SR processes, including the literature search (13). The aim of the AHRQ project was to provide a ‘snapshot’ of the state of knowledge, rather than an in-depth assessment. The review referred to 12 studies where text-mining tools were used for development of ‘topic’ search strategies and identified several general approaches to development.  These included assessing word frequency in citations (using tools such as PubReminer or EndNote) and automated term extraction (using tools such as Termine).  The review reported that all of the identified studies found benefit in automating term selection for SRs, especially those comprising large unfocused topics. The AHRQ review made no conclusions which were specific to the use of text-mining tools for the literature search process. The general conclusions on the use of text-mining for SR processes were that text-mining tools appeared promising, but further research was warranted.

Studies cited in the AHRQ review included a study by O'Mara-Eves et al which evaluated whether additional search terms for the topic of ‘community engagement’ were generated when using the text-mining data-extraction tool Termine (14) in addition to typical search development techniques.  The study authors reported that although in many cases the terms generated by text-mining had already been identified by the reviewers as relevant, text-mining did reveal some useful synonyms and terms associated with the topic that had not previously been considered.  The study authors stated that the text-mining approach studied should never be used on its own but alongside usual search development processes. The authors concluded that text mining helped to identify relevant search terms for a broad topic that was inconsistently referred to in the literature.

The AHRQ review also cited 2 studies published by researchers at the German HTA agency IQWiG. In the first study, the authors proposed an 'objective approach' to strategy development using text analysis methods (15). The authors argued that this method ensures the process of selecting search terms is transparent and reproducible and allows a searcher with little specialist knowledge of the search topic to make decisions on the inclusion of terms that are informed by evidence.  In the second study the authors aimed to validate the ‘objective approach’, and concluded that it was noninferior to the standard 'conceptual approach' (16). Subsequent correspondence on this publication (17, 18) and the authors' responses to this correspondence (19, 20) debated the study’s conclusions and the strengths and limitations of the methods used for this research. Since the publication of the AHRQ review, IQWiG researchers have published a third paper on their ‘objective approach’, comparing it with the ‘conceptual approach’ (21). The authors reported that the ‘objective approach’ yielded higher sensitivity than the ‘conceptual approach’, with similar precision, and stated that ‘objective approaches’ should be routinely used in the development of high-quality search strategies.

Stansfield et al (22) used a case study of searching to inform a guideline on the care and support of older people with learning disabilities, and other examples, to reflect on the utility of text-mining technologies in improving the precision and sensitivity of search strategies.   The technologies investigated include term frequency–inverse document frequency (TF-IDF) analysis and Lingo3G automated clustering tool within EPPI-Reviewer 4.0, Termine, BibExcel, and EndNote. The authors concluded that text mining could aid the discovery of search terms for search strategies for diversely-described topics to support an iterative search strategy development process, and that using multiple tools appeared to be particularly fruitful, though the overriding challenge of finding efficient ways to identify an unknown body of literature for incorporation in SRs still remained.

Combining search terms with Boolean operators and other search syntax

The Cochrane Handbook describes how a search strategy should be built up using controlled vocabulary terms, text words, synonyms and related terms for each concept at a time, joining together each of the terms within each concept with the Boolean ‘OR’ operator. The sets of terms may then be combined with AND which limits the results to those records that contain at least one search term from each of the sets. If an article does not contain at least one of the search terms from each of the sets then it will not be retrieved. Cochrane advise against the use of the NOT operator where possible to avoid inadvertently excluding relevant records (1).

The AHRQ manual refers searchers to the PRESS (Peer Review of Electronic Search Strategies) Checklist (23) and states that search strategies should make use of the advanced search techniques such as truncation, wildcards and proximity searching described in the PRESS document (3).  In 2015, the PRESS 2015 Guideline Statement was published, which updated and expanded on the previous PRESS publications (24).

Although search strategy development and construction for SRs conventionally aims for sensitivity, researchers have investigated the potential of 'focusing' search terms to reduce the number of search results and therefore screening burden. Focusing techniques which have been investigated include searching with subject headings limited to those with a major focus (major subject headings) and searching using terms in titles and abstracts alone (i.e. not including controlled vocabulary in the search strategy).

In a 2015 report produced by CADTH (25), researchers reran the search strategies reported in HTAs or SRs produced by a range of agencies, varying the use of major Emtree headings. The impact of the changes on the retrieval of the known relevant records (included studies) in the HTAs or SRs was assessed. The authors stated that overall their findings suggested that focusing Emtree headings was likely to reduce already suboptimal sensitivity for only small gains in precision. The report's recommendations for practice stated that searchers who were confident that their strategy was highly sensitive might wish to use focused Emtree terms for the intervention concept of their search. They suggested using caution when considering focusing the Emtree terms for the population concept, when considering focusing Emtree terms in more than two concepts, or when considering focusing terms in non-drug treatment reviews.

In a 2018 study, Bramer et al investigated whether researchers could use 'focused' searches to reduce the screening time burden (26). The original search strategies (designed by a single librarian) from a broad range of SRs were modified in four ways: by searching Embase thesaurus terms as major descriptors; by removing thesaurus terms from the Embase search so that terms were searched in the title and/or abstract fields only; by searching both MEDLINE and Embase thesaurus terms as major descriptors; by searching both MEDLINE and Embase for terms in the title and/or abstract fields only.  The authors concluded that if the number of search results retrieved was too high for the project resource context, search strategies in Embase alone or in both Embase and MEDLINE could be focused by searching for thesaurus terms as major descriptors. They stated that this approach 'may not ultimately have negative consequences in SRs', as long as a thorough searches in other databases (such as Web of Science) were performed in addition to the MEDLINE and Embase searches. They also stated however that the reduction in search result numbers was likely to be limited. The authors did not recommend searching Embase and MEDLINE using terms in titles and abstracts alone, as this resulted in too many relevant articles being missed.

Checking and testing search strategies

Search strategies should be checked to ensure they are fit for purpose: that they are likely to find relevant studies. This is difficult to ascertain but checking of search strategies can be carried out by expert / peer review (for example, using the PRESS Checklist (23, 24)), comparing against previously published strategies, or by testing that known relevant documents are retrieved by the strategy (3). 

Alternatively, more formal testing can be undertaken. Such methods are summarised by Booth, whose brief review identified eight methods for determining optimal retrieval of studies for inclusion in HTAs (27). The review concluded that although numerous methods were described in the literature, there was little formal evaluation of the strengths and weakness of each approach.

Sampson and McGowan developed and assessed a method (Inquisitio Validus Index Medicus) for validation of MEDLINE search strategies (28). The method used a version of the known relevant item approach, testing recall of relevant indexed studies identified through all search methods and indexed in the database being tested. The validation occurred once screening had been completed and the eligible studies were known. Poorly performing search strategies could be amended, re-tested and re-run. New studies identified by the amended search could be screened and any relevant studies could be included in the review.  The authors reported that the validation method was robust and was able to demonstrate that the retrieval of relevant studies from MEDLINE in a sample of six updated Cochrane reviews was sub-optimal. The authors concluded that the Inquisitio Validus test was a simple method of validating the search, and could determine whether the search of the main database performed adequately or needed to be revised to improve recall, allowing the searcher an opportunity to improve their search strategy.

One aspect of testing searches is to inform reviewers when searching has retrieved 'enough' studies. There is little research evidence on empirically based 'stopping rules' but methods such as capture-mark-recapture have been explored for developing such rules (29). Capture-mark-recapture has also been reported as being used to evaluate searches by estimating retrospectively their closeness to capturing the total body of literature (30, 31). The process involved hand-searching a sample journal and running a search strategy on information sources indexing the same journal. The number of relevant records identified by each process was then used to gain a statistical estimate of what had been missed by all searches conducted (30).

Despite these investigations the ARHQ guidelines state that no currently available method can be easily applied to searches for comparative effectiveness reviews. It is argued that the searcher’s judgement is required to decide whether searching additional sources is likely to result in the retrieval of unique items or whether the search has reached the point of saturation. The decision must balance the desire to identify all relevant studies with the resources available to carry out the search (3).

 

Reference list

  • (1)   Lefebvre C, Manheimer E, Glanville J.  Chapter 6: searching for studies. In: Higgins JPT, Green S (editors). Cochrane handbook for systematic reviews of interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration; 2011. [Further reference details] [Publication appraisal] [Free full text]
  • ​(2)   Systematic Reviews. CRD’s guidance for undertaking reviews in health care. Centre for Reviews and Dissemination, University of York; 2009. [Further reference details] [Publication appraisal] [Free full text]
  • (3)   Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Publication No. 10(14)-EHC063-EF. Rockville, MD: Agency for Healthcare Research and Quality. 2014 update. [Further reference details] [Publication appraisal] [Free full text]
  • (4) Sampson M, Tetzlaff J, Urquhart, C. Precision of healthcare systematic review searches in a cross-sectional sample. Res Synth Methods 2011 2(2): 119-125. [Further reference details] [Publication appraisal] [Free full text]
  • (5) Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev. 2016 May 4;5:74. doi: 10.1186/s13643-016-0249-x. [Further reference details] [Publication appraisal] [Free full text]
  • (6) Harris JL, Booth A, Cargo M, Hannes K, Harden A, Flemming K, Garside R, Pantoja T, Thomas J, Noyes J. Cochrane Qualitative and Implementation Methods Group guidance series - paper 2: methods for question formulation, searching, and protocol development for qualitative evidence synthesis. J Clin Epidemiol 2018 97: 39-48. [Further reference details] [Publication appraisal] [Free full text]
  • (7) Buchberger B, Krabbe L, Lux B, Mattivi JT. Evidence mapping for decision making: feasibility versus accuracy - when to abandon high sensitivity in electronic searches.Ger Med Sci. 2016 Jul 19;14:Doc09. doi: 10.3205/000236.[Further reference details] [Publication appraisal] [Free full text]
  • (8) Cooper C, Lovell R, Husk K, Booth A, Garside R. Supplementary search methods were more effective and offered better value than bibliographic database searching: A case study from public health and environmental enhancement. Res Syn Meth. 2017;1–29. https://doi.org/10.1002/jrsm.1286
  • (9)  Cooke A, Smith D, Booth A. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res 2012;22(10);1435-1443. [Further reference details] [Publication appraisal] [Free full text]
  • (10) Booth A, Carroll C. Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable? Health Info Libr J. 2015 Sep;32(3):220-35. Epub 2015 Jun 11. [Further reference details] [Publication appraisal] [Free full text]
  • (11) Eriksen, MB, Frandsen, TF. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review. J Med Libr Assoc 2018 106(4): 420-431. [Further reference details] [Publication appraisal] [Free full text]
  • (12) Bramer WM, Rethlefsen ML, Mast F, Kleijnen J. Evaluation of a new method for librarian-mediated literature searches for systematic reviews. Res Synth Methods. 2017 Oct 26. doi: 10.1002/jrsm.1279. [Further reference details] [Publication appraisal] [Free full text]
  • (13) Paynter RA, Bañez LL, Berliner E, Erinoff E, Lege-Matsuura J, Potter S, Uhl S. EPC Methods: An Exploration of the Use of Text-Mining Software in Systematic Reviews. Research White Paper. (Prepared by the Scientific Resource Center and the Vanderbilt and ECRI Evidence-based Practice Centers under Contract Nos. 290-2012-00004-C [SRC], 290-2012-00009-I [Vanderbilt], and 290-2012-00011-I [ECRI].) AHRQ Publication 16-EHC023-EF. Rockville, MD: Agency for Healthcare Research and Quality; April 2016.  [Further reference details] [Publication appraisal] [Free full text]
  • (14)  O'Mara-Eves A, Brunton G, McDaid D, Kavanagh J, Oliver S, Thomas J. Techniques for identifying cross-disciplinary and 'hard-to-detect' evidence for systematic review. Res Synth Methods 2014. 5(1): 50-59. [Further reference details] [Publication appraisal] [Free full text]
  • (15) Hausner E, Waffenschmidt S, Kaiser T, Simon M. Routine development of objectively derived search strategies. Syst Rev 2012;1(19):1-10. [Further reference details] [Publication appraisal] [Free full text]
  • (16) Hausner E, Guddat C, Hermanns T, Lampert U, Waffenschmidt S. Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach. J Clin Epidemiol. 2015;68(2):191-199. [Further reference details] [Publication appraisal] [Free full text]
  • (17) Boeker M, Motschall E, Vach W. Literature search methodology for systematic reviews: conventional and natural language processing enabled methods are complementary (Letter commenting on: J Clin Epidemiol. 2015;68:191-9). J Clin Epidemiol. 2016 Jan;69:255-7. Epub 2015 Jun 3. [Further reference details] [Publication appraisal] [Free full text]
  • (18) Dintsios CM, Niederstadt C. When suddenly the evaluation became a validation. (Letter commenting on: J Clin Epidemiol. 2015;68:191-9). J Clin Epidemiol. 2016 Jan;69:262-4. Epub 2015 Jun 18. [Further reference details] [Publication appraisal] [Free full text]
  • (19) Hausner E, Waffenschmidt S. Response to letter by Boeker et al. Development of search strategies for systematic reviews: further issues regarding the objective and conceptual approaches. J Clin Epidemiol. 2016 Jan;69:253-5. Epub 2015 Jun 3. [Further reference details] [Publication appraisal] [Free full text]
  • (20) Hausner E, Waffenschmidt S. Development of search strategies for systematic reviews: reply to commentary by Dintsios and Niederstadt. J Clin Epidemiol. 2016 Jan;69:265-6. Epub 2015 Jun 18. [Further reference details] [Publication appraisal] [Free full text]
  • (21) Hausner E, Guddat C, Hermanns T, Lampert U, Waffenschmidt S. Prospective comparison of search strategies for systematic reviews: an objective approach yielded higher sensitivity than a conceptual one. J Clin Epidemiol. 2016 May 30. pii: S0895-4356(16)30134-2. doi: 10.1016/j.jclinepi.2016.05.002. [Further reference details] [Publication appraisal] [Free full text]
  • (22) Stansfield C, O'Mara-Eves A, Thomas J. Text mining for search term development in systematic reviewing: A discussion of some methods and challenges. Res Synth Methods. 2017 Sep;8(3):355-365. [Further reference details] [Publication appraisal] [Free full text]
  • (23)  Sampson M, McGowan J, Lefebvre C, et al. PRESS: Peer Review of Electronic Search Strategies. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2008. [Further reference details] [Publication appraisal] [Free full text]
  • (24) McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 guideline statement. J Clin Epidemiol. 2016 Jul;75:40-6. [Further reference details] [Publication appraisal] [Free full text]
  • (25) Glanville J, Kaunelis D, Mensinkai S, Picheca L. Pruning Emtree: does focusing Embase Subject Headings impact search strategy precision and sensitivity? Ottawa: CADTH; 2015 Apr. Accessed  25 Mar 2019. [Further reference details] [Publication appraisal] [Free full text]
  • (26) Bramer, WM, Giustini D, Kleijnen J, Franco OH. Searching Embase and MEDLINE by using only major descriptors or title and abstract fields: a prospective exploratory study. Syst Rev 2018 7(1): 200. [Further reference details] [Publication appraisal] [Free full text]
  • (27)  Booth A. How much searching is enough? Comprehensive versus optimal retrieval for technology assessment.  Int J Technol Assess Health Care 2010;26(4):431-435. [Further reference details] [Publication appraisal] [Free full text]
  • (28) Sampson M, McGowan J. Inquisitio validus Index Medicus: A simple method of validating MEDLINE systematic review searches. Res Synth Methods. 2011 2(2): 103-109. [Further reference details] [Publication appraisal] [Free full text]
  • (29)  Kastner M, Straus SE, McKibbon KA. The capture-mark-recapture technique can be used as a stopping rule when searching in systematic reviews. J Clin Epidemiol 2009;62(2):149-157. [Further reference details] [Publication appraisal] [Free full text]
  • (30) Spoor P, Airey M, Bennett C, Greensill J, Williams R. Use of the capture-recapture technique to evaluate the completeness of systematic literature searches. BMJ 1996;313:342-343. [Further reference details] [Publication appraisal] [Free full text]
  • (31) Bennett, DA, Latham, NK, Stretton C, Anderson, CS. Capture-recapture is a potentially useful method of assessing publication bias. J Clin Epidemiol 2004;57(4):349-357. [Further reference details] [Publication appraisal] [Free full text]