Search strategy development
The Cochrane Information Retrieval Methods Group have published an evidence-based chapter on search methods for the Cochrane Handbook (1), which provides the basis for this summary alongside guidance produced by the Centre for Reviews and Dissemination (2) and the Agency for Healthcare Research and Quality (AHRQ) (3).
The revised and updated searching chapter of the Cochrane Handbook is in preparation. To avoid duplication of effort with the development of the Cochrane Handbook, appraisals have not been prepared for studies in this chapter. Once the revised Cochrane Handbook is available it will be used to update this chapter.
Sensitivity and precision
In order to retrieve as many studies relevant to the review as possible, and to compensate for the limitations of information source records and indexing, search strategy development and construction for systematic reviews should generally aim for sensitivity (1). Increasing the sensitivity of a search increases the possibility of identifying all relevant studies, but also tends to reduce precision because the number of irrelevant results is increased (1, 2). Sampson et al examined a cross section of 94 health related systematic reviews that reported the flow of bibliographic records through the review process and found that search precision of approximately 3% was typical (4). The number of results retrieved, which therefore must be screened against eligibility criteria, has implications for the resources required to conduct a systematic review. This trade-off between sensitivity and precision should be acknowledged and discussed with the wider review team, and an appropriate balance sought within the context of the resources available.
The emphasis on search sensitivity above precision typically reflects the context of systematic reviews of quantitative research. This emphasis may not be the same in searches developed for different purposes within the HTA context. In the context of qualitative systematic reviews or qualitative evidence syntheses for example, there is discussion as to whether these type of reviews share the same need as systematic reviews of quantitative research for ‘comprehensive’, ‘exhaustive’ bibliographic database searches (5). Similarly, in the context of conducting a search to inform an ‘evidence-map’ (where an overview of the extent, nature and characteristics of a research area is of interest) research has indicated that less sensitive searches may be appropriate. In a study which compared a ‘highly sensitive’ search strategy with a ‘highly specific’ search strategy for an evidence-mapping exercise on diabetes and driving to inform clinical guidance development, the authors report that the results of the ‘highly specific’ search would have been sufficient for answering the research question (6). The authors conclude that using highly specific instead of sensitive search strategies is “fully adequate for evidence maps with the aim of covering mainly the breadth rather than depth of a research spectrum”.
Structuring the search
The Cochrane Handbook suggests a search strategy should be structured around the main concepts being examined by the review. For reviews of interventions, this can be expressed using PICO (Patient (or Participant or Population), Intervention, Comparison and Outcome). It is usually seen as undesirable to include all elements of the PICO in the search strategy as some concepts are often poorly described or non-existent in the title and abstract of a database record or the assigned indexed terms. For reviews of many interventions a search may reasonably be comprised of the population, intervention, and a study design filter (1) if appropriate. A validated search filter is recommended where one exists for the concept of interest (3).
In some topic areas, for example complex interventions, where many of the concepts are particularly ill-defined, it may be preferable to use a broader search strategy (such as searching only for the population or intervention) and increase the resources allocated to sifting records (2). Alternatives to the PICO framework have also been evaluated for searches in some fields; examples include the SPIDER tool to structure searches for qualitative and mixed methods research (7) and the BeHEMoTh tool to structure searches for theory (8). In a structured methodological review on searching for qualitative research, Booth lists 11 different notations for use in this context (including PICO, SPIDER and BeHEMoTh), but states that, as with quantitative reviews, there is little empirical data to support the merits of question formulation (5). Methley et al tested the SPIDER search tool in a systematic narrative review of qualitative literature, comparing it with use of the PICO tool and a modified version of PICO with added qualitative search terms (PICOS) (9). The authors conclude that where comprehensiveness is a key factor the PICO tool should be used preferentially due to the risk of missing relevant studies using the SPIDER tool.
Selecting search terms
The Cochrane Handbook recommends that in order to identify as many relevant records as possible, search strategies should combine subject headings selected from the database’s controlled vocabulary or thesaurus (with appropriate “explosions”) and a wide range of free-text terms (1). The choice of free-text terms should include consideration of synonyms, related terms and variant spellings.
Methods for identifying search terms have traditionally included techniques such as checking the bibliographic records of known relevant studies, consulting topic experts and scanning database subject indexing guides (3). However, text mining is a rapidly developing tool with potential application in a range of tasks associated with the production of systematic reviews, including the identification of search terms (2). AHRQ have published a review on the use of text-mining tools as an emerging methodology within systematic review processes, including the literature search (10). The aim of the AHRQ project was to provide a ‘snapshot’ of the state of knowledge, rather than an in-depth assessment. The review refers to 12 studies where text-mining tools were used for development of ‘topic’ search strategies and identifies several general approaches to development. These include assessing word frequency in citations (using tools such as PubReminer or EndNote) and automated term extraction (using tools such as Termine). The review reports that all of the identified studies found benefit in automating term selection for systematic reviews, especially those comprising large unfocused topics. The AHRQ review made no conclusions which were specific to the use of text-mining tools for the literature search process. The general conclusions on the use of text-mining for systematic review processes were that text-mining tools appear promising, but further research is warranted.
Studies cited in the AHRQ review include a study by O'Mara-Eves et al which evaluated whether additional search terms for the topic of ‘community engagement’ were generated when using the text-mining data-extraction tool Termine (11), in addition to typical search development techniques. The study authors report that although in many cases the terms generated by text-mining had already been identified by the reviewers as relevant, text-mining did reveal some useful synonyms and terms associated with the topic that had not previously been considered. The study authors state that the text-mining approach studied should never be used on its own but alongside usual search development processes. The authors conclude that text mining helped to identify relevant search terms for a broad topic that was inconsistently referred to in the literature.
The AHRQ review also cited 2 studies published by researchers at the German HTA agency IQWiG. In the first study, the authors propose an 'objective approach' to strategy development using text analysis methods (12). The authors argue that this method ensures the process of selecting search terms is transparent and reproducible and allows a searcher with little specialist knowledge of the search topic to make decisions on the inclusion of terms that are informed by evidence. In the second study the authors aim to validate the ‘objective approach’, and conclude that it was noninferior to the standard 'conceptual approach' (13). Subsequent correspondence on this publication (14, 15) and the authors' responses to this correspondence (16, 17) has debated the study’s conclusions and the strengths and limitations of the methods used for this research. Since the publication of the AHRQ review, IQWiG researchers have published a third paper on their ‘objective approach’, comparing it with the ‘conceptual approach’ (18). The authors report that the ‘objective approach’ yielded higher sensitivity than the ‘conceptual approach’, with similar precision and state that ‘objective approaches’ should be routinely used in the development of high-quality search strategies.
Combining search terms with Boolean operators and other search syntax
The Cochrane Handbook describes how a search strategy should be built up using controlled vocabulary terms, text words, synonyms and related terms for each concept at a time, joining together each of the terms within each concept with the Boolean ‘OR’ operator. The sets of terms may then be combined with AND which limits the results to those records that contain at least one search term from each of the sets. If an article does not contain at least one of the search terms from each of the sets then it will not be retrieved. Cochrane advise against the use of the NOT operator where possible to avoid inadvertently excluding relevant records (1).
The AHRQ manual refers searchers to the PRESS (Peer Review of Electronic Search Strategies) Checklist (19) and states that search strategies should make use of the advanced search techniques such as truncation, wildcards and proximity searching described in the PRESS document (3). In 2015, the PRESS 2015 Guideline Statement was published, which updated and expanded on the previous PRESS publications (20).
Testing search strategies and deciding when to stop searching
Search strategies should be tested to ensure they are fit for purpose: that they find relevant studies. This is difficult to ascertain but testing of search strategies can be carried out informally by expert review, checking that known relevant documents are retrieved by the strategy, or by comparing against previously published strategies (3).
Alternatively, more formal testing can be undertaken. Such methods are summarised by Booth, whose brief review identified eight methods for determining optimal retrieval of studies for inclusion in HTAs (21). The review concludes that although numerous methods are described in the literature, there is little formal evaluation of the strengths and weakness of each approach. Sampson and McGowan developed and assessed a method (Inquisitio Validus Index Medicus) for validation of MEDLINE search strategies (22). The method uses a version of the known relevant item approach, testing recall of relevant indexed studies identified through all search methods and indexed in the database being tested. The validation occurs once screening has been completed and the eligible studies are known. Poorly performing search strategies can be amended, re-tested and re-run. New studies identified by the amended search can be screened and any relevant studies can be included in the review. The authors report that the validation method was robust and was able to demonstrate that the retrieval of relevant studies from MEDLINE in a sample of six updated Cochrane reviews was sub-optimal. The authors conclude that the Inquisitio Validus test is a simple method of validating the search, and can determine whether the search of the main database performs adequately or needs to be revised to improve recall, allowing the searcher an opportunity to improve their search strategy.
One aspect of testing searches is to inform reviewers when searching has retrieved 'enough' studies. There is little research evidence on empirically based 'stopping rules' but methods such as capture-mark-recapture have been explored for developing such rules (23). Capture-mark-recapture has also been reported as being used to evaluate searches by estimating retrospectively their closeness to capturing the total body of literature (24, 25). It involves hand-searching a sample journal and running a search strategy on information sources indexing the same journal. The number of relevant records identified by each process is then used to gain a statistical estimate of what has been missed by all searches conducted (24).
Despite these investigations the ARHQ guidelines state that no currently available method can be easily applied to searches for comparative effectiveness reviews. It is argued that the searcher’s judgement is required to decide whether searching additional sources is likely to result in the retrieval of unique items or whether the search has reached the point of saturation. The decision must balance the desire to identify all relevant studies with the resources available to carry out the search (3).
- (1) Lefebvre C, Manheimer E, Glanville J. Chapter 6: searching for studies. In: Higgins JPT, Green S (editors). Cochrane handbook for systematic reviews of interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration; 2011. [Further reference details] [Publication appraisal] [Free full text]
- (2) Systematic Reviews. CRD’s guidance for undertaking reviews in health care. Centre for Reviews and Dissemination, University of York; 2009. [Further reference details] [Publication appraisal] [Free full text]
- (3) Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Publication No. 10(14)-EHC063-EF. Rockville, MD: Agency for Healthcare Research and Quality. 2014 update. [Further reference details] [Publication appraisal] [Free full text]
- (4) Sampson M, Tetzlaff J, Urquhart, C. Precision of healthcare systematic review searches in a cross-sectional sample. Res Synth Methods 2011 2(2): 119-125. [Further reference details] [Publication appraisal] [Free full text]
- (5) Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev. 2016 May 4;5:74. doi: 10.1186/s13643-016-0249-x. [Further reference details] [Publicataion appraisal] [Free full text]
- (6) Buchberger B, Krabbe L, Lux B, Mattivi JT. Evidence mapping for decision making: feasibility versus accuracy - when to abandon high sensitivity in electronic searches.Ger Med Sci. 2016 Jul 19;14:Doc09. doi: 10.3205/000236.[Further reference details] [Publicataion appraisal] [Free full text]
- (7) Cooke A, Smith D, Booth A. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res 2012;22(10);1435-1443. [Further reference details] [Publication appraisal] [Free full text]
- (8) Booth A, Carroll C. Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable? Health Info Libr J. 2015 Sep;32(3):220-35. Epub 2015 Jun 11. [Further reference details] [Publication appraisal] [Free full text]
- (9) Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BCM Health Serv Res. 2014 Nov 21;14(1):579. [Further reference details] [Publication appraisal] [Free full text]
- (10) Paynter RA, Bañez LL, Berliner E, Erinoff E, Lege-Matsuura J, Potter S, Uhl S. EPC Methods: An Exploration of the Use of Text-Mining Software in Systematic Reviews. Research White Paper. (Prepared by the Scientific Resource Center and the Vanderbilt and ECRI Evidence-based Practice Centers under Contract Nos. 290-2012-00004-C [SRC], 290-2012-00009-I [Vanderbilt], and 290-2012-00011-I [ECRI].) AHRQ Publication 16-EHC023-EF. Rockville, MD: Agency for Healthcare Research and Quality; April 2016. [Further reference details] [Publication appraisal] [Free full text]
- (11) O'Mara-Eves A, Brunton G, McDaid D, Kavanagh J, Oliver S, Thomas J. Techniques for identifying cross-disciplinary and 'hard-to-detect' evidence for systematic review. Res Synth Methods 2014. 5(1): 50-59. [Further reference details] [Publication appraisal] [Free full text]
- (12) Hausner E, Waffenschmidt S, Kaiser T, Simon M. Routine development of objectively derived search strategies. Syst Rev 2012;1(19):1-10. [Further reference details] [Publication appraisal] [Free full text]
- (13) Hausner E, Guddat C, Hermanns T, Lampert U, Waffenschmidt S. Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach. J Clin Epidemiol. 2015;68(2):191-199. [Further reference details] [Publication appraisal] [Free full text]
- (14) Boeker M, Motschall E, Vach W. Literature search methodology for systematic reviews: conventional and natural language processing enabled methods are complementary (Letter commenting on: J Clin Epidemiol. 2015;68:191-9). J Clin Epidemiol. 2016 Jan;69:255-7. Epub 2015 Jun 3. [Further reference details] [Publication appraisal] [Free full text]
- (15) Dintsios CM, Niederstadt C. When suddenly the evaluation became a validation. (Letter commenting on: J Clin Epidemiol. 2015;68:191-9). J Clin Epidemiol. 2016 Jan;69:262-4. Epub 2015 Jun 18. [Further reference details] [Publication appraisal] [Free full text]
- (16) Hausner E, Waffenschmidt S. Response to letter by Boeker et al. Development of search strategies for systematic reviews: further issues regarding the objective and conceptual approaches. J Clin Epidemiol. 2016 Jan;69:253-5. Epub 2015 Jun 3. [Further reference details] [Publication appraisal] [Free full text]
- (17) Hausner E, Waffenschmidt S. Development of search strategies for systematic reviews: reply to commentary by Dintsios and Niederstadt. J Clin Epidemiol. 2016 Jan;69:265-6. Epub 2015 Jun 18. [Further reference details] [Publication appraisal] [Free full text]
- (18) Hausner E, Guddat C, Hermanns T, Lampert U, Waffenschmidt S. Prospective comparison of search strategies for systematic reviews: an objective approach yielded higher sensitivity than a conceptual one. J Clin Epidemiol. 2016 May 30. pii: S0895-4356(16)30134-2. doi: 10.1016/j.jclinepi.2016.05.002. [Further reference details] [Publicataion appraisal] [Free full text]
- (19) Sampson M, McGowan J, Lefebvre C, et al. PRESS: Peer Review of Electronic Search Strategies. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2008. [Further reference details] [Publication appraisal] [Free full text]
- (20) McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 guideline statement. J Clin Epidemiol. 2016 Jul;75:40-6. [Further reference details] [Publicataion appraisal] [Free full text]
- (21) Booth A. How much searching is enough? Comprehensive versus optimal retrieval for technology assessment. Int J Technol Assess Health Care 2010;26(4):431-435. [Further reference details] [Publication appraisal] [Free full text]
- (22) Sampson M, McGowan J. Inquisitio validus Index Medicus: A simple method of validating MEDLINE systematic review searches. Res Synth Methods. 2011 2(2): 103-109. [Further reference details] [Publication appraisal] [Free full text]
- (23) Kastner M, Straus SE, McKibbon KA. The capture-mark-recapture technique can be used as a stopping rule when searching in systematic reviews. J Clin Epidemiol 2009;62(2):149-157. [Further reference details] [Publication appraisal] [Free full text]
- (24) Spoor P, Airey M, Bennett C, Greensill J, Williams R. Use of the capture-recapture technique to evaluate the completeness of systematic literature searches. BMJ 1996;313:342-343. [Further reference details] [Publication appraisal] [Free full text]
- (25) Bennett, DA, Latham, NK, Stretton C, Anderson, CS. Capture-recapture is a potentially useful method of assessing publication bias. J Clin Epidemiol 2004;57(4):349-357. [Further reference details] [Publication appraisal] [Free full text]