Value of using different search approaches

Last revised: 

The conventional search approach of applying Boolean logic to subject heading and free-text queries continues to dominate literature reviews as it remains an effective method for searching the major online bibliographic databases. However, sensitivity and specificity issues relating to Boolean searching have led researchers to investigate a variety of alternative search approaches: checking reference lists (backward citation), citation tracking (forward citation), using the “similar articles” function in the database, hand searching, and methods of automated retrieval are some examples.

Although there is only limited evidence of the effectiveness of these approaches, the available evidence indicates that checking references lists, using the “similar articles” function, and (with limitations) citation tracking, might be useful approaches in a comprehensive literature search. For instance, this applies to searches for complex evidence (1), to cases where the research question cannot be clearly defined (2), or the topic of interest is commonly missing from the title, abstract or subject headings of a reference (e.g. outcome measures (3,4)).

Related citations

There is evidence that using the “related citations” link in PubMed (now called “similar articles”) is an efficient search approach (5,6) and can usefully be used for scoping searches or to identify the need to update a systematic review. Two studies reported sensitivity above 80% for this approach (with one study using only the first 20 related citations) compared with other search techniques. In the study by Waffenschmidt et al (6)the combination of using the 'related citations' link and a simple Boolean search in PubMed resulted in 98% sensitivity when searching for RCTs of drugs.

Reference checking (backward citation) and citation tracking (forward citation)

There is evidence on checking reference lists to supplement traditional searching (i.e. database and hand searching) for systematic reviews. A Cochrane review (7) included 12 studies examining manually checking reference lists and concluded that there is some evidence to support this method when traditional searching is difficult, but that the studies were heterogeneous and at a high risk of bias. A recent study by Preston et al. (8) showed that search strategies in MEDLINE and Embase identified 85% of studies on diagnostic test accuracy included in 9 systematic reviews; 24 further studies (8%) were identified by checking reference lists. A single study (9) comparing an automated system using the Scopus database with manual reference checking found the automated system to be equally sensitive but considerably more (62.5%) time efficient than the manual method. However, this automated method is limited by the fact that Scopus requires a paid subscription and does not include the reference lists of Cochrane reviews.

Citation tracking seems not to be a useful approach to supplement traditional searching. Three studies (10-12) showed that citation tracking could only identify between 0-12% unique references on a certain subject compared with other search approaches. However, citation tracking appears to offer some added value compared with Boolean searching when performing more challenging searches (e.g. for outcome measures) (4).

A combined approach of both techniques was presented by Janssens and Gwinn (13).  In two independent studies, they performed a forward search for relevant publications in the Web of Science database. The complete reference lists were extracted from the publications identified. Sensitivities between 79 and 82 % were achieved. The number needed to screen was reduced by 50 to 89% (median).

Full text search

A paper by Linder (4) evaluated Google Scholar's full text search feature in order to find studies on 'outcome measurement instruments' - terms for which are often omitted from the title, abstract and subject headings of articles. The keyword (full text) search using Google Scholar yielded the highest sensitivity (70%). However, one must bear in mind that searching Google Scholar is time-consuming and difficult, as its functionality (e.g. incomplete bibliographic information, no reference export) remains limited (14).

Automated retrieval methods

At the moment automated retrieval methods have not been sufficiently evaluated to decide whether they are a useful approach for performing sensitive searches. One study comparing Boolean searching with ranked querying in MEDLINE (Ovid) reported that ranked retrieval alone was not reliable for a search task requiring high recall (15,16).

Hand searching

Several current studies comparing hand searching with electronic database searching concluded that there was little or no benefit offered by hand searching. This was found to be the case when searching for additional RCTs (17), diagnostic test accuracy (DTA) studies (18), and occupational health intervention studies published in a language other than English (19). This contradicts the conclusions of a 2007 Cochrane review which found hand searching to be an effective approach when searching for systematic reviews; however, this review is now outdated as its most recent search was performed in 2002 and its main comparison was between hand searching and the "old" (1994), three-section Cochrane Highly Sensitive Search Strategy for RCTs (20).


Reference list

  • 1. Greenhalgh T, Peacock RC. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005;331(7524):1064-5.  [Further reference details] [Publication appraisal] [Free Full text]
  • 2. Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. Pharmacoeconomics. 2015;33(1):5-11.  [Further reference details] [Publication appraisal] [Free Full text]
  • 3. O'Leary N, Tiernan E, Walsh D, Lucey N, Kirkova J, Davis MP. The pitfalls of a systematic MEDLINE review in palliative medicine: symptom assessment instruments. Am J Hosp Palliat Care. 2007;24(3):181-4.  [Further reference details] [Publication appraisal] [Free Full text]
  • 4. Linder SK, Kamath GR, Pratt GF, Saraykar SS, Volk RJ. Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments. J Clin Epidemiol. 2015;68(4):412-7.  [Further reference details] [Publication appraisal] [Free Full text]
  • 5. Sampson MJ. Updating searches for systematic reviews [dissertation]. Department of Information Studies: University of Aberystwyth; 2009.  [Further reference details] [Publication appraisal] [Free Full text]
  • 6. Waffenschmidt S, Janzen T, Hausner E, Kaiser T. Simple search techniques in PubMed are potentially suitable for evaluating the completeness of systematic reviews. J Clin Epidemiol. 2013;66(6):660-5.  [Further reference details[Publication appraisal] [Free Full text]
  • 7. Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev. 2011 Aug 10;(8):MR000026.  [Further reference details] [Publication appraisal] [Free Full text]
  • 8. Preston L, Carroll C, Gardois P, Paisley S, Kaltenthaler E. Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking. Syst Rev 2015; 4: 82. [Further reference details] [Publication appraisal] [Free full text]
  • 9. Chapman AL, Morgan LC, Gartlehner G. Semi-automating the manual literature search for systematic reviews increases efficiency. Health Info Libr J. 2010;27(1):22-7.  [Further reference details] [Publication appraisal] [Free Full text]
  • 10. Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Info Libr J. 2010;27(2):114-22.  [Further reference details] [Publication appraisal] [Free Full text]
  • 11. Westphal A, Kriston L, Holzel LP, Harter M, von Wolff A. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. J Public Health Res. 2014;3(2):177.  [Further reference details] [Publication appraisal] [Free Full text]
  • 12. Wright K, Golder S, Rodriguez-Lopez R. Citation searching: a systematic review case study of multiple risk behaviour interventions. BMC Med Res Methodol. 2014; 14:73.  [Further reference details] [Publication appraisal] [Free Full text]
  • 13. Janssens AC, Gwinn M. Novel citation-based search method for scientific literature: application to meta-analyses. BMC Med Res Methodol 2015; 15: 84. [Further reference details] [Publication appraisal] [Free full text]
  • 14. Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with Web of Science and Google Scholar. Res Synth Methods. 2016;7:34-45.  [Further reference details] [Publication appraisal] [Free Full text]
  • 15. Karimi S, Pohl S, Scholer F, Cavedon L, Zobel J. Boolean versus ranked querying for biomedical systematic reviews. BMC Med Inform Decis Mak. 2010;10(1):58.  [Further reference details] [Publication appraisal] [Free Full text]
  • 16. Pohl S. Boolean and Ranked Information Retrieval for Biomedical Systematic Reviewing. University of Melbourne, Department of Computer Science and Software Engineering [thesis]. Department of Computer Science and Software Engineering: University of Melbourne; 2012.  [Further reference details] [Publication appraisal] [Free Full text]
  • 17. Armstrong R, Jackson N, Doyle J, Waters E, Howes F. It's in your hands: the value of handsearching in conducting systematic reviews of public health interventions. J Public Health (Oxf). 2005;27(4):388-91.  [Further reference details] [Publication appraisal] [Free Full text]
  • 18. Glanville J, Cikalo M, Crawford F, Dozier M, McIntosh H. Handsearching did not yield additional unique FDG-PET diagnostic test accuracy studies compared with electronic searches: a preliminary investigation. Res Synth Methods. 2012;3(3):202-13.  [Further reference details] [Publication appraisal] [Free Full text]
  • 19. Mattioli S, Farioli A, Cooke RMT, Baldasseroni A, Ruotsalainen J, Placidi D, et al. Hidden effectiveness? Results of hand-searching Italian language journals for occupational health interventions. Occup Environ Med. 2012;69(7):522-4.  [Further reference details] [Publication appraisal] [Free Full text]
  • 20. Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 2007(2):MR000001.  [Further reference details] [Publication appraisal] [Free Full text]