Value of using different search approaches

The conventional search approach of applying Boolean logic to subject heading and free-text queries continues to dominate literature reviews as it remains an effective method for searching bibliographic databases. However, sensitivity and specificity issues relating to Boolean searching have led researchers to investigate a variety of alternative search approaches: citation searching (direct or indirect citation relationships), the use of the “similar articles” function, hand searching, and methods of automated retrieval are some examples.

Similar articles

There is evidence that the “similar articles” link in PubMed (formerly known as “related citations”) is an efficient search approach (1-3) that can be used for scoping searches or for identifying the need to update a systematic review. The results of three analyses suggest that there is real benefit in using the 'similar articles' link and a simple Boolean search in PubMed. In the study by Waffenschmidt et al. (3) the combination of these two search methods resulted in 98% sensitivity when searching for randomized controlled trials (RCTs) of drugs. Sampson et al. searched for various topics and the combination of the two search methods reached 90% sensitivity in the original testing (1) and 91% (2) when retested.

Citation searching

The evidence available indicates that especially the use of so-called indirect citation relationships, such as checking co-citations, seems to be superior to the commonly used direct citation relationships (5).

Direct citation relationships: Checking reference lists and citation tracking

A review article by Cooper et al. (4) discusses the current literature on supplementary search techniques, such as citation searching, including the advantages and disadvantages of these techniques. A Cochrane review (6) included 12 studies examining manually checking reference lists and concluded that there is some evidence to support this method when traditional searching is difficult, but that the studies were heterogeneous and at a high risk of bias. A recent study by Preston et al. (7) showed that search strategies in MEDLINE and Embase identified 85% of studies on diagnostic test accuracy included in 9 systematic reviews; 24 further studies (8%) were identified by checking reference lists. A single study (8)comparing an automated system using the Scopus database with manual reference checking found the automated system to be equally sensitive but considerably more (62.5%) time efficient than the manual method.

Citation tracking (“cited by” function) does not seem to be a useful approach to supplement traditional searching. Three studies (9-11) showed that citation tracking could only identify between 0 and 12% unique references on a certain subject compared with other search approaches. However, citation tracking appears to offer some added value compared with Boolean searching when performing more challenging searches (e.g. for outcome measures) (12).

Indirect citation relationships: Co-citations

In two independent studies, Janssens and Gwinn (13) analysed co-citations (i.e. the reference lists of articles citing key articles). Sensitivities between 79 and 82% were achieved when combining co-citation searching and direct citation searching. The number needed to screen was reduced by 50 to 89% (median). In a further study (14), the approach was validated and achieved a median sensitivity of 75%. Belter (15, 16) used a combination of approaches (similar articles, direct and indirect citation relationships) in the Web of Science database. He tested and modified the method in two studies and retrieved a median of 74 to 87% of the references included in Cochrane reviews.

Full-text search

Linder (12) evaluated Google Scholar's full-text search feature in order to find studies on 'outcome measurement instruments' - terms that are often omitted from the title, abstract and subject headings of articles. The keyword (full text) search using Google Scholar yielded the highest sensitivity (70%). However, one must bear in mind that searching Google Scholar is time-consuming and difficult, as its functionality (e.g. incomplete bibliographic information, no reference export) remains limited (17).

Automated retrieval methods

At present automated retrieval methods implemented e.g. in public available interfaces have not been sufficiently evaluated to decide whether they are a useful approach for performing sensitive searches. One study comparing Boolean searching with ranked querying in MEDLINE (Ovid) reported that ranked retrieval alone was not reliable for a search task requiring high recall (18, 19).

Hand searching

Several current studies comparing hand searching with electronic database searching concluded that there was little or no benefit offered by hand searching. This was found to be the case when searching for additional RCTs (20), diagnostic test accuracy (DTA) studies (21), and occupational health intervention studies published in a language other than English (22). This contradicts the conclusions of a 2007 Cochrane review which found hand searching to be an effective approach when searching for systematic reviews; however, this review is now outdated as its most recent search was performed in 2002 and its main comparison was between hand searching and the "old" (1994), three-section Cochrane Highly Sensitive Search Strategy for RCTs (23).


Reference list

  • (1) Sampson M, Shojania KG, McGowan J, Daniel R, Rader T, Iansavichene AE, et al. Surveillance search techniques identified the need to update systematic reviews. J Clin Epidemiol. 2008;61(8):755-62. [Further reference details] [Publication appraisal] [Free Full text]
  • (2) Sampson M. Complementary approaches to searching MEDLINE may be sufficient for updating existing systematic reviews. J Clin Epidemiol. 2016;78:108-15. [Further reference details[Publication appraisal] [Free Full text]
  • (3) Waffenschmidt S, Janzen T, Hausner E, Kaiser T. Simple search techniques in PubMed are potentially suitable for evaluating the completeness of systematic reviews. J Clin Epidemiol. 2013;66(6):660-5. [Further reference details[Publication appraisal] [Free Full text]
  • (4) Cooper C, Booth A, Britten N, Garside R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev. 2017;6(1):234. [Further reference details] [Publication appraisal] Free full text]
  • (5) Robinson KA, Dunn AG, Tsafnat G, Glasziou P. Citation networks of related trials are often disconnected: implications for bidirectional citation searches. J Clin Epidemiol 2014. p. 793-9. [Further reference details] [Publication appraisal] [Free full text]
  • (6) Horsley T, Dingwall O, Tetzlaff JM, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev. 2009(1):MR000026. [Further reference details] [Publication appraisal] [Free Full text]
  • (7)  Preston L, Carroll C, Gardois P, Paisley S, Kaltenthaler E. Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking. Syst Rev. 2015;4:82. [Further reference details] [Publication appraisal] [Free full text]
  • (8) Chapman AL, Morgan LC, Gartlehner G. Semi-automating the manual literature search for systematic reviews increases efficiency. Health Info Libr J. 2010;27(1):22-7.  [Further reference details] [Publication appraisal] [Free Full text]
  • (9) Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Info Libr J. 2010;27(2):114-22. [Further reference details] [Publication appraisal] [Free Full text]
  • (10) Westphal A, Kriston L, Holzel LP, Harter M, von Wolff A. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. J Public Health Res. 2014;3(2):177. [Further reference details] [Publication appraisal] [Free Full text]
  • (11) Wright K, Golder S, Rodriguez-Lopez R. Citation searching: a systematic review case study of multiple risk behaviour interventions. BMC Med Res Methodol. 2014;14:73. [Further reference details] [Publication appraisal] [Free Full text]
  • (12) Linder SK, Kamath GR, Pratt GF, Saraykar SS, Volk RJ. Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments. J Clin Epidemiol. 2015;68(4):412-7. [Further reference details] [Publication appraisal] [Free Full text]
  • (13) Janssens AC, Gwinn M. Novel citation-based search method for scientific literature: application to meta-analyses. BMC Med Res Methodol. 2015;15:84. [Further reference details[Publication appraisal] [Free full text]
  • (14) Janssens ACJW, Gwinn M, Brockman E, Powell K, Goodman M. Novel Citation-Based Search Method for Scientific Literature: A Validation Study. PeerJ Prepints. 2019;7:e27646v1. [Further reference details] [Publication appraisal] [Free full text]
  • (15) Belter CW. Citation analysis as a literature search method for systematic reviews. J Assn Inf Sci Tec. 2016;67(11):2766-77. [Further reference details] [Publication appraisal] [Free Full text]
  • (16) Belter CW. A relevance ranking method for citation-based search results. Scientometrics. 2017;112(2):731-46. [Further reference details] [Publication appraisal] [Free full text]
  • (17) Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with Web of Science and Google Scholar. Res Synth Methods. 2015;7(1):34-45. [Further reference details] [Publication appraisal] [Free Full text]
  • (18) Karimi S, Pohl S, Scholer F, Cavedon L, Zobel J. Boolean versus ranked querying for biomedical systematic reviews. BMC Med Inform Decis Mak. 2010;10:58. [Further reference details] [Publication appraisal] [Free Full text]
  • (19) Pohl S. Boolean and ranked information retrieval for biomedical systematic reviewing [Dissertation]. Melbourne: University of Melbourne; 2012. [Further reference details] [Publication appraisal] [Free Full text]
  • (20) Armstrong R, Jackson N, Doyle J, Waters E, Howes F. It's in your hands: the value of handsearching in conducting systematic reviews of public health interventions. J Public Health. 2005;27(4):388-91. [Further reference details] [Publication appraisal] [Free Full text]
  • (21) Glanville J, Cikalo M, Crawford F, Dozier M, McIntosh H. Handsearching did not yield additional unique FDG-PET diagnostic test accuracy studies compared with electronic searches: a preliminary investigation. Res Syn Meth. 2012;3(3):202-13. [Further reference details] [Publication appraisal] [Free Full text]
  • (22) Mattioli S, Farioli A, Cooke RM, Baldasseroni A, Ruotsalainen J, Placidi D, et al. Hidden effectiveness? Results of hand-searching Italian language journals for occupational health interventions. Occup Environ Med. 2012;69(7):522-4. [Further reference details] [Publication appraisal] [Free Full text]
  • (23) Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 2007(2):MR000001. [Further reference details] [Publication appraisal] [Free Full text]