Documenting and reporting the search process

Last revised: 
2016-12-20

Introduction

Thorough documentation and clear reporting are essential when conducting a comprehensive literature search for a health technology assessment or systematic review. The search encompasses one of the most important aspects of the review as the evidence base it identifies forms the foundation for the project. A faulty search can result in a flawed review, therefore, documenting all aspects of the search to showcase that it is methodologically sound aids in establishing a review’s credibility (1, 2). It can then be said that the ultimate goal of the reporting/documenting process is transparency and reproducibility with the added benefit of increasing the reader’s confidence in the research. Thorough documentation of the search also facilitates future updating of a review and may also serve as a resource for research team members should they need to refer to this information during the production of other sections of the report (1, 3).

Unfortunately, it has been found that within the health sciences literature large numbers of systematic searches (including those used in Cochrane Reviews) (4) are inadequately documented and described in the final reports of the project, leading to questions surrounding the validity of the search methodology used and rendering them irreproducible for readers (4-7).

Publication appraisals were not prepared for handbooks and guidance documents.

Background

To date, there exist many reporting and searching standards/published recommendations for systematic reviews and HTAs that touch upon what components of the search strategy to describe to achieve transparency and replicability for readers. In terms of standards these include: The Cochrane Handbook (8), MECIR (9), The PRISMA Checklist (10), The Institute of Medicine’s Standards for Systematic Reviews (11), HugeNet (12), RAMSES (13), CRD Guidance (14), ENTREQ (15), The DACEHTA Handbook (16), EUnetHTA guideline for information retrieval (17), The INAHTA Checklist (18) and the Campbell Collaboration guide (19).

In addition, a number of published studies aim to set forth guidelines for search strategy reporting, notably Booth’s article on the mnemonic STARLITE (20). The work of Rader et. al. (3), Sampson et al. (1), and Atkinson (21) involve reviewing current guidance and/or surveying search practitioners on search strategy documentation. They also provide additional suggestions in areas where information is lacking (1, 3, 21-25).  

There is, however, high variability among all these recommendations and sometimes little consensus as to what elements of the search are essential to describe and also what level of detail should be provided for each element (1, 22).

An Inventory of Search Documentation/Reporting Elements and Methods

The attached table (please see below) represents a synthesis of the literature (reporting standards and research articles) surrounding the documentation and reporting of search strategies. It only presents recommendations as found in the literature and therefore it may seem lacking in some areas due to a paucity of published information on those areas.  The literature describes search strategy documentation and reporting for various types of products that relate to a systematic review or HTA, mainly the full report, the abstract of the report, the project protocol and the report appendices. Information for each of these types of documents is represented in the table. In addition, while sometimes used interchangeably, the activity of recording the search strategy is often divided up by the literature into two separate processes, that of documenting the search strategy and that of reporting the search strategy. Documenting the search can be seen as recording information about the search simultaneously as it is run. This information is often used for internal purposes and records. Reporting the search involves the final formal write up of the search methodology that will be included in the published version of the review/HTA. Information from the documenting phase feeds into the reporting phase (8, 9, 14). Once again information for each of these processes is described in the table below. Finally, a portion of the literature covering this area looks specifically at documenting and reporting the search process for reviews of the qualitative literature. Search strategies for qualitative reviews may encompass a different process than traditional systematic reviews of clinical interventions. The elements and methodology for reporting/documenting thus vary in some aspects and can be unique to these types of reviews. These items are also flagged in the table as being relevant for the description of search strategies with a qualitative emphasis.

Reference list

  • (1) Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. J Clin Epidemiol. 2008;61(8):748-54. [Further reference details] [Publication appraisal]  [Free full text]
  • (2) Booth A, Papaioannou D, Sutton A. Systematic Approaches to a Successful Literature Review. Thousand Oaks, CA: Sage; 2012. [Further reference details] [Publication appraisal]  [Free full text]
  • (3) Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Res Synth Method. 2014;5(2):98-115 [Further reference details] [Publication appraisal]  [Free full text]
  • (4) Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. J Med Libr Assoc. 2009;97(1):21-9. [Further reference details] [Publication appraisal]  [Free full text]
  • (5) Faggion CM, Atieh MA, Park S. Search strategies in systematic reviews in periodontology and implant dentistry. J Clin Periodontol. 2013;40(9):883-8. [Further reference details] [Publication appraisal]  [Free full text]
  • (6) Golder S, Loke Y, McIntosh HM. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. J Clin Epidemiol. 2008;61(5):440-8. [Further reference details] [Publication appraisal]  [Free full text]
  • (7) Roundtree AK, Kallen MA, Lopez-Olivo MA, Kimmel B, Skidmore B, Ortiz Z, et al. Poor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: a systematic review. J Clin Epidemiol. 2009;62(2):128-37. [Further reference details] [Publication appraisal]  [Free full text]
  • (8) Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. [Further reference details] [Publication appraisal] [Free full text]
  • (9) Higgins JPT, Lasserson T, Chandler J, Tovey D, Churchill R. Methodological Expectations of Cochrane Intervention Reviews. Cochrane: London, 2016. [Further reference details] [Publication appraisal]  [Free full text]
  • (10) Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264-9. [Further reference details] [Publication appraisal]  [Free full text]
  • (11) Eden J, Levit L, Berg A, Morton S, editors. Finding What Works in Health Care: Standards for Systematic Reviews. Washington (DC): National Academies Press; 2011. [Further reference details] [Publication appraisal]  [Free full text]
  • (12) Little J, Higgins J, Bray M, Ioannidis J, Khoury M, Manolio T, et al. The HuGENet™ HuGE review handbook, version 1.0. Ottawa, Ontario, Canada: HuGENet Canada Coordinating Centre. 2006. [Further reference details] [Publication appraisal]  [Free full text]
  • (13) Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: realist syntheses. BMC Med. 2013;11(1):1. [Further reference details] [Publication appraisal]  [Free full text]
  • (14) Akers J, Aguiar-Ibáñez R, Baba-Akbari Sari A. Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care. York, UK: Centre for Reviews and Dissemination (CRD); 2009. [Further reference details] [Publication appraisal]  [Free full text]
  • (15) Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol. 2012;12(1):1. [Further reference details] [Publication appraisal]  [Free full text]
  • (16) Kristensen FB, Sigmund H, editors. Health Technology Assessment Handbook. Copenhagen: Danish Centre for Health Technology Assessment, National Board of Health 2008. [Further reference details] [Publication appraisal]  [Free full text]
  • (17) European Network for Health Technology Assessment (EUnetHTA). Guideline: Process of Information Retrieval for Systematic Reviews and Health Technology Assessments on Clinical Effectiveness. EUnetHTA; 2015. [Further reference details] [Publication appraisal]  [Free full text]
  • (18) INAHTA Secretariat. INAHTA: A Checklist for Health Technology Assessment Reports. INAHTA; 2007. [Further reference details] [Publication appraisal]  [Free full text]
  • (19) Hammerstrøm K, Wade A, Jørgensen A-MK. Searching for studies: A guide to information retrieval for Campbell Systematic Reviews. Campbell Systematic Reviews. 2010;Supplement 1. [Further reference details] [Publication appraisal]  [Free full text]
  • (20) Booth AC. "Brimful of STARLITE": toward standards for reporting literature searches. J Med Libr Assoc. 2006;94:421-9, e205. [Further reference details] [Publication appraisal]  [Free full text]
  • (21) Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Res Synth Method. 2015(1):87. [Further reference details] [Publication appraisal]  [Free full text]
  • (22) Niederstadt C, Droste S. Reporting and presenting information retrieval processes: the need for optimizing common practice in health technology assessment. Int J Technol Assess Health Care. 2010;26(4):450-7. [Further reference details] [Publication appraisal]  [Free full text]
  • (23) Briscoe S. Web searching for systematic reviews: a case study of reporting standards in the UK Health Technology Assessment programme. BMC Res Notes. 2015;8:153-. [Further reference details] [Publication appraisal]  [Free full text]
  • (24) Maggio LA, Tannery NH, Kanter SL. Reproducibility of literature search reporting in medical education reviews. Acad Med. 2011;86(8):1049-54. [Further reference details] [Publication appraisal]  [Free full text]
  • (25) Finfgeld-Connett D, Johnson ED. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. J Adv Nurs. 2013;69(1):194-204. [Further reference details] [Publication appraisal]  [Free full text]
AttachmentSize
Elements of the Search to be Reported Table 1.pdf212.79 KB