Documenting

Scope: guidance on documenting and reporting literature searches, what to report, evaluations of documentation quality

Appraisal of: "Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev, 5(1). doi:10.1186/s13643-016-0249-x"

Reviewer(s): 
Short description: 

This review provides a systematic overview of the available published evidence of searching methods to inform qualitative evidence synthesis (QES). The author sought to assess and identify:

1) the current state of knowledge in relation to searching for qualitative evidence

2) the robustness of the evidence base

3) research gaps and future priorities.

The studies were obtained from the Reference Manager database of the Cochrane Qualitative Methods Group’s study register, of which the author is responsible for updating and maintaining. Supplementary citation searches via Google Scholar was also carried out for 15 key papers. 113 studies were assessed for inclusion. Quality assessment of the included studies was not deemed feasible due to a large proportion of the included studies providing only narrative findings, the lack of a common appraisal instrument and the high levels of heterogeneity across the remaining studies.

The evidence underpinning systematic approaches to searching for qualitative evidence is classified and summarized within one or more of eight headings/ “7 S structure/ framework”: overviews and methodological guidelines, sampling, sources, structured questions, search procedures, search strategies and filters, supplementary strategies and standards. The author summarizes the available evidence and key issues within each section and makes recommendations for further empirical research. Table 7 breaks down the key starting principles in reference to the “7S structure” of searching to inform qualitative evidence synthesis to inform future guidance and Table 8 provides an overview of research priorities.

The review concludes that there is a lack of empirical data to inform information retrieval for QES and that the strength of the evidence is weak and largely based on personal/ professional experience and case studies. Advances have been made in reporting QES, however, validated standards are lacking. 

Limitations stated by the author(s): 

The author states that in order for studies to be included, the references needed to include terms specifically related to searching or retrieval in their titles or abstracts, cite a number of key texts, or be referred to from previously identified items. The full-text of all papers reporting QES were not examined. There is a possibility that potentially relevant reviews reporting emerging information retrieval methods that were not reported in the title or abstract were missed. However, these risks are offset by the sensitive search approach and the currency and comprehensiveness of the Cochrane Qualitative Methods Group study register. The author also notes that some papers were excluded as they did not distinguish between qualitative and quantitative approaches, which could potentially be useful for mixed methods reviews.

Limitations stated by the reviewer(s): 
1. This was a single reviewer methodology review so judgements on eligibility and interpretations of potential significance and contribution of individual studies may not be consistent and/or reproducible. [Study Selection Bias] 2. Although as a methodology review there is no formal requirement to follow PRISMA reporting standards this review may have benefited from more complete and transparent reporting. [Incomplete Reporting Bias] 3. As the reviewer was author on a high proportion of included studies this may have consciously impacted on study identification and subconsciously on study selection and interpretation. [Citation Bias; Observer Bias]
Study Type: 
Single study

Appraisal of: “Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. J Clin Epidemiol. 2008;61(8):748-54.”

Short description: 

The aim of this paper was to identify validated search reporting instruments and to compare reported and recommended reporting practices.  11 instruments and 18 different reporting items were identified.  The highest number of reporting items used by an instrument was 11. The study found that there was better reporting of electronic databases (998.7%) compared to other elements such as qualifications of the searcher (11.4%). 

Limitations stated by the author(s): 

None stated.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Review

Appraisal of: “Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Res Synth Method. 2014(5):98-115.”

Short description: 

A discussion of current practice for documenting systematic review searches. The paper presents issues and recommendations that arise from results of a survey of 260 systematic review authors and information specialists.  The authors suggest specific elements to record during a search including database and non-database elements. Authors list the steps of the information management process supplemented by a discussion for each step. The paper concludes with implications for future research and recommends the use of templates to ensure complete reporting. A sample template is provided.

Limitations stated by the author(s): 

None stated.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Single study

Appraisal of: “Niederstadt C, Droste S. Reporting and presenting information retrieval processes: the need for optimizing common practice in health technology assessment. Int J Technol Assess Health Care. 2010;26(4):450-7.”

Short description: 

The authors aim to promote good information retrieval practices by presenting the retrieval and transcription of complex search strategies in a transparent way and to document more accurately the search strategies. Conducted a comprehensive search for reporting standards and describe elements of search strategy reporting from the standards.  The authors make recommendations for reporting the whole information retrieval process, which consist of the following steps: 1. Defining search components using PICO and additional components 2. Defining the search models. 3. Selection of information sources 4. Transcripts of search strategies. They provide templates with examples for each of the steps.  They also recommend transcribing search details from PubMed and using the PRESS checklist for peer review of the search.

Limitations stated by the author(s): 

None stated.

Limitations stated by the reviewer(s): 
A final listing of all the included reporting standards found in the search would have been helpful.
Study Type: 
Review

Appraisal of: “Maggio LA, Tannery NH, Kanter SL. Reproducibility of Literature Search Reporting in Medical Education Reviews. Acad Med. 2011;86(8):1049-54.”

Short description: 

The aim of this paper was to examine the reproducibility of search strategies as reported in medical education literature reviews.  The findings indicate that documentation of search strategies in medical education reviews is highly variable and none of the selected reviews included reproducible searches.  The authors make recommendations for reporting search strategies and create checklist of items to be included to allow for reproducibility.

Limitations stated by the author(s): 

The authors only looked at reviews from three journals that focus on medical education and are not indexed in Medline. They did not look at the clinical literature on medical education.  Furthermore, they only looked at reviews from 2009, which may not be representative of years earlier and later.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Review

Appraisal of: “Finfgeld-Connett D, Johnson ED. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. J Adv Nurs. 2013;69(1):194-204.”

Short description: 

This article aims to examine literature searches of qualitative systematic reviews to add to the knowledge base and inform searching practice in this area. The authors identified these reviews through known relevant papers, and a literature search of Scopus and CINAHL.  The authors discuss expansive versus exhaustive literature searches and that searching for qualitative literature involves a non-linear approach.  As a result, documenting these types of searches should include detailed accounts of their search processes, electronic databases and grey literature sources, and why they were selected, how keywords emerged, how barriers were overcome, whether or not citations were systematically tracked and how it was decided to stop the search. These items should be included, as they are not always clearly and accurately presented. 

Limitations stated by the author(s): 

This is not a comprehensive sample –“These findings do not account for the actual numbers of qualitative versus quantitative research studies”

Limitations stated by the reviewer(s): 
There was no mention of Booth’s article : Booth AC. "Brimful of STARLITE": toward standards for reporting literature searches. J Med Libr Assoc. 2006;94:421-9, e205 as one of the first standards for reporting search strategies for qualitative systematic reviews.
Study Type: 
Review

Appraisal of: “Booth AC. "Brimful of STARLITE": toward standards for reporting literature searches. J Med Libr Assoc. 2006;94:421-9, e205.”

Short description: 

The aim of this paper is to stimulate improvements in conducting and reporting both qualitative and quantitative systematic reviews. The authors surveyed 44 reports of qualitative systematic reviews, characterized techniques used to identify articles for inclusion and proposed standards for reporting of literature searches called STARLITE which includes sampling strategy, type of study, approaches, range of years, limits, inclusion and exclusion, terms used and electronic sources. Findings from this work can inform groups of information specialists including the Cochrane Collaboration Information Retrieval Methods Group.

Limitations stated by the author(s): 

Only one reviewer was used to make judgments on inclusion and exclusion which opens the possibility of bias.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Review

Appraisal of: “Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Res Synth Method. 2015(1):87.”

Short description: 

This paper provides an overview of recommendations for reporting a search strategy when conducting a research synthesis. The methodology consisted of: 1) retrieving existing reporting guidelines for research syntheses. 2) integrating all their recommendations into a single document. From the compilation of recommendations, gaps were identified. Additional recommendations were integrated based on comprehensive searching experience of the authors, and based on any missing information that would be needed to a) replicate the synthesis and b) evaluate its quality.  3) The final set of literature search reporting guidelines was pilot tested. 

Limitations stated by the author(s): 

None stated.

Limitations stated by the reviewer(s): 
It was not clearly stated which guidelines recommend which reporting items. It would also be helpful to indicate more clearly the identified gaps in reporting standards for search strategies.

Appraisal of: "Briscoe S. Web searching for systematic reviews: a case study of reporting standards in the UK Health Technology Assessment programme. BMC Research Notes. 2015;8:153-."

Short description: 

The aim of this paper was to assess the reporting of searching the web for systematic reviews conducted by the UK HTA programme by identifying and examining HTA reports from 2004-2013. The study makes recommendations about reporting this type of searching to ensure transparency and reproducibility. Findings indicate that in the majority of cases, only the names of web sites or search engines are listed. Recommendations are made for including the dates searched, URL, search terms and the search results.

Limitations stated by the author(s): 

The web searching assessed was likely to have been only a peripheral part of an already supplemental search.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Review
Syndicate content