Guidance document

Scope: a guideline, checklist or other guide describing how to go about for information retrieval for HTA or other purposes relevant to HTA

Appraisal of: "Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev, 5(1). doi:10.1186/s13643-016-0249-x"

Reviewer(s): 
Short description: 

This review provides a systematic overview of the available published evidence of searching methods to inform qualitative evidence synthesis (QES). The author sought to assess and identify:

1) the current state of knowledge in relation to searching for qualitative evidence

2) the robustness of the evidence base

3) research gaps and future priorities.

The studies were obtained from the Reference Manager database of the Cochrane Qualitative Methods Group’s study register, of which the author is responsible for updating and maintaining. Supplementary citation searches via Google Scholar was also carried out for 15 key papers. 113 studies were assessed for inclusion. Quality assessment of the included studies was not deemed feasible due to a large proportion of the included studies providing only narrative findings, the lack of a common appraisal instrument and the high levels of heterogeneity across the remaining studies.

The evidence underpinning systematic approaches to searching for qualitative evidence is classified and summarized within one or more of eight headings/ “7 S structure/ framework”: overviews and methodological guidelines, sampling, sources, structured questions, search procedures, search strategies and filters, supplementary strategies and standards. The author summarizes the available evidence and key issues within each section and makes recommendations for further empirical research. Table 7 breaks down the key starting principles in reference to the “7S structure” of searching to inform qualitative evidence synthesis to inform future guidance and Table 8 provides an overview of research priorities.

The review concludes that there is a lack of empirical data to inform information retrieval for QES and that the strength of the evidence is weak and largely based on personal/ professional experience and case studies. Advances have been made in reporting QES, however, validated standards are lacking. 

Limitations stated by the author(s): 

The author states that in order for studies to be included, the references needed to include terms specifically related to searching or retrieval in their titles or abstracts, cite a number of key texts, or be referred to from previously identified items. The full-text of all papers reporting QES were not examined. There is a possibility that potentially relevant reviews reporting emerging information retrieval methods that were not reported in the title or abstract were missed. However, these risks are offset by the sensitive search approach and the currency and comprehensiveness of the Cochrane Qualitative Methods Group study register. The author also notes that some papers were excluded as they did not distinguish between qualitative and quantitative approaches, which could potentially be useful for mixed methods reviews.

Limitations stated by the reviewer(s): 
1. This was a single reviewer methodology review so judgements on eligibility and interpretations of potential significance and contribution of individual studies may not be consistent and/or reproducible. [Study Selection Bias] 2. Although as a methodology review there is no formal requirement to follow PRISMA reporting standards this review may have benefited from more complete and transparent reporting. [Incomplete Reporting Bias] 3. As the reviewer was author on a high proportion of included studies this may have consciously impacted on study identification and subconsciously on study selection and interpretation. [Citation Bias; Observer Bias]
Study Type: 
Single study

Appraisal of: "Arber M, Wood H, Isojarvi J, Glanville J. Which information sources should be used to identify studies for systematic reviews of economic evaluations in healthcare? Value Health. 2017 Oct/Nov;20(9):A738. Abstract PRM46."

Reviewer(s): 
Short description: 

NHS EED and HEED, two key databases for retrieving health economics information, have closed. Based on this changed landscape, this abstract assesses which database are now the best sources of information for retrieving economic evaluations for models and systematic reviews. The authors built a quasi-gold standard database of 351 records compiled from 46 systematic reviews of economic evaluations. Nine databases were searched for each record. Embase had the highest yield (0.89), followed by Scopus (0.84) and MEDLINE and PubMed (both 0.81). The HTA database identified the highest number of unique citations (13/351). All nine database combined retrieved 337/351 (0.96) records. The authors conclude that for most systematic reviews, Embase, the HTA database and either PubMed or MEDLINE are likely sufficient to identify economic evaluations found in bibliographic databases. Searching a multi-disciplinary database may also be useful, especially in non-clinical topics. Supplementary search techniques may be more efficient than searching a larger number of databases.

MEDLINE search strategies reported in source systematic reviews were also assessed. 10/29 (34.5%) of re-run search strategies missed at least one of the included records found in MEDLINE (with 25 citations missed in total). Weaknesses in the population or intervention concepts, rather than the economics concept, were identified as negatively impacting search retrieval.

Limitations stated by the author(s): 

None stated.

Limitations stated by the reviewer(s): 
Since the results were published only in abstract form, very limited information was presented on methods, results and conclusions.
Study Type: 
Single study

Appraisal of: "Ara R, Brazier J, Peasgood T, Paisley S. The identification, review and synthesis of health state utility values from the literature. Pharmacoeconomics. 2017;35:43–55."

Short description: 

This paper is a guidance document providing an overview of how health state utility values can be identified, reviewed and synthesised when conducting systematic reviews. The paper includes a case study of a review in osteoporosis-related conditions.

In terms of study identification, the authors note that a range of study designs could be relevant and that a number of instruments could be required including condition-specific preference based measures or generic preference based measures. They recommend that a variety of resources and methods should be used to identify studies. As well as electronic databases, searchers should also look at reference lists, conduct key author and citation searches and contact experts.

The authors recommend caution in using filters too early in the search process, to avoid missing potentially relevant studies. The authors also note the absence of dedicated subject headings within MeSH and EMTREE, and that although general subject headings such as ‘Quality of life’ will yield relevant studies, they are likely to demonstrate poor precision. Free text terms should be included in searches and are categorised as general terms (such as QALY), instrument specific terms (such as EQ-5D) and terms describing methods of utility elicitation such as standard gamble.

Limitations stated by the author(s): 

The authors discuss the issues of searching for utility studies including current limitations. They do not discuss limitations of their review and do not provide details of their methods in this publication.

Limitations stated by the reviewer(s): 
The overview and recommendations in relation to searching do not seem controversial.
Study Type: 
Single study

Appraisal of: Thielen FW, Van Mastrigt G, Burgers LT, Bramer WM, Majoie H, Evers S, Kleijnen J. How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3). Exp

Short description: 

This article is the second in a three-part series on how to prepare a systematic review of economic evaluations. It provides a good overview of the literature on how to select relevant databases and develop a search strategy for retrieving economic evaluations. While the main target audience of the article is developers of clinical practice guidelines, the process described is helpful for systematic review researchers as well as those undertaking health technology assessments. The authors identify four steps in the search process: (1) selecting relevant databases (basic, specific and optional); (2) developing a comprehensive search strategy; (3) performing the searches (including documentation), and (4) selecting the relevant studies. The authors note the recent discontinuation of two health economics databases (HEED and NHS EED), which has resulted in an increased reliance on the use of search filters designed to capture economic evaluations. Also noted are unsettled issues, such as the lack of consensus on how many and which databases should be searched, as well a lack of uniform guidance on the methodology of developing a sound search strategy. Validated search filters and automated processes may help to overcome problems created by the lack of health economics-specific databases.

Limitations stated by the author(s): 

No limitations stated by the study authors.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Review

Appraisal of: “Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. J Clin Epidemiol. 2008;61(8):748-54.”

Short description: 

The aim of this paper was to identify validated search reporting instruments and to compare reported and recommended reporting practices.  11 instruments and 18 different reporting items were identified.  The highest number of reporting items used by an instrument was 11. The study found that there was better reporting of electronic databases (998.7%) compared to other elements such as qualifications of the searcher (11.4%). 

Limitations stated by the author(s): 

None stated.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Review

Appraisal of: “Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Res Synth Method. 2014(5):98-115.”

Short description: 

A discussion of current practice for documenting systematic review searches. The paper presents issues and recommendations that arise from results of a survey of 260 systematic review authors and information specialists.  The authors suggest specific elements to record during a search including database and non-database elements. Authors list the steps of the information management process supplemented by a discussion for each step. The paper concludes with implications for future research and recommends the use of templates to ensure complete reporting. A sample template is provided.

Limitations stated by the author(s): 

None stated.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Single study

Appraisal of: “Niederstadt C, Droste S. Reporting and presenting information retrieval processes: the need for optimizing common practice in health technology assessment. Int J Technol Assess Health Care. 2010;26(4):450-7.”

Short description: 

The authors aim to promote good information retrieval practices by presenting the retrieval and transcription of complex search strategies in a transparent way and to document more accurately the search strategies. Conducted a comprehensive search for reporting standards and describe elements of search strategy reporting from the standards.  The authors make recommendations for reporting the whole information retrieval process, which consist of the following steps: 1. Defining search components using PICO and additional components 2. Defining the search models. 3. Selection of information sources 4. Transcripts of search strategies. They provide templates with examples for each of the steps.  They also recommend transcribing search details from PubMed and using the PRESS checklist for peer review of the search.

Limitations stated by the author(s): 

None stated.

Limitations stated by the reviewer(s): 
A final listing of all the included reporting standards found in the search would have been helpful.
Study Type: 
Review

Appraisal of: “Maggio LA, Tannery NH, Kanter SL. Reproducibility of Literature Search Reporting in Medical Education Reviews. Acad Med. 2011;86(8):1049-54.”

Short description: 

The aim of this paper was to examine the reproducibility of search strategies as reported in medical education literature reviews.  The findings indicate that documentation of search strategies in medical education reviews is highly variable and none of the selected reviews included reproducible searches.  The authors make recommendations for reporting search strategies and create checklist of items to be included to allow for reproducibility.

Limitations stated by the author(s): 

The authors only looked at reviews from three journals that focus on medical education and are not indexed in Medline. They did not look at the clinical literature on medical education.  Furthermore, they only looked at reviews from 2009, which may not be representative of years earlier and later.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Review

Appraisal of: “Finfgeld-Connett D, Johnson ED. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. J Adv Nurs. 2013;69(1):194-204.”

Short description: 

This article aims to examine literature searches of qualitative systematic reviews to add to the knowledge base and inform searching practice in this area. The authors identified these reviews through known relevant papers, and a literature search of Scopus and CINAHL.  The authors discuss expansive versus exhaustive literature searches and that searching for qualitative literature involves a non-linear approach.  As a result, documenting these types of searches should include detailed accounts of their search processes, electronic databases and grey literature sources, and why they were selected, how keywords emerged, how barriers were overcome, whether or not citations were systematically tracked and how it was decided to stop the search. These items should be included, as they are not always clearly and accurately presented. 

Limitations stated by the author(s): 

This is not a comprehensive sample –“These findings do not account for the actual numbers of qualitative versus quantitative research studies”

Limitations stated by the reviewer(s): 
There was no mention of Booth’s article : Booth AC. "Brimful of STARLITE": toward standards for reporting literature searches. J Med Libr Assoc. 2006;94:421-9, e205 as one of the first standards for reporting search strategies for qualitative systematic reviews.
Study Type: 
Review

Appraisal of: “Booth AC. "Brimful of STARLITE": toward standards for reporting literature searches. J Med Libr Assoc. 2006;94:421-9, e205.”

Short description: 

The aim of this paper is to stimulate improvements in conducting and reporting both qualitative and quantitative systematic reviews. The authors surveyed 44 reports of qualitative systematic reviews, characterized techniques used to identify articles for inclusion and proposed standards for reporting of literature searches called STARLITE which includes sampling strategy, type of study, approaches, range of years, limits, inclusion and exclusion, terms used and electronic sources. Findings from this work can inform groups of information specialists including the Cochrane Collaboration Information Retrieval Methods Group.

Limitations stated by the author(s): 

Only one reviewer was used to make judgments on inclusion and exclusion which opens the possibility of bias.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Review
Syndicate content