B. Designing strategies - general

Scope: use this term to index papers reporting explorations of the whole process of strategy design and testing; it may be added alongside more specific techniques – see below; may also be used in addition to ‘Guidance document’

Appraisal of: "Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev, 5(1). doi:10.1186/s13643-016-0249-x"

Reviewer(s): 
Short description: 

This review provides a systematic overview of the available published evidence of searching methods to inform qualitative evidence synthesis (QES). The author sought to assess and identify:

1) the current state of knowledge in relation to searching for qualitative evidence

2) the robustness of the evidence base

3) research gaps and future priorities.

The studies were obtained from the Reference Manager database of the Cochrane Qualitative Methods Group’s study register, of which the author is responsible for updating and maintaining. Supplementary citation searches via Google Scholar was also carried out for 15 key papers. 113 studies were assessed for inclusion. Quality assessment of the included studies was not deemed feasible due to a large proportion of the included studies providing only narrative findings, the lack of a common appraisal instrument and the high levels of heterogeneity across the remaining studies.

The evidence underpinning systematic approaches to searching for qualitative evidence is classified and summarized within one or more of eight headings/ “7 S structure/ framework”: overviews and methodological guidelines, sampling, sources, structured questions, search procedures, search strategies and filters, supplementary strategies and standards. The author summarizes the available evidence and key issues within each section and makes recommendations for further empirical research. Table 7 breaks down the key starting principles in reference to the “7S structure” of searching to inform qualitative evidence synthesis to inform future guidance and Table 8 provides an overview of research priorities.

The review concludes that there is a lack of empirical data to inform information retrieval for QES and that the strength of the evidence is weak and largely based on personal/ professional experience and case studies. Advances have been made in reporting QES, however, validated standards are lacking. 

Limitations stated by the author(s): 

The author states that in order for studies to be included, the references needed to include terms specifically related to searching or retrieval in their titles or abstracts, cite a number of key texts, or be referred to from previously identified items. The full-text of all papers reporting QES were not examined. There is a possibility that potentially relevant reviews reporting emerging information retrieval methods that were not reported in the title or abstract were missed. However, these risks are offset by the sensitive search approach and the currency and comprehensiveness of the Cochrane Qualitative Methods Group study register. The author also notes that some papers were excluded as they did not distinguish between qualitative and quantitative approaches, which could potentially be useful for mixed methods reviews.

Limitations stated by the reviewer(s): 
1. This was a single reviewer methodology review so judgements on eligibility and interpretations of potential significance and contribution of individual studies may not be consistent and/or reproducible. [Study Selection Bias] 2. Although as a methodology review there is no formal requirement to follow PRISMA reporting standards this review may have benefited from more complete and transparent reporting. [Incomplete Reporting Bias] 3. As the reviewer was author on a high proportion of included studies this may have consciously impacted on study identification and subconsciously on study selection and interpretation. [Citation Bias; Observer Bias]
Study Type: 
Single study

Appraisal of: "Sampson M. Complementary approaches to searching MEDLINE may be sufficient for updating existing systematic reviews. J Clin Epidemiol 2016; 78: 108-115."

Short description: 

The aim of the article was to test whether the combined approach of a focused Boolean search paired with a second search using the similar articles feature of PubMed or support vector machine (SVM) can yield high recall with reasonable precision.

The general approach of a Boolean plus a ranking search is effective in MEDLINE retrieval for systematic reviews. Very high levels of identification of relevant MEDLINE records, with adequate precision, are possible using a focused Boolean search complemented by a document similarity or ranking method.

Limitations stated by the author(s): 

There are two limitations to our proposed strategy. Other databases should be searched in the unusual event that numerous studies, representing more than a small proportion of the total N, are not included in MEDLINE. Second, when it is important to find articles too new to be indexed by MEDLINE, systematic reviewers may wish to conduct a simple PubMed search limited to the nonindexed subsets.

Limitations stated by the reviewer(s): 
It is not possible to reproduce or use the support vector machine functionality.
Study Type: 
Single study

Appraisal of: Thielen FW, Van Mastrigt G, Burgers LT, Bramer WM, Majoie H, Evers S, Kleijnen J. How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3). Exp

Short description: 

This article is the second in a three-part series on how to prepare a systematic review of economic evaluations. It provides a good overview of the literature on how to select relevant databases and develop a search strategy for retrieving economic evaluations. While the main target audience of the article is developers of clinical practice guidelines, the process described is helpful for systematic review researchers as well as those undertaking health technology assessments. The authors identify four steps in the search process: (1) selecting relevant databases (basic, specific and optional); (2) developing a comprehensive search strategy; (3) performing the searches (including documentation), and (4) selecting the relevant studies. The authors note the recent discontinuation of two health economics databases (HEED and NHS EED), which has resulted in an increased reliance on the use of search filters designed to capture economic evaluations. Also noted are unsettled issues, such as the lack of consensus on how many and which databases should be searched, as well a lack of uniform guidance on the methodology of developing a sound search strategy. Validated search filters and automated processes may help to overcome problems created by the lack of health economics-specific databases.

Limitations stated by the author(s): 

No limitations stated by the study authors.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Review

Appraisal of: O'Leary N, Tiernan E, Walsh D, Lucey N, Kirkova J, Davis MP. The pitfalls of a systematic MEDLINE review in palliative medicine: symptom assessment instruments. Am J Hosp Palliat Care. 2007; 24(3): 181-184.

Short description: 

The authors describe the usefulness of the related articles link in PubMed as part of a systematic MEDLINE review. Of 1181 citations identified, through a PubMed search, 10 articles met the indusion criteria. as did a further 21 articles identified through hand-searching the references of the 10 articles. The PubMed related articles link of the 10 articles yielded 15% of all reports finally included in the systematic review. The authors suggest that it is a useful tool in PubMed for revietrieving complex evidence.

Limitations stated by the author(s): 

No limitations stated by the authors.

Limitations stated by the reviewer(s): 
Different numbers are described in the result section: references that met the inclusion criteria and references finally included in the systematic review. This selection process is not clearly documented and it is not clear how many references finally included in the systematic review are identified through which source.
Study Type: 
Single study

Appraisal of: Droste S, Dintsios CM, Gerber A. Information on ethical issues in health technology assessment: How and where to find them. Int J Technol Assess Health Care 2010; 26(4): 441–449.

Short description: 

The aim of the article is to present a procedure to identify information on ethical implications of the use and value-related decision-making in the assessment of health technologies. The purpose of the project is to develop, propose and test a step-by step workflow adapted to the specific issues of information on ethical issues (searching for arguments and qualitative data).

The article provides information on the background of developing the methodological approach as well as on the 8 working steps - Step 1 “Translation of the search question using the PICO scheme and additional components” to Step 8 “Final quality check and calculation of precision and recall”. The article is supplemented by the application results of the example “Ethical issues of autologous stem cell transplantation in metastatic breast cancer”.

Limitations stated by the author(s): 
  • Not all ethical issues related to health technologies can be identified by means of the approach described.
  • Some ethical issues cannot be found with systematic approaches - additional non-systematic searching may be useful.
  • The ethical issues relevant to the health technology investigated should be known in more detail before starting the systematic information retrieval.
  • Alleged innovations cannot be identified as such by the methodological approach.
Limitations stated by the reviewer(s): 
- Reference(s) to the validation work could have been provided. - The authors claim in the objective that if choosing the proper methodological approach: the central ethical dimensions then should be answerable. This seems like a too strong claim, what the authors do is to provide a brilliant methodological approach to address central ethical dimensions, to the extent there is anything published concerning these. Whether that answers ethical concerns is to some extent another matter and might call for a separate ethics analysis. Both since the sources might give conflicting answers, or give no answers at all. - Lack references to the database Philosopher’s index which in some cases might provide sources for a more general but relevant ethical discussion.
Study Type: 
Single study

Appraisal of: Bak G, Mierzwinski-Urban M, Fitzsimmons H, Morrison A, Maden-Jenkins M. A pragmatic critical appraisal instrument for search filters: introducing the CADTH CAI. Health Info Libr J. 2009 Sep;26(3):211-9

Reviewer(s): 
Short description: 

This paper describes the search for a pragmatic critical appraisal instrument (CAI) that experienced systematic review searchers can use in the selection of robust search filters. A focused search of the literature identified a single candidate - the critical appraisal tool developed by Michelle Jenkins. Testing of this showed that its use requires an in-depth understanding of search filter methodology and statistics that not all information specialists may possess. In light of these findings, the authors substantially revised the tool to consist of 6 domains and 10 closed questions. Instructions about how to use the tool are provided. It features a key question about whether the filter being appraised is up to date.

Limitations stated by the author(s): 

The authors acknowledged that the CADTH CAI is a pragmatic tool for use by information specialists and suggested that research methodologists would be better served using the original, more sophisticated, appraisal tool developed by Jenkins. Even less experienced searchers with little knowledge of controlled vocabularies and search filter methodologies could potentially find the CADTH CAI difficult to use. They also highlighted the lack of research demonstrating whether the most relevant studies are included in the smaller results sets obtained when a filter is incorporated into the search strategy.

Limitations stated by the reviewer(s): 
No additional limitations were detected by the reviewers.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Glanville J, Bayliss S, Booth A, Dundar Y, Fernandes H, Fleeman N D, Foster L, Fraser C, Fry-Smith A, Golder S, Lefebvre C, Miller C, Paisley S, Payne L, Price A and Welch K. So many filters, so little time: the development of a search

Reviewer(s): 
Short description: 

Study describes the development and testing of a tool used to assess the quality of search filters designed to retrieve records for studies with a specific research design. The resulting tool comprises a search filter appraisal checklist that reports both filter design methods and search performance i.e. sensitivity and precision. As well as helping information specialists select the most appropriate filter, it can also be used by researchers developing search filters by indicating what aspects of filter development should be reported to ensure that the methodology can be successfully evaluated.

Limitations stated by the author(s): 

The performance of the checklist has not been assessed using independent assessors and a range of filters.

Subsequent to the development of the issg tool, CADTH produced a critical appraisal and ranking tool. No comparison between the two has been conducted to date.

The issg checklist has restricted its focus to search filters designed to retrieve studies with a specific research type (e.g. RCT or diagnostic test accuracy study). There has not yet been any exploration of whether any of the processes could also be applied to other search filters.

Limitations stated by the reviewer(s): 
Since 2008 when this project was undertaken there has been increased interest in the use of text mining techniques as an alternative method of identifying and retrieving relevant information from large databases. Although still at the research stage currently, if these methods become well established the need for search filters will decrease over time.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Wilczynski NL, Morgan D, Haynes RB; Hedges Team. An overview of the design and methods for retrieving high-quality studies for clinical care. BMC Med Inform Decis Mak. 2005 Jun 21;5:20.

Reviewer(s): 
Short description: 

The authors describe the methods they use to develop optimal search filters (hedges) to identify a range of study types: causation, prognosis, diagnosis, treatment, economics, clinical prediction guides, reviews, costs, and qualitative:  the filters were designed for MEDLINE and Embase. The authors built a large gold standard (reference set) by hand searching 170 journals for one year: 2000.  Relevant records were defined and were selected to represent best research methods. The gold standard records were then downloaded from  MEDLINE, Embase, CINAHL, and PsycINFO with the subject indexing assigned by each database.  Candidate search terms were identified from the gold standard records and consulting experts. The sensitivity, specificity, precision, and accuracy of unique search terms and combinations of search terms were calculated. Once the performance parameters of individual search terms were computed, the authors  selected individual terms for the construction of search strategies by choosing search terms with specific levels of sensitivity and specificity (which varied by database). The authors also used logistic regression to explore ways to improve filter performance. Strategies were developed in a random selection of 60% of the gold standard and validated in the remaining 40%. No statistical differences in performance were found between the two strategy development methods or between the test and validation results, so the majority of filter development used the Boolean approach and search strategies were developed using all records in the database.

The gold standard database numbered 60,330 records, each with up to 11 data fields. Filters were developed for studies of causation, prognosis, diagnosis, treatment, economics, clinical prediction guides, reviews, costs, and those of a qualitative nature.

Limitations stated by the author(s): 

The authors do not state any limitations to their methods.

Limitations stated by the reviewer(s): 
The journals contributing to the gold standard were largely general (internal medicine, general medical practice, mental health and general nursing practice) rather than specialty focused. There may be issues about generalizability to the latter journals. Records were selected for the gold standard according to the authors’ definition of the specific topics and best study designs for those topics. These definitions need to be compared to the definitions of any user. The authors decided, following testing, to use all of the database records to both develop and test the filters. Ideally validation data on a different reference standard should be provided.
Study Type: 
Single study

Appraisal of: Zechmeister-Koss I, Schnell-Inderst P, Zauner G. Appropriate evidence sources for populating decision analytic models within health technology assessment (HTA): a systematic review of HTA manuals and health economic guidelines. Med Decis Mak

Short description: 

This was a systematic review of HTA manuals and health economic guidelines focused on identifying search approaches and information sources to populate decision analytic models. 28 documents in English and German were identified. The overall messages from these guidelines were that searches should be as systematic as possible in terms of being transparent, but that only some model parameters, chiefly effects evidence, need by informed by systematic reviews. A large number of data sources of varying types may provide information for the parameters, but there seems to be little consensus about which are essential.

There are two online supplements. One lists excluded guidelines and the second provides some detail on each of the included guidelines.

Limitations stated by the author(s): 

The search was restricted to publications in English and German and to Western countries. The focus was on information to inform model parameters rather than other aspects of developing decision models. The information is generic rather than focusing on specific model approaches.

Limitations stated by the reviewer(s): 
The resources listed in individual guidelines are not presented. The search approaches suggested in the individual guidelines are not summarized in detail.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Droste S, Rixen S. [Information on legal issues in health technologies: methodological proposal to identify them in a systematic and comprehensible manner]. Z Evid Fortbild Qual Gesundhwes 2012;106:509-522.

Reviewer(s): 
Short description: 

The aim of this study was to develop a methodological approach for systematic retrieval of information on legal issues associated with health technologies, and to present this proposal in detail. In the absence of existing relevant methodological approaches verified by an extensive literature search, the authors developed an information retrieval procedure consisting of the following 8 work steps:

1. Pre-search: identification of the relevant rules, regulations and patient-related legal issues
2. Translation of the search question (using adapted PICO scheme)
3. Concept building
4. Identification of synonyms
5. Selection of relevant information sources
6. Design of the search strategies
7. Execution and quality check
8. Saving the results and reporting.

This procedure was applied on three examples. One of them, "Ultrasound screening in pregnancy", was used in this article to illustrate the proposed work steps in detail. Since there is a substantial overlap between legal, ethical and social issues of health technologies, the authors point out the potential benefit of joint information retrieval processes for these three aspects. This publication is in German. Abstract is provided in English.

Limitations stated by the author(s): 
The developed procedure was applied on three specific research questions. The example on ‘‘Ultrasound screening in pregnancy’’ was reported in detail. The authors discussed the representative value of this presented example and the need for further evaluation of the external validity of the proposed procedure.
Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewer.
Study Type: 
Single study
Related Chapters: 
Syndicate content