B. Designing strategies - general

Scope: use this term to index papers reporting explorations of the whole process of strategy design and testing; it may be added alongside more specific techniques – see below; may also be used in addition to ‘Guidance document’

Appraisal of: "Sampson M. Complementary approaches to searching MEDLINE may be sufficient for updating existing systematic reviews. J Clin Epidemiol 2016; 78: 108-115."

Short description: 

The aim of the article was to test whether the combined approach of a focused Boolean search paired with a second search using the similar articles feature of PubMed or support vector machine (SVM) can yield high recall with reasonable precision.

The general approach of a Boolean plus a ranking search is effective in MEDLINE retrieval for systematic reviews. Very high levels of identification of relevant MEDLINE records, with adequate precision, are possible using a focused Boolean search complemented by a document similarity or ranking method.

Limitations stated by the author(s): 

There are two limitations to our proposed strategy. Other databases should be searched in the unusual event that numerous studies, representing more than a small proportion of the total N, are not included in MEDLINE. Second, when it is important to find articles too new to be indexed by MEDLINE, systematic reviewers may wish to conduct a simple PubMed search limited to the nonindexed subsets.

Limitations stated by the reviewer(s): 
It is not possible to reproduce or use the support vector machine functionality.
Study Type: 
Single study

Appraisal of: Thielen FW, Van Mastrigt G, Burgers LT, Bramer WM, Majoie H, Evers S, Kleijnen J. How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3). Exp

Short description: 

This article is the second in a three-part series on how to prepare a systematic review of economic evaluations. It provides a good overview of the literature on how to select relevant databases and develop a search strategy for retrieving economic evaluations. While the main target audience of the article is developers of clinical practice guidelines, the process described is helpful for systematic review researchers as well as those undertaking health technology assessments. The authors identify four steps in the search process: (1) selecting relevant databases (basic, specific and optional); (2) developing a comprehensive search strategy; (3) performing the searches (including documentation), and (4) selecting the relevant studies. The authors note the recent discontinuation of two health economics databases (HEED and NHS EED), which has resulted in an increased reliance on the use of search filters designed to capture economic evaluations. Also noted are unsettled issues, such as the lack of consensus on how many and which databases should be searched, as well a lack of uniform guidance on the methodology of developing a sound search strategy. Validated search filters and automated processes may help to overcome problems created by the lack of health economics-specific databases.

Limitations stated by the author(s): 

No limitations stated by the study authors.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Review

Appraisal of: O'Leary N, Tiernan E, Walsh D, Lucey N, Kirkova J, Davis MP. The pitfalls of a systematic MEDLINE review in palliative medicine: symptom assessment instruments. Am J Hosp Palliat Care. 2007; 24(3): 181-184.

Short description: 

The authors describe the usefulness of the related articles link in PubMed as part of a systematic MEDLINE review. Of 1181 citations identified, through a PubMed search, 10 articles met the indusion criteria. as did a further 21 articles identified through hand-searching the references of the 10 articles. The PubMed related articles link of the 10 articles yielded 15% of all reports finally included in the systematic review. The authors suggest that it is a useful tool in PubMed for revietrieving complex evidence.

Limitations stated by the author(s): 

No limitations stated by the authors.

Limitations stated by the reviewer(s): 
Different numbers are described in the result section: references that met the inclusion criteria and references finally included in the systematic review. This selection process is not clearly documented and it is not clear how many references finally included in the systematic review are identified through which source.
Study Type: 
Single study

Appraisal of: Droste S, Dintsios CM, Gerber A. Information on ethical issues in health technology assessment: How and where to find them. Int J Technol Assess Health Care 2010; 26(4): 441–449.

Short description: 

The aim of the article is to present a procedure to identify information on ethical implications of the use and value-related decision-making in the assessment of health technologies. The purpose of the project is to develop, propose and test a step-by step workflow adapted to the specific issues of information on ethical issues (searching for arguments and qualitative data).

The article provides information on the background of developing the methodological approach as well as on the 8 working steps - Step 1 “Translation of the search question using the PICO scheme and additional components” to Step 8 “Final quality check and calculation of precision and recall”. The article is supplemented by the application results of the example “Ethical issues of autologous stem cell transplantation in metastatic breast cancer”.

Limitations stated by the author(s): 
  • Not all ethical issues related to health technologies can be identified by means of the approach described.
  • Some ethical issues cannot be found with systematic approaches - additional non-systematic searching may be useful.
  • The ethical issues relevant to the health technology investigated should be known in more detail before starting the systematic information retrieval.
  • Alleged innovations cannot be identified as such by the methodological approach.
Limitations stated by the reviewer(s): 
- Reference(s) to the validation work could have been provided. - The authors claim in the objective that if choosing the proper methodological approach: the central ethical dimensions then should be answerable. This seems like a too strong claim, what the authors do is to provide a brilliant methodological approach to address central ethical dimensions, to the extent there is anything published concerning these. Whether that answers ethical concerns is to some extent another matter and might call for a separate ethics analysis. Both since the sources might give conflicting answers, or give no answers at all. - Lack references to the database Philosopher’s index which in some cases might provide sources for a more general but relevant ethical discussion.
Study Type: 
Single study

Appraisal of: Bak G, Mierzwinski-Urban M, Fitzsimmons H, Morrison A, Maden-Jenkins M. A pragmatic critical appraisal instrument for search filters: introducing the CADTH CAI. Health Info Libr J. 2009 Sep;26(3):211-9

Reviewer(s): 
Short description: 

This paper describes the search for a pragmatic critical appraisal instrument (CAI) that experienced systematic review searchers can use in the selection of robust search filters. A focused search of the literature identified a single candidate - the critical appraisal tool developed by Michelle Jenkins. Testing of this showed that its use requires an in-depth understanding of search filter methodology and statistics that not all information specialists may possess. In light of these findings, the authors substantially revised the tool to consist of 6 domains and 10 closed questions. Instructions about how to use the tool are provided. It features a key question about whether the filter being appraised is up to date.

Limitations stated by the author(s): 

The authors acknowledged that the CADTH CAI is a pragmatic tool for use by information specialists and suggested that research methodologists would be better served using the original, more sophisticated, appraisal tool developed by Jenkins. Even less experienced searchers with little knowledge of controlled vocabularies and search filter methodologies could potentially find the CADTH CAI difficult to use. They also highlighted the lack of research demonstrating whether the most relevant studies are included in the smaller results sets obtained when a filter is incorporated into the search strategy.

Limitations stated by the reviewer(s): 
No additional limitations were detected by the reviewers.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Glanville J, Bayliss S, Booth A, Dundar Y, Fernandes H, Fleeman N D, Foster L, Fraser C, Fry-Smith A, Golder S, Lefebvre C, Miller C, Paisley S, Payne L, Price A and Welch K. So many filters, so little time: the development of a search

Reviewer(s): 
Short description: 

Study describes the development and testing of a tool used to assess the quality of search filters designed to retrieve records for studies with a specific research design. The resulting tool comprises a search filter appraisal checklist that reports both filter design methods and search performance i.e. sensitivity and precision. As well as helping information specialists select the most appropriate filter, it can also be used by researchers developing search filters by indicating what aspects of filter development should be reported to ensure that the methodology can be successfully evaluated.

Limitations stated by the author(s): 

The performance of the checklist has not been assessed using independent assessors and a range of filters.

Subsequent to the development of the issg tool, CADTH produced a critical appraisal and ranking tool. No comparison between the two has been conducted to date.

The issg checklist has restricted its focus to search filters designed to retrieve studies with a specific research type (e.g. RCT or diagnostic test accuracy study). There has not yet been any exploration of whether any of the processes could also be applied to other search filters.

Limitations stated by the reviewer(s): 
Since 2008 when this project was undertaken there has been increased interest in the use of text mining techniques as an alternative method of identifying and retrieving relevant information from large databases. Although still at the research stage currently, if these methods become well established the need for search filters will decrease over time.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Wilczynski NL, Morgan D, Haynes RB; Hedges Team. An overview of the design and methods for retrieving high-quality studies for clinical care. BMC Med Inform Decis Mak. 2005 Jun 21;5:20.

Reviewer(s): 
Short description: 

The authors describe the methods they use to develop optimal search filters (hedges) to identify a range of study types: causation, prognosis, diagnosis, treatment, economics, clinical prediction guides, reviews, costs, and qualitative:  the filters were designed for MEDLINE and Embase. The authors built a large gold standard (reference set) by hand searching 170 journals for one year: 2000.  Relevant records were defined and were selected to represent best research methods. The gold standard records were then downloaded from  MEDLINE, Embase, CINAHL, and PsycINFO with the subject indexing assigned by each database.  Candidate search terms were identified from the gold standard records and consulting experts. The sensitivity, specificity, precision, and accuracy of unique search terms and combinations of search terms were calculated. Once the performance parameters of individual search terms were computed, the authors  selected individual terms for the construction of search strategies by choosing search terms with specific levels of sensitivity and specificity (which varied by database). The authors also used logistic regression to explore ways to improve filter performance. Strategies were developed in a random selection of 60% of the gold standard and validated in the remaining 40%. No statistical differences in performance were found between the two strategy development methods or between the test and validation results, so the majority of filter development used the Boolean approach and search strategies were developed using all records in the database.

The gold standard database numbered 60,330 records, each with up to 11 data fields. Filters were developed for studies of causation, prognosis, diagnosis, treatment, economics, clinical prediction guides, reviews, costs, and those of a qualitative nature.

Limitations stated by the author(s): 

The authors do not state any limitations to their methods.

Limitations stated by the reviewer(s): 
The journals contributing to the gold standard were largely general (internal medicine, general medical practice, mental health and general nursing practice) rather than specialty focused. There may be issues about generalizability to the latter journals. Records were selected for the gold standard according to the authors’ definition of the specific topics and best study designs for those topics. These definitions need to be compared to the definitions of any user. The authors decided, following testing, to use all of the database records to both develop and test the filters. Ideally validation data on a different reference standard should be provided.
Study Type: 
Single study

Appraisal of: Zechmeister-Koss I, Schnell-Inderst P, Zauner G. Appropriate evidence sources for populating decision analytic models within health technology assessment (HTA): a systematic review of HTA manuals and health economic guidelines. Med Decis Mak

Short description: 

This was a systematic review of HTA manuals and health economic guidelines focused on identifying search approaches and information sources to populate decision analytic models. 28 documents in English and German were identified. The overall messages from these guidelines were that searches should be as systematic as possible in terms of being transparent, but that only some model parameters, chiefly effects evidence, need by informed by systematic reviews. A large number of data sources of varying types may provide information for the parameters, but there seems to be little consensus about which are essential.

There are two online supplements. One lists excluded guidelines and the second provides some detail on each of the included guidelines.

Limitations stated by the author(s): 

The search was restricted to publications in English and German and to Western countries. The focus was on information to inform model parameters rather than other aspects of developing decision models. The information is generic rather than focusing on specific model approaches.

Limitations stated by the reviewer(s): 
The resources listed in individual guidelines are not presented. The search approaches suggested in the individual guidelines are not summarized in detail.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Droste S, Rixen S. [Information on legal issues in health technologies: methodological proposal to identify them in a systematic and comprehensible manner]. Z Evid Fortbild Qual Gesundhwes 2012;106:509-522.

Reviewer(s): 
Short description: 

The aim of this study was to develop a methodological approach for systematic retrieval of information on legal issues associated with health technologies, and to present this proposal in detail. In the absence of existing relevant methodological approaches verified by an extensive literature search, the authors developed an information retrieval procedure consisting of the following 8 work steps:

1. Pre-search: identification of the relevant rules, regulations and patient-related legal issues
2. Translation of the search question (using adapted PICO scheme)
3. Concept building
4. Identification of synonyms
5. Selection of relevant information sources
6. Design of the search strategies
7. Execution and quality check
8. Saving the results and reporting.

This procedure was applied on three examples. One of them, "Ultrasound screening in pregnancy", was used in this article to illustrate the proposed work steps in detail. Since there is a substantial overlap between legal, ethical and social issues of health technologies, the authors point out the potential benefit of joint information retrieval processes for these three aspects. This publication is in German. Abstract is provided in English.

Limitations stated by the author(s): 
The developed procedure was applied on three specific research questions. The example on ‘‘Ultrasound screening in pregnancy’’ was reported in detail. The authors discussed the representative value of this presented example and the need for further evaluation of the external validity of the proposed procedure.
Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewer.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Golder S, Glanville J, Ginnelly L. Populating decision-analytic models: the feasibility and efficiency of database searching for individual parameters. Int J Technol Assess Health Care 2005 Summer;21(3):305-311.

Reviewer(s): 
Short description: 
The authors investigated the efficacy of searching selected databases to retrieve specific information to populate a decision-analytic (economic) model. Altogether 42 specific questions were developed from a single case study and then grouped into 18 search strategies. Information required to populate the model for the case study was divided into five categories: baseline event rates, health-related quality of life, resource use and cost units, relative treatment effects and antibiotic resistance. Databases searched included MEDLINE, Embase, the Incidence and Prevalence Database (IPD), Cochrane Database of Systematic Reviews (CDSR), the Database of Abstracts of Reviews of Effects (DARE), the Health Technology Assessment (HTA) database, Cochrane Central, NHS Economic Evaluation Database (NHS EED) and the Health Economic Evaluations Database (HEED). Results were presented describing the most useful databases for each of the five question types. Issues and strategies in developing searches for each question type were also discussed. Searching multiple databases using focused search strategies for each question resulted in the identification of important data for inclusion in the model. It was noted that while searches of this nature may not be highly sensitive, the search process nevertheless can still be systematic, transparent and well-documented.
Limitations stated by the author(s): 
The paper describes only one case study which retrieved a limited number of relevant citations, making it difficult to generalize to other decision-analytic models. The information required to populate a different decision-analytic model may require an adjustment to resources and strategies used. Project time and resources will dictate the acceptable level of focus in the search strategies as well as the number of source searched.
Limitations stated by the reviewer(s): 
National or regional sources of information may be required to find some relevant data to populate models. Grey literature searching is not discussed.
Study Type: 
Single study
Related Chapters: 
Syndicate content