Clinical effectiveness

Last revised: 
2019-05-14

Introduction

This domain focuses on the identification of evaluations of the efficacy or effectiveness of a technology (device, medicine, vaccine, procedure or system) or intervention. These evaluations focus on whether a technology works, as well as the magnitude of health benefits or harms caused by the technology. Searching for adverse effects is summarized in the SuRe Info Safety chapter, and diagnostic tests in the SuRe Info Diagnostic Accuracy chapter. Study designs used to assess clinical effectiveness of a technology or intervention include randomized controlled trials (RCTs), quasi-experimental studies, and observational studies. Clinical effectiveness searches will focus on the identification of reports of these study types (1,2).

This chapter is primarily based on Cochrane’s 2019 update of the Searching for and selecting studies chapter (Chapter 4) of the Cochrane Handbook for Systematic Reviews of Interventions (3) and the EUnetHTA guideline on information retrieval for systematic reviews and health technology assessments of clinical effectiveness (4).

This chapter is the result of extensive Cochrane work, and appraisals have not been prepared for studies in this chapter. Future chapter updates will include publication appraisals for new studies.

Sources to search

The choice and number of sources selected to search will depend on the research question, which resources can be accessed, and time and budget constraints (3,5). The Cochrane Handbook states the following databases should be searched as a minimum to identify as many relevant efficacy studies as possible and minimise the risk of publication bias:

  • Cochrane Central Register of Controlled Trials (CENTRAL)
  • MEDLINE
  • Embase

While CENTRAL includes records from MEDLINE, Embase, clinical trial registers and other bibliographic databases, the Cochrane Handbook (section 4.3.1.1) recommends supplementary searches of MEDLINE and Embase for comprehensiveness and currency since there is a time delay between records being indexed in MEDLINE/Embase and appearing in CENTRAL.

There are various different interfaces for these databases, some free and some fee-paying, check your own institution library to see which resources you can access. For more information see the SuRe Info section on Service providers and search interfaces.

Subject specific databases

The 2019 Cochrane Handbook (section 4.3.1.4) states that it is highly desirable to search appropriate subject-specific databases (3). There is mixed evidence on whether searching subject-specific databases adds value in terms of identifying additional unique references, and the decision about whether to search subject specific databases may depend on the research question or topic area (6–11).

Examples of subject-specific bibliographic databases are listed in an Appendix. (Further information will also be available in the Technical Supplement associated with the Cochrane Handbook.)

National/regional databases

The 2019 Cochrane Handbook (section 4.3.1.4) recommends searching national and regional databases (3), as these may index journals not included in international bibliographic databases such as MEDLINE and Embase, and could minimize the risk of language bias (12–14). There may be particular relevance in searching regional databases for certain topic areas, for example, searching Chinese databases may identify additional trials in searches on Chinese traditional medicine (15). Examples of regional and national bibliographic databases are listed in the Appendix. (Further information will also be available in the Technical Supplement associated with the Cochrane Handbook).

Bethel and Rogers recommend using search summary tables to evaluate the impact of decisions regarding bibliographic database selection (5). Search summary tables make it easier to identify the resources that contribute unique records and may aid decision-making for update searches.

Clinical trial registers, ongoing studies and unpublished data

Sources of ongoing and unpublished studies should be included in the search. Searching for such studies can prove challenging, but should be undertaken to minimise bias (16). Ongoing studies can be found in trials registries, whose records are updated when the trials are completed and published. 

The following is a selected list of trial registries and a search engine:

  • Clinicaltrials.gov
  • The World Health Organization International Clinical Trials Registry Platform Search Portal (WHO ICTRP)
  • The EU Clinical Trials Register (EUCTR)

Some clinical trials registries are region or topic specific (for a more exhaustive list see Chapter 4 of the 2019 Cochrane Handbook (3) or the YHEC clinical trials website).

Investigators’ documentation and clinical study reports are another source of unpublished data so contact with principle investigators or researchers may be tried. Isojärvi et al. have summarised research evidence around identifying unpublished data and highlight the need to search multiple trials registers (17).

Supplementary searching

Consider which supplementary search methods to use; sources will depend on the topic of the search (2019 Cochrane Handbook section 4.3.5). These methods may bring up publications that were not found in the original search or identify other concepts that have been missed. They may also identify comments, errata, retractions or related studies (18,19).

There is evidence that suggests hand searching may identify unique studies not identified through bibliographic database searches, particularly in non-English language journals (20). However, the value of hand searching may vary across subject areas (21).

Citation index searching can help identify studies particularly where subject searches are challenging (22, 23), and the 2019 Cochrane Handbook recommends screening previous reviews on the same topic to identify studies (3). For further information on citation searching, see the SuRe Info section on the Value of using different search approaches.

Web-searching may identify individual studies or organisations such as academic units or research bodies that publish relevant material. The usefulness of this is also topic dependent and further information will be available in the Technical Supplement associated with the Cochrane Handbook.

Searching for grey literature (or literature not controlled by traditional commercial publishing including reports, dissertations, theses, databases of conference abstracts) is considered highly desirable for Cochrane Reviews (2019 Cochrane Handbook section 4.3.5). Sources of grey literature are listed in the Appendix. The current Cochrane Handbook states that searching for grey literature is highly desirable (3), however, there is discussion regarding the value of grey literature for certain topics, so the time and effort spent searching grey literature sources should be carefully considered when planning the search (22,24).

Regulatory agency, manufacturer websites and clinical study reports may be useful for providing more extensive detail on interventions than is available in journal articles (17). Regulatory agencies provide access to detailed pharmaceutical submission documents such as the EMA’s public assessment reports (EPARs), FDA drug approval package or DAPs, as well as product recalls, market withdrawals or safety alerts. 

Health technology assessments (HTAs) published by national health technology assessment agencies can provide detailed information on the clinical effectiveness, economic analysis and patient related issues around new health technologies.

Designing search strategies

The 2019 Cochrane Handbook (section 4.4) provides guidance on issues to consider when designing search strategies for systematic reviews (3). The PICO model (Patient or Population or Problem; Intervention; Comparison; Outcome) is commonly used to develop the structure of a search strategy for clinical effectiveness research questions, but other models are available (25).

In many bibliographic database strategies, the search is likely to have 3 sets of terms, using a combination of subject headings and free-text terms to describe:

  • the population or health problem of interest,
  • the intervention(s) evaluated,
  • eligible types of study design (for e.g. randomized controlled trials).

For further information, see the SuRe Info chapter on Search strategy development.

Search filters

Search filters are combinations of search terms designed to retrieve particular types of references, including specific methodological study designs. The 2019 Cochrane Handbook (section 4.4.7) recommends using specially designed and tested filters (such as the Cochrane Highly Sensitive Search Strategies for identifying randomized trials in Ovid MEDLINE) only when appropriate (3). A search filter should not be used in a pre-filtered database such as CENTRAL.

For further information on the sources of methodological filters (including filters to identify randomized controlled trials and observational studies), see the SuRe Info section on Search filters.

Documenting searches

For further information regarding documenting and reporting searches, refer to the SuRe Info chapter.

Reference list

  • (1) Systematic reviews. CRD’s guidance for undertaking reviews in health care. York: Centre for Reviews and Dissemination, University of York; 2009. [Further reference details] [Publication appraisal] [Free full text
  • (2)  EUnetHTA Joint Action 2, Work Package 8. HTA Core Model ® version 3.0 (Pdf); 2016. [Further reference details] [Publication appraisal] [Free full text]
  • (3) Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokranen F, Thomas J, Wieland LS.  Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from www.training.cochrane.org/handbook.[Further reference details] [Publication appraisal] [Free full text]
  • (4) European Network for Health Technology Assessment (EUnetHTA). Guideline. Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness. Version 1.2  European Network for Health Technology Assessment; 2017. [Further reference details] [Publication appraisal] [Free full text
  • (5) Bethel A, Rogers M. Choosing the right databases and the right search techniques. In: Systematic searching: practical ideas for improving results. London, UK: Facet Publishing; 2019. p. 73–94. [Further reference details] [Publication appraisal] [Free full text]
  • (6) Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complement Ther Med. 2004;12(4):228–32. [Further reference details] [Publication appraisal] [Free full text]
  • (7) Levay P, Raynor M, Tuvey D. The contributions of MEDLINE, other bibliographic databases and various search techniques to NICE Public Health Guidance. Evid Based Libr Inf Pract. 2015;10(1). [Further reference details] [Publication appraisal] [Free full text]
  • (8)  Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Med Res Methodol. 2016;16:127. [Further reference details] [Publication appraisal] [Free full text]
  • (9) Aagaard T, Lund H, Juhl C. Optimizing literature search in systematic reviews – are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders? BMC Med Res Methodol. 2016;16:161. [Further reference details] [Publication appraisal] [Free full text]
  • (10)  Beckles Z, Glover S, Ashe J, Stockton S, Boynton J, Lai R, Alderson P. Searching CINAHL did not add value to clinical questions posed in NICE guidelines. J Clin Epidemiol. 2013 Sep;66(9):1051–7. [Further reference details] [Publication appraisal] [Free full text]  
  • (11) Kwon Y, Powelson SE, Wong H, Ghali WA, Conly JM. An assessment of the efficacy of searching in biomedical databases beyond MEDLINE in identifying studies for a systematic review on ward closures as an infection control intervention to control outbreaks. Syst Rev. 2014 Nov 11;3(1):135. [Further reference details] [Publication appraisal] [Free full text]
  • (12) Manriquez JJ. Searching the LILACS database could improve systematic reviews in dermatology. Arch Dermatol. 2009;145(8):947–8. [Further reference details] [Publication appraisal] [Free full text]
  • (13) Atsawawaranunt K, Adams CE, Roberts S. Searching for randomised controlled trials and clinical controlled trials in Thai online bibliographical biomedical databases. Health Inf Libr J. 2011;28(1):68–76. [Further reference details] [Publication appraisal] [Free full text]
  • (14)  Bonfill X, Osorio D, Posso M, Solà I, Rada G, Torres A, et al. Identification of biomedical journals in Spain and Latin America. Health Inf Libr J. 2015 Dec 1;32(4):276–86. [Further reference details] [Publication appraisal] [Free full text]
  • (15)  Wu XY, Tang JL, Mao C, Yuan JQ, Qin Y, Chung VC. Systematic reviews and meta-analyses of traditional chinese medicine must search chinese databases to reduce language bias. Evid Based Complement Altern Med. 2013;2013:812179. [Further reference details] [Publication appraisal] [Free full text]
  • (16) Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A, McIlroy W, Oxman AD. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA. 1993 Jun 2;269(21):2749–53. [Further reference details] [Publication appraisal] [Free full text]
  • (17) Isojärvi J, Wood H, Lefebvre C, Glanville J. Challenges of identifying unpublished data from clinical trials: Getting the best out of clinical trials registers and other novel sources. Res Synth Methods. 2018 Dec;9(4):561–78. [Further reference details] [Publication appraisal] [Free full text]
  • (18) Allahbadia GN. Why correcting the literature with errata and retractions is good medical practice? J Obstet Gynaecol India. 2014;64(6):377–80. [Further reference details] [Publication appraisal] [Free full text]
  • (19) Hopewell S, Clarke MJ, Stewart L, Tierney J. Time to publication for results of clinical trials. Cochrane Database Syst Rev 2007 Apr 18;(2):MR000011. [Further reference details] [Publication appraisal] [Free full text]
  • (20) Nasser M, Al Hajeri A. Comparison of handsearching versus Embase searching of the Archives of Iranian Medicine to identify reports of randomized controlled trials. Arch Iran Med. 2006;9(3):192–5. [Further reference details] [Publication appraisal] [Free full text]
  • (21) Stovold E, Hansen S. Handsearching respiratory conference abstracts: a comparison with abstracts identified by an EMBASE search. In: Abstracts of the 19th Cochrane Colloquium; 2011 19-22 Oct; Madrid, Spain. John Wiley & Sons; 2011. [Further reference details] [Publication appraisal] [Free full text]
  • (22) Cooper C, Booth A, Britten N, Garside R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev. 2017 Nov 28;6(1):234. [Further reference details] [Publication appraisal] [Free full text]
  • (23) Janssens ACJW, Gwinn M. Novel citation-based search method for scientific literature: application to meta-analyses. BMC Med Res Methodol. 2015 Oct 13;15:84. [Further reference details] [Publication appraisal] [Free full text]
  • (24) Delaney A, Tamas PA. Searching for evidence or approval? A commentary on database search in systematic reviews and alternative information retrieval methodologies. Res Synth Methods. 2018 Mar;9(1):124–31. [Further reference details] [Publication appraisal] [Free full text]
  • (25) Booth A. Unpacking your literature search toolbox: on search styles and tactics. Health Info Libr J. 2008;25:313–7. [Further reference details] [Publication appraisal] [Free full text]
AttachmentSize
ClinicalEffectivenessChapter_Appendix.pdf138.08 KB