Diagnostic accuracy

Last revised: 
2017-10-31

Introduction

HTA may include assessment of new diagnostic technologies or techniques. These can involve the identification and review of diagnostic test accuracy (DTA) studies designed to differentiate between individuals with and without a target condition (1). The Cochrane Collaboration has published an evidence-based guide to searching for DTA studies, which provides the basis for this summary (2).

DTA studies tend to be poorly reported and searching for them can be problematic due to this inadequate reporting and inconsistent terminology, the absence of appropriate indexing terms in some databases for this publication type, and inconsistent use of suitable indexing terms where they are available (2).

Sources to search

Relying only on searching MEDLINE is not recommended, as it is unlikely to be the most comprehensive source of diagnostic information and because diagnostic studies are not easy to retrieve efficiently in bibliographic databases (3). Relative recall analysis of systematic reviews has also suggested other databases might yield additional studies including Science Citation Index, BIOSIS and LILACS (3). Recent analyses have suggested that fewer databases might be adequate, but are weakened by their reliance on known-item searches (4,5,6). Review searches may not detect all the records in MEDLINE that might be relevant to a review, so searching other databases provides opportunities to pick up (MEDLINE indexed) studies by other routes. An analysis of ten meta-analyses found that only using studies indexed in MEDLINE did not impact significantly on the sensitivity and specificity estimates of the meta-analyses in those reviews (4). A second analysis of 16 meta-analyses of diagnostic accuracy studies of depression screening tools found 94% (range: 83-100%) of the primary studies included in the meta-analyses were indexed in MEDLINE (5). The remaining non-MEDLINE indexed studies were located in Scopus, PsycINFO, and Embase. The authors acknowledged that the quality of the majority of the original reviews could not be determined. Another recent study of nine reviews performed by a single research group found that the reviewers’ original searches would have found 85% of their included studies from MEDLINE and Embase (range: 60-100%) (6). Adding reference checking to the process would have found 93% of the included studies. The Cochrane Handbook (2), based on available research-evidence, currently recommends that searches should include the following databases for reviews and primary studies:

●      MEDLINE
●      Embase
●      ARIF
●      HTA database
●      DARE (closed to new records from March 2015)
●      Cochrane Database of Systematic Reviews
●      Searches for unpublished studies in dissertations databases and grey literature databases
●      Reference checking
●      Citation searches in services such as Science Citation Index, Scopus, and Google Scholar, as well as related  articles options in interfaces such as PubMed or Ovid.

In addition the following new databases could also be searched:

●     PROSPERO - register of systematic reviews
●     Epistemonikos - collection of systematic reviews and their included studies
●     PDQ Evidence - collection of systematic reviews about health systems and their included studies 

HTA agencies may also undertake assessments of diagnostic tests and so agency websites should also be explored, for example NICE diagnostic test guidance can be accessed here.

Although the proportion of ongoing studies investigating diagnostic test accuracy may still be relatively low (7), some are being recorded prospectively on trials registers such as ClinicalTrials.gov and the ICTRP portal.

The evidence for the value of handsearching is currently sparse, with one recent study of one topic showing that handsearching contributed little (8). It is possible that the topic of the research was well defined and the database searches were exemplary, and therefore the handsearching contribution would be different in other topics (8). More evidence is required on the yield and value of handsearching. Where a topic is published in journals that are not indexed in bibliographic databases, handsearching can still serve a purpose but this needs to be evaluated question by question.

Designing search strategies

Search strategies should be designed to be highly sensitive using a wide variety of search terms, both text words and subject indexing, to ensure that the many different ways that a test may be described feature in the search (2).  Information specialists should be aware of the weaknesses of reporting in abstracts of diagnostic accuracy studies. One exploratory study evaluating the comprehensiveness of reporting in the abstracts of 12 high-impact journals found 50% of the articles did not identify the study as a diagnostic accuracy study in the title and 65% included the sensitivity and/or specificity estimates in the abstract (9).

The search should reflect some, but not necessarily all, of the key concepts of the review (2). The search is likely to capture the index test being investigated and the target condition being diagnosed (2,9). A third set of terms can be considered to capture the patient description or the reference standard. The development of search strategies for DTA studies can be challenging and may involve several iterations to reach a strategy that captures the complex way records may present concepts of diagnosis (2). Cochrane Reviews of diagnostic test accuracy studies and the Cochrane handbook provide examples of search approaches for these, often complex, topics. Strategies may include both general terms (such as the generic type of diagnostic method, for example dipsticks) and specific terms such as named dipstick tests (2).

There are many published methodological search filters designed to capture studies of diagnostic test accuracy and including test measurement terms such as sensitivity and accuracy (10). The evidence, however, on the performance of DTA search filters suggests that combining filters with a search for a population and an index test is likely to miss relevant studies (11,12,13,14). Search filters for DTA studies do not seem to perform consistently and may result in unacceptable reductions in sensitivity (10, 11,12,13,14). Some studies have found that there may be instances where these methodological filters could be used, but these are not within the context of information retrieval for the production of health technology assessments (15,16).  When all the research is considered together, current evidence suggests that for search strategies designed to support systematic reviews of diagnostic accuracy, as long as DTA filters are not the only approach, they may be useful as one component of a search strategy which involves several search approaches:  a “multi-stranded” approach involves multiple queries run sequentially and using different combinations of concepts. Search filters can be identified from the InterTASC Information Specialists' Sub-Group (ISSG) Search Filter Resource (10).

Subheadings (floating subheadings and subheadings attached to the index test or the target condition) may be a helpful component of the search strategy (2).

Developing a search strategy can be iterative and complex and it can be helpful to have topic experts to review samples of search results for relevance and it is always helpful to be able to test retrieval against sets of known relevant records.

We have used the searching chapter of the Cochrane Handbook (2) as our baseline and SuRe Info appraisals have only been prepared for recently identified studies.

Reference list

  • (1) EUnetHTAWork Package 4. HTA Core Model® for Diagnostic Technologies v. 1.0r; 2008. [Further reference details] [Publication appraisal] [ Free Full text ]
  • (2) de Vet H C W, Eisinga A, Riphagen I I, Aertgeerts B, Pewsner D. Searching for studies. In Cochrane handbook for systematic reviews of diagnostic test accuracy. Version 0.4 ed. The Cochrane Collaboration, 2008. [Further reference details] [Publication appraisal] [Free Full text]
  • (3) Whiting P, Westwood M, Burke M, Sterne J, Glanville J. Systematic reviews of test accuracy should search a range of databases to identify primary studies. J Clin Epidemiol 2008;61(4): 357.e1-357.e10. [Further reference details ] [Publication appraisal] [Free Full text]
  • (4) van Enst WA, Scholten RJ, Whiting P, Zwinderman AH, Hooft L. Meta-epidemiologic analysis indicates that MEDLINE searches are sufficient for diagnostic test accuracy systematic reviews. J Clin Epidemiol 2014;67(11): 1192-1199. [Further reference details] [Publication appraisal] [Free Full text]
  • (5) Rice DB, Kloda LA, Levis B, Qi B, Kingsland E, Thombs BD. Are MEDLINE searches sufficient for systematic reviews and meta-analyses of the diagnostic accuracy of depression screening tools? A review of meta-analyses. J Psychosom Res. 2016 Aug;87:7-13. [Further reference details] [Publication appraisal] [Free Full text]
  • (6) Preston L, Carroll C, Gardois P, Paisley S, Kaltenthaler E. Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking. Syst Rev. 2015;4: 82. [Further reference details] [Publication appraisal] [Free full text]
  • (7) Korevaar DA, Bossuyt PMM, Hooft L. Infrequent and incomplete registration of test accuracy studies: analysis of recent study reports. BMJ Open. 2014;4: e004596. [Further reference details] [Publication appraisal[Free full text]
  • (8) Glanville J, Cikalo M, Crawford F, Dozier M, McIntosh H. Handsearching did not yield additional unique FDG-PET diagnostic test accuracy studies compared with electronic searches: a preliminary investigation. Res Synth Methods 2012;3(3): 202-213. [Further reference details] [Publication appraisal] [Free Full text]
  • (9) Korevaar DA, Cohen JF, Hooft L, Bossuyt PMM. (2015). Literature survey of high-impact journals revealed reporting weaknesses in abstracts of diagnostic accuracy studies. J Clin Epidemiol. 68(6): 708-715. [Further reference details] [Publication appraisal] [Free Full text]
  • (10) ISSG Search Filter Resource [web page]. York: Centre for Reviews and Dissemination and InterTASC Information Specialists SubGroup; 2013. [Further reference details] [Publication appraisal] [Free Full text ]
  • (11) Leeflang M M G, Scholten R J, Rutjes A W S, Reitsma J B, Bossuyt P M. Use of methodological search filters to identify diagnostic accuracy studies can lead to the omission of relevant studies. J Clin Epidemiol 2006;59(3): 234-240. [Further reference details] [Publication appraisal] [Free Full text]
  • (12) Ritchie G, Glanville J, Lefebvre C. Do published search filters to identify diagnostic test accuracy studies perform adequately? Health Info Libr J 2007;24(3):188–192. [Further reference details] [Publication appraisal] [Free Full text ]
  • (13) Whiting P, Westwood M, Beynon R, Burke M, Sterne J A, Glanville J. Inclusion of methodological filters in searches for diagnostic test accuracy studies misses relevant studies. J Clin Epidemiol 2011;64(6): 602-607. [Further reference details] [Publication appraisal] [Free Full text]
  • (14) Beynon R, Leeflang MM, McDonald S, Eisinga A, Mitchell RL, Whiting P, Glanville JM. Search strategies to identify diagnostic accuracy studies in MEDLINE and EMBASE. Cochrane Database of Systematic Reviews. 2013, Issue 9. Art. No.: MR000022. DOI: 10.1002/14651858.MR000022.pub3. [Further reference details[Publication appraisal] [Free full text]
  • (15) Rogerson TE, Ladhani M, Mitchell R, Craig JC, Webster AC. Efficient strategies to find diagnostic test accuracy studies in kidney journals. Nephrology (Carlton). 2015 Aug;20(8):513-8. [Further reference details] [Publication appraisal] [Free Full text]
  • (16) Huang Y, Yang Z, Wang J, Zhuo L, Li Z, Zhan S. (2016). Performance of search strategies to retrieve systematic reviews of diagnostic test accuracy from the Cochrane Library. J Evid Based Med. 9: 77-83 [Further reference details] [Publication appraisal] [Free Full text]