Diagnostic accuracy

Appraisal of: "Yao X, Vella E, Brouwers M. How to conduct a high-quality systematic review on diagnostic research topics. Surg Oncol. 2018 Mar;27(1):70-75."

Reviewer(s): 
Short description: 

Authors suggest six steps are necessary for conducting high-quality systematic reviews on diagnostic topics around the objective, research question, searching for literature, assessing risk of bias, analyzing, and interpreting data. The review aims to instruct researchers on diagnostic methodology and recent updates in the field.

Rather than designing a study around the elements of PICO, PIRO is recommended in which “I” represents index test, and “R” represents reference standard.

The literature search for diagnostic topics can begin with using an existing systematic review that answers the research question, and is of satisfactory quality. The search date of that study can be used as a starting point to search forward in MEDLINE, Embase, Cochrane Database of Systematic Reviews, and PROSPERO. A grey literature search is also recommended.

When no appropriate systematic reviews are available, a search of primary studies will need to be conducted. Databases to be used in this case include MEDLINE, Embase, and Cochrane Library databases, as well as related conference proceedings. It is preferred that language limits are not used and that a grey literature search is carried out if deemed appropriate by researchers. The search should include Medical Subject Headings terms and keywords representing the diagnostic test and target condition. Adding terms to capture diagnostic accuracy or study design is not recommended, as indexing of this concept has not been proven to be dependable.

This is a second article in a series. First article: Yao X, Vella E. How to conduct a high-quality original study on a diagnostic research topic. Surg Oncol. 2017;26(3):305-9.

 

Limitations stated by the author(s): 

Authors noted that in order to keep the article brief, they did not fully explore the concepts of how to design strategies for different databases or peer review.

Limitations stated by the reviewer(s): 
Authors made general observations about the quality of diagnostic studies needing improvement without expanding on any evidence to support the claim.
Study Type: 
Review

Appraisal of: “Korevaar DA, Hooft L, Askie LM, Barbour V, Faure H, Gatsonis CA, et al. Facilitating Prospective Registration of Diagnostic Accuracy Studies: A STARD Initiative. Clin Chem. 2017;63: 1331–1341.”

Short description: 

This paper provides the arguments for prospective registration of diagnostic test accuracy (DTA) studies in public databases. The authors do not advocate the creation of a specific registry for DTA studies, but registration is recommended in any available known public registry.  A list of possible registries is provided in Table 1 of the paper.

The authors propose a modified version of the WHO trial registration data set (Table 2 in the paper) which specifies the fields that would be needed for registry records to capture the key elements of a DTA study. 

Using their modified version of the WHO trial registration data set, the authors evaluated the reported information for the 30 most recently registered DTA studies in the WHO ICTRP in November 2016.  This small analysis provides some information for searchers who are searching for DTA studies in ICTRP.   In the sample of thirty studies, 90% reported that the study was a DTA study in the title of the record, 97% reported the target condition and 100% reported the index test.

The authors call on trial registries to ensure informative data are included in registry records to enhance identification.  They also call on journal editors to encourage DTA study registration prior to publication.

Limitations stated by the author(s): 

This is a position paper and it does not dwell upon limitations of its methods.

Limitations stated by the reviewer(s): 
The information for searchers is largely from the analysis of a small sample of thirty recent study records in ICTRP, but this information is highly valuable in indicating which aspects of DTA studies might be identifiable and retrievable within that database.

Appraisal of: "Rice DB, Kloda LA, Levis B, Qi B, Kingsland E, Thombs BD. Are MEDLINE searches sufficient for systematic reviews and meta-analyses of the diagnostic accuracy of depression screening tools? A review of meta-analyses. J Psychosom Res. 2016 Au

Short description: 

This study builds on the findings of Enst, et al (2014), testing to see if limiting literature searches to only MEDLINE is sufficient when conducting a systematic search for diagnostic accuracy studies of depression screening tools.

Authors used a peer-reviewed strategy to search MEDLINE and PsycINFO for meta-analyses of diagnostic accuracy of depression screening tools, screened titles and abstracts for inclusion, which resulted in the identification of 16 meta-analyses to inform this review. Authors then conducted a known item search in MEDLINE to determine if primary studies used in the meta-analyses were indexed in that database. For those studies not found in MEDLINE, authors searched a core set of databases and Google Scholar to identify where studies were indexed.

94% (375/398) (range: 83-100%) of the primary studies included in the 16 meta-analyses were indexed in MEDLINE. Of the 23 primary studies not indexed in MEDLINE, 71% were located in Scopus, 64% in PsycINFO, and/or 50% in Embase.

  

Limitations stated by the author(s): 
  • Authors did not re-analyze meta-analyses to determine if results changed when non-MEDLINE indexed studies were removed from the analyses;
  • The search used to identify sample was not updated from the original 2014 search date, therefore there is the potential for missed studies in sample;
  • Study design retrospectively reviewed existing meta-analyses, which authors acknowledge is not equivalent as conducting a ‘live’ systematic review and meta-analysis; 
  • Unable to confirm whether the search strategies used in each meta-analysis did identify the articles indexed in MEDLINE as most included meta-analyses did not provide reproducible search strategies. Known-item search may have resulted in an overestimation of studies found in MEDLINE.
 
Limitations stated by the reviewer(s): 
Authors did not provide a rationale for limiting their search to MEDLINE and PsycINFO; It is unclear why the authors decided to exclude systematic reviews that did not meta-analyze since excluding these studies likely resulted in a small sample and authors did not re-analyze the meta-analyses with non-MEDLINE studies removed; The authors raise the concern of MEDLINE not being updated as frequently having implications if groups were to adopt this approach for conducting systematic reviews and meta-analyses of diagnostic test accuracy studies, but did not highlight the importance of updating searches or setting up database search alerts.
Study Type: 
Single study

Appraisal of: "Korevaar DA, Cohen JF, Hooft L, Bossuyt PMM. (2015). Literature survey of high-impact journals revealed reporting weaknesses in abstracts of diagnostic accuracy studies. J Clin Epidemiol. 68(6): 708-715."

Short description: 

This study presents the results of an evaluation to determine the “informativeness” (i.e., reporting of methods used, reference test/standard used, balanced reporting of results) of abstracts for diagnostic accuracy studies published in 12 high-impact journals in 2012.

Of interest to information retrieval practices, specifically the use of diagnostic accuracy filters, the study found that:

  • 50% (51/103) of articles in the sample identified the study as a diagnostic accuracy study in the title;
  • 100% (103/103) of article abstracts identified the index test;
  • Of the 93% (96/103) of abstracts that reported a diagnostic accuracy estimate, 65% (67/103) were sensitivity and/or specificity estimates; 19% (20/103) reported the negative and/or positive predictive value. 
Limitations stated by the author(s): 
  • Sample of studies was limited to high-impact journals and may have produced an overestimate of the number of items reported in study abstracts. The findings may not be generalisable to lower-impact journals.
  • This is an exploratory analysis: the authors did not calculate the sample size required to power this study, therefore results may not be generalizable.
Limitations stated by the reviewer(s): 
In addition to the self-identified limitations of the sample, this study used a diagnostic filter published in 2000 (Deville et al.) to create the sample. Research, including this study, has consistently found weak reporting of study design and results of diagnostic accuracy studies, therefore using a diagnostic filter to create the sample may have resulted in missing relevant studies.
Study Type: 
Single study
Related Chapters: 

Appraisal of: "Rogerson TE, Ladhani M, Mitchell R, Craig JC, Webster AC. Efficient strategies to find diagnostic test accuracy studies in kidney journals. Nephrology (Carlton). 2015 Aug;20(8):513-8. "

Short description: 

The aim of this study was to assess the performance of published diagnostic accuracy search filters for retrieval of diagnostic accuracy studies in nephrology journals. Note: the authors do emphasize that the study was designed to assist practitioners in routine clinical practice, therefore the findings are not intended to support information retrieval for the production of systematic reviews, health technology assessments.

To create the reference set, two reviewers hand searched the 2002, 2003, 2009 and 2010 issues of three nephrology journals. These years were selected as a way to compare the performance of search filters before and after the release of the 2003 STAndards for Reporting Diagnostic accuracy studies (STARD) statements—a secondary objective of this study.  The authors assessed the performance of 14 published diagnostic test accuracy filters in MEDLINE; limiting searches to the years and journals that created the reference set.

The reference set consisted of 103 diagnostic test accuracy studies. Of the 14 published filters assessed, the top performers in the following domains were:

  • Sensitivity: van der Weijden 1997 Extended search strategy overall (0.95; 95% CI 0.89-0.98)
  • Specificity: Haynes 2004 Narrow search strategy 0.99 (95% CI 0.99-0.99)
  • Best balance between sensitivity and specificity: Bachmann 2002 and van der Weijden 1997 Short strategy.
Limitations stated by the author(s): 
  •  A small reference set was used to test filter performance, thus this study lacked the statistical power to detect increases in filter performances between 2002-2003 and 2009-2010 (stated secondary objective).
  • The authors acknowledged that restricting to three general nephrology journals may have impacted the reference set. Nephrology-related research is published in a wide range of journals, including the following disciplines: transplantation, urology, general medicine journals. Therefore, the findings of this study may not be generalizable to diagnostic filter performance in other journals.
  • Within the reference set, the authors acknowledged an underrepresentation of transplant patients; an important patient group within nephrology.
Limitations stated by the reviewer(s): 
The authors failed to provide a rationale for the selection of the three journals used to create the reference set.
Study Type: 
Single study
Related Chapters: 

Appraisal of: "Huang Y, Yang Z, Wang J, Zhuo L, Li Z, Zhan S. (2016). Performance of search strategies to retrieve systematic reviews of diagnostic test accuracy from the Cochrane Library. J Evid Based Med. 9: 77-83. Erratum in: J Evid Based Med 2017;10"

Short description: 

The objective of this study was to assess the performance of nine diagnostic accuracy filters in retrieving systematic reviews (SRs) of diagnostic accuracy in Cochrane Library databases, specifically Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE).

Study authors adapted SIGN’s diagnostic studies filter (http://www.sign.ac.uk/search-filters.html), adding concepts of “accuracy measures” and “diagnosis” to create nine strategies. The reference set used to test filter performance in both CDSR and DARE was created using a strategy that combined all the terms used in the filters. Study authors also identified additional diagnostic accuracy studies to include by reviewing each record in CDSR under the topic of diagnosis.

 Overall, filters containing the concept of “accuracy measures” achieved similar sensitivities as those containing “diagnosis” but demonstrated higher precision. Filters containing both concepts demonstrated the highest sensitivity.    

Limitations stated by the author(s): 
  • The use of a highly sensitive strategy, which combined all terms being tested, to create the reference set likely overestimated the performance of other strategies and may have missed some relevant articles which did not contain the terms used in filters.
  • The focus of the study was the retrieval of systematic reviews of diagnostic test accuracy and not retrieval of primary studies. Authors acknowledged methodological filters are primarily designed to retrieve original studies.
Limitations stated by the reviewer(s): 
Although the paper was published in 2016, the authors do not share with readers that although DARE is archived and can still be searched, it is no longer being updated (searches for systematic reviews ceased at the end of 2014; structured abstracts were published on DARE until March 31, 2015); The authors acknowledged that using diagnostic accuracy filters designed to retrieve primary studies is a limitation when compiling the set of relevant reviews. This means that this study’s findings are not necessarily generalizable to filter performances for the retrieval of primary diagnostic accuracy studies.
Study Type: 
Single study
Related Chapters: 
Supplemental publications to the study: 

Appraisal of:"Korevaar DA, Bossuyt PM, Hooft L. Infrequent and incomplete registration of test accuracy studies: analysis of recent study reports. BMJ Open. 2014 Jan 31;4(1):e004596. doi: 10.1136/bmjopen-2013-004596"

Short description: 
The aim of this study was to identify the proportion of recently published test accuracy studies which had been registered  to evaluate whether registration had preceded study initiation and to assess whether registration included the published primary outcomes. The authors searched PubMed for primary studies evaluating the accuracy of tests or markers for screening, diagnosis, staging, monitoring, prediction or prognosis against a reference standard  published in 536 high impact (>5) journals. 
 
The authors identified 351 articles for inclusion of which 248 (71%) were diagnostic test accuracy studies and 103 (29%) were studies assessing prognostic tests. The authors identified 52 (15%) registered studies 27 of which included the registration number in the publication and 12 of which provided a reference to the publication in the registry.   Of the studies that had been registered, 33% reported industry involvement; significantly more often than studies reporting another source of funding (11%), and studies without a (reported) source of funding (9%; p<0.001). Furthermore, 16/52 (31%) studies (5% of the total sample) had published primary outcomes before the study’s completion while among a further 12 (23%) the published primary aim had been registered but was somewhat different or vague than was reported in the published results. The authors reported that overall, very few test accuracy studies published in high impact journals are registered and only one in 22 test accuracy studies register their primary outcomes before study completion. These findings raise questions regarding transparency and the potential for selective reporting within published test accuracy studies.
 
Limitations stated by the author(s): 

There were a number of limitations discussed by the study’s authors, including the unknown registration status of 126/351 (36%) of the included studies due to author non-reply to email inquiries. The restriction of the search to journals with impact factor of ≥5 could have introduced bias as it is possible that studies published in high impact journals are more likely to register studies and this could have overestimated results. Though the search limited to studies published between May and June 2012, the sample may have included older studies conducted before the trend toward registering study protocols. 

Limitations stated by the reviewer(s): 
Authors limited search to published articles and did not search for unpublished reports.
Study Type: 
Single study
Related Chapters: 

Appraisal of: "Preston L, Carroll C, Gardois P, Paisley S, Kaltenthaler E. Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking."

Short description: 

This exploratory study analysed a convenience sample of nine Health Technology Assessments (HTA) reviews of diagnostic test accuracy (DTA) to identify which of the included citations were indexed on MEDLINE or EMBASE; to identify the number and proportion of citations that were retrieved by the search strategies with or without the addition of checking the reference lists of included studies.                     

Of the 302 included citations included in the nine reviews, 287 (95%) were indexed in MEDLINE and EMBASE. The reviews searches accounted for 256 (85%) of the included citations and a further 24 (8%) could be identified from the reference lists of included citations. 7% of the citations (22/302) were not found by searching or reference checking.

The proportion of citations identified using both searches of MEDLINE, and EMBASE and checking the reference lists resulted in the identification of 280/302 (93%) included studies. The authors suggest that there might be a case for restricting systematic review searches  of DTA studies to MEDLINE, EMBASE and the reference lists of included studies. 

Limitations stated by the author(s): 

The study’s pragmatic, convenience sample of a small number of published DTA reviews was intended as an exploratory analysis only. The  authors did not conduct an analysis to assess the impact of excluding the unidentified studies on the conclusions of the HTAs, so the results of the study are uncertain. 

Limitations stated by the reviewer(s): 
Six of the nine studies in the sample used a published filter for diagnostic studies, which is not recommended by Cochrane. This may have impacted the sensitivity of the searches and the sample not be representative of other DTA reviews.
Study Type: 
Single study
Related Chapters: 

Appraisal of: van Enst WA, Scholten RJ, Whiting P, Zwinderman AH, Hooft L. Meta-epidemiologic analysis indicates that MEDLINE searches are sufficient for diagnostic test accuracy systematic reviews. J Clin Epidemiol 2014;67(11): 1192-1199.

Short description: 

This paper describes an investigation into the impact of excluding studies which could not be retrieved from MDELINE on the summary relative diagnostic odds ratio of a diagnostic test accuracy systematic review.   The authors identified ten reviews with 15 meta-analyses which also included searches of MEDLINE and at least one other database.  The statistical analysis found that when removing studies available from databases other than MEDLINE that the summary estimates of sensitivity and specificity in the reviews remained almost unchanged.  The authors conclude that restricting to studies indexed in MEDLINE did not influence the summary estimates of the meta-analyses in the 10 reviews and it might be acceptable where resources are limited to restrict searches to MEDLINE.

Limitations stated by the author(s): 

The authors noted that a more restrictive search might miss relevant studies and decrease the power of the meta-analysis. The authors also acknowledge that searching additional databases may yield records which, while available in MEDLINE, were not retrieved by specific searches. The number of included meta-analyses was small and those that were selected were from higher impact journals.  The authors would have liked to have investigated search strategies in more detail but these were not reported in full in many reviews.  They were unable to rerun the search strategies in the reviews and had to run known-item searches which have the disadvantage that we do not know how the original searches performed.

Limitations stated by the reviewer(s): 
The authors do describe the limitations of known-item searching but do not mention this until the discussion.
Study Type: 
Single study
Related Chapters: 
Syndicate content