Appraisal of: "Korevaar DA, Salameh JP, Vali Y, Cohen JF, McInnes MDF, Spijker R, Bossuyt PM. Searching practices and inclusion of unpublished studies in systematic reviews of diagnostic accuracy. Res Synth Methods. 2020 Jan 25."

Reviewer(s): 
Short description: 

There have been numerous studies reporting that anywhere from 25%-50% of diagnostic accuracy studies do not reach full-text publication in peer-reviewed publications, which may negatively impact systematic reviews of diagnostic accuracy (DTA-SR) by introducing biases if sources searched do not include unpublished studies (e.g., conference proceedings and trial registry data).

The aim of this study was to better understand the information retrieval practices among recently published DTA-SRs, specially the practice of searching for unpublished studies. Authors created two sets of DTA-SRs for this analysis: Cochrane DTA-SRs (n=100) and non-Cochrane DTA-SRs (n=100) and reviewed the publications to identify whether efforts were made to identify unpublished studies and whether unpublished studies were included.

Results indicate suboptimal searching practices in both sets:

  • Cochrane DTA-SRs set:
    • 42 reported they considered or searched for unpublished studies for inclusion (10 reported they excluded non-peer-reviewed literature)
    • 48 did not transparently report whether they did/did not consider unpublished studies, although 35 of these reported searching in sources of unpublished studies and 9 included at least one unpublished study in the analyses
    • Overall, of the 2,780 included primary studies in this set, unpublished studies represented 2.3% (64/2780)
  • Non-Cochrane DTA-SR set:
    • 10 reported they considered or searched for unpublished studies for inclusion (36 reported they excluded non-peer-reviewed literature)
    • 23 studies referred to unpublished or grey literature in general
    • 54 did not transparently report whether they did/did not consider unpublished studies, although 13 of these reported searching in sources of unpublished studies and 10 included at least one unpublished study in the analyses
    • Overall, of the 2,082 included primary studies in this set, unpublished studies represented 1.9% (39/2082)
Limitations stated by the author(s): 

The authors acknowledge that this study relies on the reporting of methods of the included DTA-SRs, which research has demonstrated often fail to include necessary information. They also acknowledge that there are different understandings and definitions of “unpublished studies”. To illustrate, they did find DTA-SRs which reported excluding unpublished studies, but in fact included conference abstracts. Finally, the included DTA-SR sets were created using convenience sampling and timeframes differed, which, as mentioned above, prevented the authors from conducting a statistical comparison between the Cochrane and non-Cochrane sets.

Limitations stated by the reviewer(s): 
It was unclear why the authors decided to use convenience sampling techniques to create the two DTA-SR sets rather than via random sampling.
Study Type: 
Single study