Appraisal of: "Janssens, A. C., Gwinn M. Novel citation-based search method for scientific literature: application to meta-analyses. BMC Medical Research Methodology 2015 15: 84."

Reviewer(s): 
Elke Hausner
Full Reference: 
Novel citation-based search method for scientific literature: application to meta-analyses.
Short description: 

In two independent studies, the authors aimed to reproduce the results of literature searches for sets of published meta-analyses (n = 10 and n = 42). For each meta-analysis, they extracted co-citations for the randomly selected ‘known’ articles from the Web of Science database, counted their frequencies and screened all articles with a score above a selection threshold. In the second study, the authors extended the method by retrieving direct citations for all selected articles.

In the first study, they retrieved 82 % of the studies included in the meta-analyses while screening only 11 % as many articles as were screened for the original publications. Articles that were missed were published in non-English languages, published before 1975, published very recently, or available only as conference abstracts. In the second study, they retrieved 79 % of included studies while screening half the original number of articles.

Citation searching appears to be an efficient and reasonably accurate method for finding articles similar to one or more articles of interest for meta-analysis and reviews.

Limitations stated by the author(s): 

Clearly, the method does not work when the topic of the meta-analysis is heterogeneous and the studies of interest are unlikely to have cited each other. The second study also included several meta-analyses with very small sample sizes, including one in which half of the studies were case reports that had few or no references [21], as well as a meta-analysis for which the ‘known’ studies were cited only four times in total [22]. The percentage of retrieved studies jumped to 89 % when these five metaanalyses were excluded.

The accuracy and efficiency of co-citation searching depends on characteristics of the underlying citation network. By design, our method misses the studies that the collective community of researchers apparently did not find worth citing. In our analysis, these included abstracts, articles in non-English languages, very old articles, and publications in semi-scientific journals, reports, websites, and theses. In addition, some newer and some very old articles were not cited often enough to rank high in our search.

Limitations stated by the reviewer(s): 
The authors performed 2 independent studies with slightly different methods (e.g. thresholds, use of “direct citations”). It is not clear which method the authors recommend for further use.
Study Type: 
Single study
Related Chapters: 
Value of using different search approaches
Tags:
  • Share this