Web of Science

Appraisal of: Robinson KA, Dunn AG, Tsafnaf G, Glasziou P. Citation networks of related trials are often disconnected: implications for bidirectional citation searches. J Clin Epidemiol. 2014;67(7): 793-799.

Reviewer(s): 
Short description: 

The authors sought to determine the connectedness of citation networks of RCTs by examining direct (referenced trials) and indirect (through references of referenced trials, etc) citation of trials to one another. 259 meta-analyses including 2,413 RCTs were used to create citation networks of RCTs addressing the same clinical questions.

For 46% (118 of 259) of networks, the RCTs formed a single connected citation group - one island. For the other 54% of networks, where at least one RCT group was not cited by others, 39% had two citation islands and 4% (10 of 257) had 10 or more islands.

Conclusion: Available evidence to answer a clinical question may be identified by using network citations created with a small initial corpus of eligible trials. However, the number of islands means that citation networks cannot be relied on for evidence retrieval.

Limitations stated by the author(s): 

The authors stated some limitations.

They used Web of Science to identify the Meta-Analysis and the RCTs included in those Meta-Analysis. Some networks are thus not complete as one or more of the RCTs from the MA are not included in their data set. Given the relatively small number of RCTs missing per network, it is not likely that these missing RCTs would substantively change the results.

Another limitation is that a trial could have cited systematic reviews rather than individual prior trials. They did not check this as other studies have suggested few trials cite up-to-date systematic reviews.

Limitations stated by the reviewer(s): 
The study is of high quality, one limitation might be, that only trials were included in the analysis. However, the results might be better when all references from the reference lists were analysed.
Study Type: 
Single study

Appraisal of: Belter CW. "A relevance ranking method for citation-based search results." Scientometrics 2017;112(2): 731-746.

Reviewer(s): 
Short description: 

This study tests a method which was modified from a previous study (Belter 2016). In this paper, the author proposed a method of ranking the relevance of citation-based search results to a set of key, or seed, papers by measuring the number of citation relationships they share with those key papers.

The method was tested against 23 published systematic reviews and it was found that the method retrieved 87% of the studies included in these reviews. Additional testing suggested that the method may be less appropriate for reviews that combine literature in ways that are not reflected in the literature itself. 

Limitations stated by the author(s): 

 

These results also suggests that the opposite is also true: that the method is less successful, and therefore less appropriate for literature retrieval, in reviews that combine topics in ways that the literature does not.
 
It may be that this method is not appropriate for literature retrieval in reviews that seek to combine the results of multiple interventions, multiple treatment populations, or multiple disorders. 
Limitations stated by the reviewer(s): 
No other limitations were detected by the reviewers.
Study Type: 
Single study

Appraisal of: “Belter CW. Citation analysis as a literature search method for systematic reviews. J Assn Inf Sci Tec 2016; 67(11): 2766-2777.”

Short description: 

The author proposed a method of searching the literature by systematically mining the various types of citation relationships between articles (cited by, citing, cociting, cocited articles).

The author tested the method by comparing its precision and recall to that of 14 published systematic reviews. The method successfully retrieved 74% of the studies included in these reviews and 90% of the studies it could reasonably be expected to retrieve. Indirect citation relationships clearly outperformed direct citation relationships in retrieving relevant publications. The method also retrieved fewer than half of the total number of publications retrieved by these reviews and can be performed in substantially less time. 

Limitations stated by the author(s): 

This analysis clearly demonstrated that certain kinds of studies can rarely be identified using citation relationships. This is due both to the citation behavior of authors and the limitations of citation databases. Because authors rarely cite conference proceedings, and even more rarely cite studies in progress, it is extremely difficult for the proposed method to identify studies published or made available in these formats. Further, because these publication formats rarely include indexed cited references, the method is even less likely to identify them. This limitation means that the proposed method is even more prone to publication bias than the traditional method, meaning that its ability to identify descriptions of negative results is minimal. In addition, studies published in journals not indexed by the Web of Science were rarely identified by the proposed method. This means that certain kinds of studies, especially studies published in non-Western journals like the traditional Chinese medicine review excluded from this analysis, will rarely be identified by this method. Although using Scopus instead of, or in addition to, the Web of Science may be able to identify some of these publications, there are still a large number of journals not indexed by either database. The proposed method would be unlikely to retrieve studies published in such journals. Because of these limitations, it is unlikely that the proposed method could ever replace text-based searching for systematic reviews.

Although the method as implemented here performed relatively well, I made several arbitrary choices in implementing it in this analysis. My decision to use two key studies on which to base the method was one of these arbitrary choices.

Other arbitrary choices in this method were the thresholds I used to filter the cociting and cocited articles.

Limitations stated by the reviewer(s): 
No additional limitations were detected by the reviewers.
Study Type: 
Single study

Appraisal of: Wright K, Golder S, Rodriguez-Lopez R. Citation searching: a systematic review case study of multiple risk behaviour interventions. BMC Med Res Methodol. 2014; 14:73.

Short description: 

Study about the effectiveness of using the citation sources Google Scholar, Scopus, Web of Science and OVIDSP MEDLINE to identify records for inclusion in a systematic review about multiple risk behaviour interventions.

40 included studies were used identified by traditional database searches from one systematic review. Each of the included studies was searched for in the four citation sources to retrieve the details of all papers that have cited these studies.

The highest number of citations was retrieved from Google Scholar (1680), followed by Scopus (1173), then Web of Science (1095) and lastly OVIDSP (213). To retrieve all the records identified by the citation tracking searching all four resources was required. Google Scholar identified the highest number of unique citations. The citation tracking identified 9 studies that met the review’s inclusion criteria (recall 22,5%).

Conclusion: Citation searching as a supplementary search method for systematic reviews may not be the best use of valuable time and resources.

Limitations stated by the author(s): 

In terms of searching, the scoping review had very broad coverage. Its aim was to identify any intervention promoting change in at least two risk behaviours and the search strategy incorporated terms for all of these (smoking, alcohol misuse, physical inactivity, unhealthy diet, illicit drug use, sexual risk behaviour, lack of seat belt use, lack of motorcycle/bicycle helmet use, lack of sunscreen use, gambling, poor oral hygiene and drink driving) in various set combinations. The resulting complexity will almost certainly have had an impact upon the overall performance of the database search strategies. As with any case study, there is uncertainty about how far the results of this study can be generalised, especially to other reviews with a more restrictive focus.

Limitations stated by the reviewer(s): 
No additional limitations were detected by the reviewers.
Study Type: 
Single study

Appraisal of: Lowe J, Peters J, Shields B, Cooper C. Methods to update systematic literature searches: full update searching vs. forward citation chasing; a case study from a systematic review of diagnostic accuracy [poster]. University of Exeter Medical

Short description: 

The aim of the study was to compare 2 methods used to update the literature searches. 1) A standard, full-update search using the Systematic Review’s (SR) original search strategy. 2) Using Web of Science (Thomson Reuters), forwards citation searching was conducted on the 10 studies included at full text in the SR.

In terms of time, the citation chasing arm was considerably more efficient. In terms of effectiveness, the full-update search produced a greater number of includable studies for full-text screening (full update search n=9, citation searching n=1). There was no cross-over between the arms of this case-study: each arm offered us unique studies, so no one arm could have been independently used.

Conclusion: full-update searching, including standard supplementary search methods, are advisable when searching for studies reporting diagnostic tests.

Limitations stated by the author(s): 

No limitations were stated by the authors.

Limitations stated by the reviewer(s): 
Abstract, no full text publication available; articles used for reference set “identified for full text screening” (not included references).
Study Type: 
Single study

Appraisal of: Linder SK, Kamath GR, Pratt GF, Saraykar SS, Volk RJ. Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments. J Clin Epidemiol. 2015; 68(4):412-417.

Short description: 

The aim of the study was to compare the effectiveness of two search methods in identifying studies that used the Control Preferences Scale (CPS), a health care decision-making instrument.

The literature was searched using two methods: (1) keyword searching using variations of ‘‘Control Preferences Scale’’ and (2) cited reference searching using two seminal CPS publications. Three bibliographic databases [PubMed, Scopus, and Web of Science (WOS)] and one full-text database (Google Scholar) were searched.

Keyword searches in bibliographic databases yielded high average precision (90%) but low average sensitivity (16%). PubMed was the most precise, followed closely by Scopus and WOS. The Google Scholar keyword search had low precision (54%) but provided the highest sensitivity (70%). Cited reference searches in all databases yielded moderate sensitivity (GS: 54%, WoS: 45%, Scopus 52%), but precision ranged from 35% to 75% with Scopus being the most precise.

When trying to identify many articles using a particular instrument, cited reference searching in Scopus or WOS is preferable to keyword searching in these resources or in PubMed/MEDLINE. However, in Google Scholar, a full-text database, a keyword search for CPS found more studies than the cited reference searches and precision was slightly higher.

If the goal is to identify a few studies that use a specific instrument, keyword searches of a bibliographic database may be the most efficient. When it is important to conduct a comprehensive search to maximize the number of studies found, both cited reference and keyword searches should be conducted in more than one database. 

Limitations stated by the author(s): 

“Our case study has limitations. The search results for the CPS may not be applicable to similar searches for other instruments. Other instruments may be more or less likely to be referred to in database records or article references. Results may also vary for a different information retrieval goal such as searching for a specific outcome reported in an instrument. In addition, our keyword search strategy was limited to the name of the instrument (and a variation of the named ‘‘preferences’’ instead of ‘‘preference’’) so that an identical search could be performed in each database. A more complex strategy that uses terms related to the instrument (ie, role preferences, patient involvement) could provide different results.”

Limitations stated by the reviewer(s): 
Authors did not state the recall in Google scholar for both articles used for citation tracking. It seems to be necessary to have initial article that introduced the instrument and the validation study.
Study Type: 
Single study

Appraisal of: Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. Pharmacoeconomics 2015; 33(1): 5-11.

Short description: 

This paper discusses the relative merits of a systematic citation searching approach as both an alternative and a concurrent method to keyword searching. A method of citation searching, both forwards and backwards, which is iterated to form a closed loop solution, is discussed. An illustrative example is presented of both methods, applying them to the topic of the UK National Institute for Health and Care Excellence (NICE) cost-effectiveness threshold. The case study finds the citation searching approach dominates the traditional keyword searching approach, finding 76 papers of relevance, including all 15 found by the alternative approach. 

Limitations stated by the author(s): 

No limitations stated regarding study methods.But limitations stated regarding search techniques under investigation:

Citation searching faces two major limitations which, it could be argued, have limited its implementation: (1) the ability to extract a paper’s references (backward citation searching) and identify those that have since cited it (forward); and (2) the completeness of citation networks and low efficiency of the search (p. 8) 

Limitations stated by the reviewer(s): 
Not stated whether study selection was performed by 2 people independent from each other; no information about performance of each search technique separately (forward and backward searching); no information about time; high number of hits (n = 4.529); results only compared to a very simple Boolean search (n = 36).
Study Type: 
Single study
Syndicate content