Citation searching

Scope: use to index papers providing advice on how to use citation analysis tools to identify relevant records for HTA and explorations of its value to the HTA process

Appraisal of: Lowe J, Peters J, Shields B, Cooper C. Methods to update systematic literature searches: full update searching vs. forward citation chasing; a case study from a systematic review of diagnostic accuracy [poster]. University of Exeter Medical

Short description: 

The aim of the study was to compare 2 methods used to update the literature searches. 1) A standard, full-update search using the Systematic Review’s (SR) original search strategy. 2) Using Web of Science (Thomson Reuters), forwards citation searching was conducted on the 10 studies included at full text in the SR.

In terms of time, the citation chasing arm was considerably more efficient. In terms of effectiveness, the full-update search produced a greater number of includable studies for full-text screening (full update search n=9, citation searching n=1). There was no cross-over between the arms of this case-study: each arm offered us unique studies, so no one arm could have been independently used.

Conclusion: full-update searching, including standard supplementary search methods, are advisable when searching for studies reporting diagnostic tests.

Limitations stated by the author(s): 

No limitations were stated by the authors.

Limitations stated by the reviewer(s): 
Abstract, no full text publication available; articles used for reference set “identified for full text screening” (not included references).
Study Type: 
Single study

Appraisal of: Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with Web of Science and Google Scholar. Res Synth Methods. 2016;7:34-45.

Short description: 

To examine how effectively forwards citation searching with Web of Science (WOS) or Google Scholar (GS) identified evidence to support public health guidance published by the National Institute for Health and Care Excellence.

The searches identified 43 relevant publications. The WOS process had 86.05% recall and 1.58% precision. The GS process had 90.7% recall and 1.62% precision. There were nine highly important publications. WOS had 100% recall and GS 88.89% recall.

WOS is recommended over GS, as citation searching was more effective, while the administrative and screening times and costs were lower.

Limitations stated by the author(s): 

The comparison is limited by what might be called linking lag. GS was searched in March 2012, and more citations might have been linked to the base set by the time WOS was searched in June 2012. The WOS records do not show the date on which a citation link has been added, and so it is impossible to replicate which citations it would have retrieved in March 2012. One of the disadvantages of citation searching is that it takes time for more recent publications to be cited and for these links to be picked up in WOS or GS. It would be worth investigating the extent to which citation searching would benefit from being supplemented by some focussed searches of PubMed to ensure very recent publications are included.

It is plausible that screening higher yields would have identified further relevant studies. Google does not provide access to the full set of 15 068 results, but additional, currently unknown, publications could have been located in the available set of 7692. This study began with the realisation that it was not feasible to screen the 26 385 MEDLINE results and that it is not known whether they contain other relevant publications. The resources were not available to screen the additional GS and MEDLINE results either for the systematic review or this supplementary study.

The functionality and scope of WOS made it the most appropriate comparator with GS for the NICE review. Other resources are available for citation searching, and Scopus (http://www.scopus.com/) has been compared with WOS and GS in other studies (Falagas et al., 2008; Kulkarni et al., 2009; Li et al., 2010; Wright et al., 2014). The resources were not available to conduct a wider study, but a direct comparison between WOS, GS and Scopus would have merit.

This case study has analysed the effectiveness and efficiency of using citation searching in a single review. A comparison of how citation searching performs in other topic areas would be beneficial to assess whether these conclusions have more general applicability, for example across the other 50 areas of NICE public health guidance. It would also be worth pursuing these methods in other disciplines that deploy complex interventions, have a disparate evidence base or lack controlled trials. It would be particularly useful to examine whether citation searching can help to overcome some of the difficulties experienced in finding evidence on social care topics (Clapton, 2010) since NICE gained additional responsibilities in this area in April 2013.

The costs of developing the base set were not calculated for this study because they were the same for WOS and GS. It would be interesting to compare the cost of developing a base set for citation searching, reference harvesting or other methods with the time taken to develop, test and implement a database search strategy. Further research is required to explore how the timing of citation searching and its relationship to database searching affect its value, given the differences between this study and Wright et al. (2014).

 

Limitations stated by the reviewer(s): 
Development of reference set was not systematic: 6 relevant publications by contacting experts; 40 relevant publications by checking reference lists. Study compared performance of WOS vs GS, but not against Boolean search e.g.
Study Type: 
Single study

Appraisal of: Linder SK, Kamath GR, Pratt GF, Saraykar SS, Volk RJ. Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments. J Clin Epidemiol. 2015; 68(4):412-417.

Short description: 

The aim of the study was to compare the effectiveness of two search methods in identifying studies that used the Control Preferences Scale (CPS), a health care decision-making instrument.

The literature was searched using two methods: (1) keyword searching using variations of ‘‘Control Preferences Scale’’ and (2) cited reference searching using two seminal CPS publications. Three bibliographic databases [PubMed, Scopus, and Web of Science (WOS)] and one full-text database (Google Scholar) were searched.

Keyword searches in bibliographic databases yielded high average precision (90%) but low average sensitivity (16%). PubMed was the most precise, followed closely by Scopus and WOS. The Google Scholar keyword search had low precision (54%) but provided the highest sensitivity (70%). Cited reference searches in all databases yielded moderate sensitivity (GS: 54%, WoS: 45%, Scopus 52%), but precision ranged from 35% to 75% with Scopus being the most precise.

When trying to identify many articles using a particular instrument, cited reference searching in Scopus or WOS is preferable to keyword searching in these resources or in PubMed/MEDLINE. However, in Google Scholar, a full-text database, a keyword search for CPS found more studies than the cited reference searches and precision was slightly higher.

If the goal is to identify a few studies that use a specific instrument, keyword searches of a bibliographic database may be the most efficient. When it is important to conduct a comprehensive search to maximize the number of studies found, both cited reference and keyword searches should be conducted in more than one database. 

Limitations stated by the author(s): 

“Our case study has limitations. The search results for the CPS may not be applicable to similar searches for other instruments. Other instruments may be more or less likely to be referred to in database records or article references. Results may also vary for a different information retrieval goal such as searching for a specific outcome reported in an instrument. In addition, our keyword search strategy was limited to the name of the instrument (and a variation of the named ‘‘preferences’’ instead of ‘‘preference’’) so that an identical search could be performed in each database. A more complex strategy that uses terms related to the instrument (ie, role preferences, patient involvement) could provide different results.”

Limitations stated by the reviewer(s): 
Authors did not state the recall in Google scholar for both articles used for citation tracking. It seems to be necessary to have initial article that introduced the instrument and the validation study.
Study Type: 
Single study

Appraisal of: Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. Pharmacoeconomics 2015; 33(1): 5-11.

Short description: 

This paper discusses the relative merits of a systematic citation searching approach as both an alternative and a concurrent method to keyword searching. A method of citation searching, both forwards and backwards, which is iterated to form a closed loop solution, is discussed. An illustrative example is presented of both methods, applying them to the topic of the UK National Institute for Health and Care Excellence (NICE) cost-effectiveness threshold. The case study finds the citation searching approach dominates the traditional keyword searching approach, finding 76 papers of relevance, including all 15 found by the alternative approach. 

Limitations stated by the author(s): 

No limitations stated regarding study methods.But limitations stated regarding search techniques under investigation:

Citation searching faces two major limitations which, it could be argued, have limited its implementation: (1) the ability to extract a paper’s references (backward citation searching) and identify those that have since cited it (forward); and (2) the completeness of citation networks and low efficiency of the search (p. 8) 

Limitations stated by the reviewer(s): 
Not stated whether study selection was performed by 2 people independent from each other; no information about performance of each search technique separately (forward and backward searching); no information about time; high number of hits (n = 4.529); results only compared to a very simple Boolean search (n = 36).
Study Type: 
Single study

Appraisal of: Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 2005; 331(7524): 1064-1065.

Short description: 

To describe where papers come from in a systematic review of complex evidence. Only 30% of sources were obtained from the protocol defined at the outset of the study (that is, from the database and hand searches). Fifty one per cent were identified by “snowballing” (such as scanning reference lists (44%) and citation tracking (7%)), and 24% by personal knowledge or personal contacts.

Limitations stated by the author(s): 

No limitations were stated by the authors.

Limitations stated by the reviewer(s): 
The article assesses the total contribution of the different sources but it is not possible to identify the additional references which could only be found by the additional search techniques. Besides that it is not mentioned if perhaps the original Boolean search was inadequate as the searches and the results are not discussed in detail. The generalizability of the results is also questionable because the authors results are based only on one review (topic “diffusion of service-level innovations in healthcare organizations”).
Study Type: 
Single study
Syndicate content