Citation searching

Scope: use to index papers providing advice on how to use citation analysis tools to identify relevant records for HTA and explorations of its value to the HTA process

Appraisal of: "Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev, 5(1). doi:10.1186/s13643-016-0249-x"

Reviewer(s): 
Short description: 

This review provides a systematic overview of the available published evidence of searching methods to inform qualitative evidence synthesis (QES). The author sought to assess and identify:

1) the current state of knowledge in relation to searching for qualitative evidence

2) the robustness of the evidence base

3) research gaps and future priorities.

The studies were obtained from the Reference Manager database of the Cochrane Qualitative Methods Group’s study register, of which the author is responsible for updating and maintaining. Supplementary citation searches via Google Scholar was also carried out for 15 key papers. 113 studies were assessed for inclusion. Quality assessment of the included studies was not deemed feasible due to a large proportion of the included studies providing only narrative findings, the lack of a common appraisal instrument and the high levels of heterogeneity across the remaining studies.

The evidence underpinning systematic approaches to searching for qualitative evidence is classified and summarized within one or more of eight headings/ “7 S structure/ framework”: overviews and methodological guidelines, sampling, sources, structured questions, search procedures, search strategies and filters, supplementary strategies and standards. The author summarizes the available evidence and key issues within each section and makes recommendations for further empirical research. Table 7 breaks down the key starting principles in reference to the “7S structure” of searching to inform qualitative evidence synthesis to inform future guidance and Table 8 provides an overview of research priorities.

The review concludes that there is a lack of empirical data to inform information retrieval for QES and that the strength of the evidence is weak and largely based on personal/ professional experience and case studies. Advances have been made in reporting QES, however, validated standards are lacking. 

Limitations stated by the author(s): 

The author states that in order for studies to be included, the references needed to include terms specifically related to searching or retrieval in their titles or abstracts, cite a number of key texts, or be referred to from previously identified items. The full-text of all papers reporting QES were not examined. There is a possibility that potentially relevant reviews reporting emerging information retrieval methods that were not reported in the title or abstract were missed. However, these risks are offset by the sensitive search approach and the currency and comprehensiveness of the Cochrane Qualitative Methods Group study register. The author also notes that some papers were excluded as they did not distinguish between qualitative and quantitative approaches, which could potentially be useful for mixed methods reviews.

Limitations stated by the reviewer(s): 
1. This was a single reviewer methodology review so judgements on eligibility and interpretations of potential significance and contribution of individual studies may not be consistent and/or reproducible. [Study Selection Bias] 2. Although as a methodology review there is no formal requirement to follow PRISMA reporting standards this review may have benefited from more complete and transparent reporting. [Incomplete Reporting Bias] 3. As the reviewer was author on a high proportion of included studies this may have consciously impacted on study identification and subconsciously on study selection and interpretation. [Citation Bias; Observer Bias]
Study Type: 
Single study

Cooper C, Booth A, Britten N, Garside R. "A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review." Syst Rev. 2017; 6(1): 234.

Reviewer(s): 
Short description: 

The purpose and contribution of supplementary search methods in systematic reviews is increasingly acknowledged. Numerous studies have demonstrated their potential in identifying studies or study data that would have been missed by bibliographic database searching alone. What is less certain is how supplementary search methods actually work, how they are applied, and the consequent advantages, disadvantages and resource implications of each search method. The aim of this study is to compare current practice in using supplementary search methods with methodological guidance.

Limitations stated by the author(s): 

The date range and age of the handbooks and studies included in this review could be considered a limitation of this study.

Comparative and non-comparative case studies form the evidence base for this study. The studies included in this review have been taken at face-value, and no formal quality appraisal has been undertaken since no suitable tool exists.

Furthermore, supplementary search methods are typically evaluated in the context of effectiveness, which is potentially a limited test of the contribution they may offer in the process of study identification. Different thresholds of effectiveness and efficiency may apply in the use of supplementary search methods in systematic reviews of qualitative studies when compared to reviews of RCTs, for example.

Whilst we have aimed to comprehensively identify and review studies for inclusion, the use of supplementary search methods is a broad field of study and it is possible that some completed studies may have been inadvertently missed or overlooked. It is possible that standard systematic review techniques, such as double-screening, would have minimised this risk, but we are confident that, whilst a more systematic approach may have improved the rigour of the study, it is unlikely to alter the conclusions below.

Appraisal of: Robinson KA, Dunn AG, Tsafnaf G, Glasziou P. Citation networks of related trials are often disconnected: implications for bidirectional citation searches. J Clin Epidemiol. 2014;67(7): 793-799.

Reviewer(s): 
Short description: 

The authors sought to determine the connectedness of citation networks of RCTs by examining direct (referenced trials) and indirect (through references of referenced trials, etc) citation of trials to one another. 259 meta-analyses including 2,413 RCTs were used to create citation networks of RCTs addressing the same clinical questions.

For 46% (118 of 259) of networks, the RCTs formed a single connected citation group - one island. For the other 54% of networks, where at least one RCT group was not cited by others, 39% had two citation islands and 4% (10 of 257) had 10 or more islands.

Conclusion: Available evidence to answer a clinical question may be identified by using network citations created with a small initial corpus of eligible trials. However, the number of islands means that citation networks cannot be relied on for evidence retrieval.

Limitations stated by the author(s): 

The authors stated some limitations.

They used Web of Science to identify the Meta-Analysis and the RCTs included in those Meta-Analysis. Some networks are thus not complete as one or more of the RCTs from the MA are not included in their data set. Given the relatively small number of RCTs missing per network, it is not likely that these missing RCTs would substantively change the results.

Another limitation is that a trial could have cited systematic reviews rather than individual prior trials. They did not check this as other studies have suggested few trials cite up-to-date systematic reviews.

Limitations stated by the reviewer(s): 
The study is of high quality, one limitation might be, that only trials were included in the analysis. However, the results might be better when all references from the reference lists were analysed.
Study Type: 
Single study

Appraisal of: Belter CW. "A relevance ranking method for citation-based search results." Scientometrics 2017;112(2): 731-746.

Reviewer(s): 
Short description: 

This study tests a method which was modified from a previous study (Belter 2016). In this paper, the author proposed a method of ranking the relevance of citation-based search results to a set of key, or seed, papers by measuring the number of citation relationships they share with those key papers.

The method was tested against 23 published systematic reviews and it was found that the method retrieved 87% of the studies included in these reviews. Additional testing suggested that the method may be less appropriate for reviews that combine literature in ways that are not reflected in the literature itself. 

Limitations stated by the author(s): 

 

These results also suggests that the opposite is also true: that the method is less successful, and therefore less appropriate for literature retrieval, in reviews that combine topics in ways that the literature does not.
 
It may be that this method is not appropriate for literature retrieval in reviews that seek to combine the results of multiple interventions, multiple treatment populations, or multiple disorders. 
Limitations stated by the reviewer(s): 
No other limitations were detected by the reviewers.
Study Type: 
Single study

Appraisal of: "Janssens, A. C., Gwinn M. Novel citation-based search method for scientific literature: application to meta-analyses. BMC Medical Research Methodology 2015 15: 84."

Reviewer(s): 
Short description: 

In two independent studies, the authors aimed to reproduce the results of literature searches for sets of published meta-analyses (n = 10 and n = 42). For each meta-analysis, they extracted co-citations for the randomly selected ‘known’ articles from the Web of Science database, counted their frequencies and screened all articles with a score above a selection threshold. In the second study, the authors extended the method by retrieving direct citations for all selected articles.

In the first study, they retrieved 82 % of the studies included in the meta-analyses while screening only 11 % as many articles as were screened for the original publications. Articles that were missed were published in non-English languages, published before 1975, published very recently, or available only as conference abstracts. In the second study, they retrieved 79 % of included studies while screening half the original number of articles.

Citation searching appears to be an efficient and reasonably accurate method for finding articles similar to one or more articles of interest for meta-analysis and reviews.

Limitations stated by the author(s): 

Clearly, the method does not work when the topic of the meta-analysis is heterogeneous and the studies of interest are unlikely to have cited each other. The second study also included several meta-analyses with very small sample sizes, including one in which half of the studies were case reports that had few or no references [21], as well as a meta-analysis for which the ‘known’ studies were cited only four times in total [22]. The percentage of retrieved studies jumped to 89 % when these five metaanalyses were excluded.

The accuracy and efficiency of co-citation searching depends on characteristics of the underlying citation network. By design, our method misses the studies that the collective community of researchers apparently did not find worth citing. In our analysis, these included abstracts, articles in non-English languages, very old articles, and publications in semi-scientific journals, reports, websites, and theses. In addition, some newer and some very old articles were not cited often enough to rank high in our search.

Limitations stated by the reviewer(s): 
The authors performed 2 independent studies with slightly different methods (e.g. thresholds, use of “direct citations”). It is not clear which method the authors recommend for further use.
Study Type: 
Single study

Appraisal of: Westphal A, Kriston L, Holzel LP, Harter M, von Wolff A. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. J Public H

Short description: 

The aim of this investigation was to assess the efficiency and contribution of additional search strategies for identifying randomized controlled trials in conducting a systematic review on the efficacy of psychotherapeutic, pharmacological and combined interventions in the treatment of chronic depression after performing a sensitive electronic database search.

Seven electronic databases, 3 journals and 11 systematic reviews were searched. All first authors of the included studies were contacted; citation tracking and a search in clinical trial registers were performed.

50 studies were included in the systematic review, wherefrom 94% were acquired by the sensitive electronic database search and 16% through additional search strategies. Screening reference lists of related systematic reviews was the most beneficial additional search strategy with a contribution of 5 additional references. Citation tracking did not lead to any further inclusion of primary studies. 

Limitations stated by the author(s): 

Since our investigation is a single case study of a systematic review on interventions of chronic depression, the generalizability of our findings to other research topics may be limited. Above, the presented search strategies were applied in the research area of therapies that focus on chronic depression so that we could draw on a relatively large number of existing systematic reviews, 11 in total, that were rewarding for our analysis. Accordingly, in areas less researched, this beneficial additional search strategy cannot be applied.23 Furthermore, we conducted a sensitive electronic database search which might have limited the gain of additional search strategies. In contrast, the use of only one database (e.g., MEDLINE) and a more restrictive search could have resulted in a significantly larger benefit through additional search strategies.

Another possible limitation of our findings may consist in the choice of the journals that were searched for relevant content since this selection may have influenced the number of identified studies through this strategy as well as the relative efficiency of other strategies conducted subsequently.

In addition, we did not consult an expert, e.g. a librarian, for designing the terms for the electronic database search, so we may have missed a few studies. However, it should be noted that several of the authors have extensive training and experience in performing systematic reviews and searches within them.

Above, the contribution of any additional search strategy strongly depends on the order of its application. In this case contents of relevant journals were searched at first. As we did not investigate any alternative sequence of additional search strategies and thus did not calculate the overlap between the applied additional search strategies, the generalizability of our findings may be limited. Although we applied many additional search strategies, the search for grey literature was limited. As the inclusion of unpublished trials in systematic reviews is a controversially discussed issue that is capable of both reducing and introducing bias, we decided not to search for additional unpublished trials.

Limitations stated by the reviewer(s): 
It is not stated which references and how many were used for citation tracking and screening reference lists of systematic reviews.
Study Type: 
Single study

Appraisal of: Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Info Lib J. 2010; 27(2): 114-122.

Short description: 

A conventional subject search was undertaken as the principal method of identifying the literature for this review. Four supplementary search methods were used including citation searching, reference list checking, contact with experts and pearl growing.

11 of  41 references were identified via citation searching, reference list checking and contact with experts (27%). Pearl growing was suspended as the nominated pearls were dispersed across numerous databases, with no single database indexing more than four pearls. Citation searching identified 3 (5 ?) references, reference list checking 4, and contact with experts 2 additional references. Problems in indexing, ambiguity of terms and limited abstract content resulted in reduced sensitivity for the main subject search.

Limitations stated by the author(s): 

No limitations were stated by the authors.

Limitations stated by the reviewer(s): 
Generalisabilty may be limited as the information retrieval of only one systematic review was assessed.
Study Type: 
Single study

Appraisal of: Sampson M, Shojania KG, McGowan J, Daniel R, Rader T, Iansavichene AE et al. Surveillance search techniques identified the need to update systematic reviews. J Clin Epidemiol 2008; 61(8): 755-762.

Short description: 

The article reports on literature surveillance methods to identify new evidence eligible for updating systematic reviews. Five surveillance search approaches are tested in the context of identifying studies that would signal major or invalidating new evidence for existing systematic reviews of health care interventions.

Searches were tested in a cohort of 77 systematic reviews. No one method yielded consistently high recall of relevant new evidence, so combinations of the strategies were examined. A search algorithm based on PubMed’s related article search in combination with a subject searching using clinical queries was the most effective combination, retrieving all relevant new records in 68 cases. Citing RCT searching found unique material in only 2 cases (recall 37%).

Surveillance for emerging evidence that signals the need to update systematic reviews is feasible using a combination of subject searching and searching based on the PubMed’s related article function.

Limitations stated by the author(s): 

The retrospective observational design used here has certain limitations including absence of control over data collectiondin the case of the CENTRAL database, where indexing dates of records are not recorded; this limited our ability to determine when new evidence would have been available to reviewers. Our assessment of the point at which a review was in need of update could have been influenced by our access to subsequent confirmatory evidence, although any resulting bias would influence each search approach equally.

Limitations stated by the reviewer(s): 
No additional limitations were detected by the authors.
Study Type: 
Single study

Appraisal of: Wright K, Golder S, Rodriguez-Lopez R. Citation searching: a systematic review case study of multiple risk behaviour interventions. BMC Med Res Methodol. 2014; 14:73.

Short description: 

Study about the effectiveness of using the citation sources Google Scholar, Scopus, Web of Science and OVIDSP MEDLINE to identify records for inclusion in a systematic review about multiple risk behaviour interventions.

40 included studies were used identified by traditional database searches from one systematic review. Each of the included studies was searched for in the four citation sources to retrieve the details of all papers that have cited these studies.

The highest number of citations was retrieved from Google Scholar (1680), followed by Scopus (1173), then Web of Science (1095) and lastly OVIDSP (213). To retrieve all the records identified by the citation tracking searching all four resources was required. Google Scholar identified the highest number of unique citations. The citation tracking identified 9 studies that met the review’s inclusion criteria (recall 22,5%).

Conclusion: Citation searching as a supplementary search method for systematic reviews may not be the best use of valuable time and resources.

Limitations stated by the author(s): 

In terms of searching, the scoping review had very broad coverage. Its aim was to identify any intervention promoting change in at least two risk behaviours and the search strategy incorporated terms for all of these (smoking, alcohol misuse, physical inactivity, unhealthy diet, illicit drug use, sexual risk behaviour, lack of seat belt use, lack of motorcycle/bicycle helmet use, lack of sunscreen use, gambling, poor oral hygiene and drink driving) in various set combinations. The resulting complexity will almost certainly have had an impact upon the overall performance of the database search strategies. As with any case study, there is uncertainty about how far the results of this study can be generalised, especially to other reviews with a more restrictive focus.

Limitations stated by the reviewer(s): 
No additional limitations were detected by the reviewers.
Study Type: 
Single study

Appraisal of: Lowe J, Peters J, Shields B, Cooper C. Methods to update systematic literature searches: full update searching vs. forward citation chasing; a case study from a systematic review of diagnostic accuracy [poster]. University of Exeter Medical

Short description: 

The aim of the study was to compare 2 methods used to update the literature searches. 1) A standard, full-update search using the Systematic Review’s (SR) original search strategy. 2) Using Web of Science (Thomson Reuters), forwards citation searching was conducted on the 10 studies included at full text in the SR.

In terms of time, the citation chasing arm was considerably more efficient. In terms of effectiveness, the full-update search produced a greater number of includable studies for full-text screening (full update search n=9, citation searching n=1). There was no cross-over between the arms of this case-study: each arm offered us unique studies, so no one arm could have been independently used.

Conclusion: full-update searching, including standard supplementary search methods, are advisable when searching for studies reporting diagnostic tests.

Limitations stated by the author(s): 

No limitations were stated by the authors.

Limitations stated by the reviewer(s): 
Abstract, no full text publication available; articles used for reference set “identified for full text screening” (not included references).
Study Type: 
Single study
Syndicate content