Citation checking

Scope: use to index papers providing descriptions of this method of checking reference lists within documents and explorations of its value to the HTA process

Appraisal of: "Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev, 5(1). doi:10.1186/s13643-016-0249-x"

Reviewer(s): 
Short description: 

This review provides a systematic overview of the available published evidence of searching methods to inform qualitative evidence synthesis (QES). The author sought to assess and identify:

1) the current state of knowledge in relation to searching for qualitative evidence

2) the robustness of the evidence base

3) research gaps and future priorities.

The studies were obtained from the Reference Manager database of the Cochrane Qualitative Methods Group’s study register, of which the author is responsible for updating and maintaining. Supplementary citation searches via Google Scholar was also carried out for 15 key papers. 113 studies were assessed for inclusion. Quality assessment of the included studies was not deemed feasible due to a large proportion of the included studies providing only narrative findings, the lack of a common appraisal instrument and the high levels of heterogeneity across the remaining studies.

The evidence underpinning systematic approaches to searching for qualitative evidence is classified and summarized within one or more of eight headings/ “7 S structure/ framework”: overviews and methodological guidelines, sampling, sources, structured questions, search procedures, search strategies and filters, supplementary strategies and standards. The author summarizes the available evidence and key issues within each section and makes recommendations for further empirical research. Table 7 breaks down the key starting principles in reference to the “7S structure” of searching to inform qualitative evidence synthesis to inform future guidance and Table 8 provides an overview of research priorities.

The review concludes that there is a lack of empirical data to inform information retrieval for QES and that the strength of the evidence is weak and largely based on personal/ professional experience and case studies. Advances have been made in reporting QES, however, validated standards are lacking. 

Limitations stated by the author(s): 

The author states that in order for studies to be included, the references needed to include terms specifically related to searching or retrieval in their titles or abstracts, cite a number of key texts, or be referred to from previously identified items. The full-text of all papers reporting QES were not examined. There is a possibility that potentially relevant reviews reporting emerging information retrieval methods that were not reported in the title or abstract were missed. However, these risks are offset by the sensitive search approach and the currency and comprehensiveness of the Cochrane Qualitative Methods Group study register. The author also notes that some papers were excluded as they did not distinguish between qualitative and quantitative approaches, which could potentially be useful for mixed methods reviews.

Limitations stated by the reviewer(s): 
1. This was a single reviewer methodology review so judgements on eligibility and interpretations of potential significance and contribution of individual studies may not be consistent and/or reproducible. [Study Selection Bias] 2. Although as a methodology review there is no formal requirement to follow PRISMA reporting standards this review may have benefited from more complete and transparent reporting. [Incomplete Reporting Bias] 3. As the reviewer was author on a high proportion of included studies this may have consciously impacted on study identification and subconsciously on study selection and interpretation. [Citation Bias; Observer Bias]
Study Type: 
Single study

Appraisal of: “Belter CW. Citation analysis as a literature search method for systematic reviews. J Assn Inf Sci Tec 2016; 67(11): 2766-2777.”

Short description: 

The author proposed a method of searching the literature by systematically mining the various types of citation relationships between articles (cited by, citing, cociting, cocited articles).

The author tested the method by comparing its precision and recall to that of 14 published systematic reviews. The method successfully retrieved 74% of the studies included in these reviews and 90% of the studies it could reasonably be expected to retrieve. Indirect citation relationships clearly outperformed direct citation relationships in retrieving relevant publications. The method also retrieved fewer than half of the total number of publications retrieved by these reviews and can be performed in substantially less time. 

Limitations stated by the author(s): 

This analysis clearly demonstrated that certain kinds of studies can rarely be identified using citation relationships. This is due both to the citation behavior of authors and the limitations of citation databases. Because authors rarely cite conference proceedings, and even more rarely cite studies in progress, it is extremely difficult for the proposed method to identify studies published or made available in these formats. Further, because these publication formats rarely include indexed cited references, the method is even less likely to identify them. This limitation means that the proposed method is even more prone to publication bias than the traditional method, meaning that its ability to identify descriptions of negative results is minimal. In addition, studies published in journals not indexed by the Web of Science were rarely identified by the proposed method. This means that certain kinds of studies, especially studies published in non-Western journals like the traditional Chinese medicine review excluded from this analysis, will rarely be identified by this method. Although using Scopus instead of, or in addition to, the Web of Science may be able to identify some of these publications, there are still a large number of journals not indexed by either database. The proposed method would be unlikely to retrieve studies published in such journals. Because of these limitations, it is unlikely that the proposed method could ever replace text-based searching for systematic reviews.

Although the method as implemented here performed relatively well, I made several arbitrary choices in implementing it in this analysis. My decision to use two key studies on which to base the method was one of these arbitrary choices.

Other arbitrary choices in this method were the thresholds I used to filter the cociting and cocited articles.

Limitations stated by the reviewer(s): 
No additional limitations were detected by the reviewers.
Study Type: 
Single study

Appraisal of: Preston, L., et al. "Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking." Syst Rev 2015 4: 82.

Reviewer(s): 
Short description: 

A convenience sample of nine Health Technology Assessment (HTA) systematic reviews of diagnostic test accuracy, with 302 included citations, was analysed to determine the number and proportion of included citations that were indexed in and retrieved from MEDLINE and EMBASE. An assessment was also made of the number and proportion of citations not retrieved from these databases but that could have been identified from the reference lists of included citations.

287/302 (95 %) of the included citations were indexed across MEDLINE and EMBASE. The reviews’ searches of MEDLINE and EMBASE accounted for 85 % of the included citations (256/302). Of the forty-six (15 %) included citations not retrieved by the published searches, 24 (8 %) could be found in the reference lists of included citations. Only 22/302 (7 %) of the included citations were not found. 

Limitations stated by the author(s): 

This study used a small, non-random sample of diagnostic test accuracy systematic reviews. This was done for reasons of pragmatism: first, because the authors had full access to the search strategies and reference databases of these reviews and, second, because of the exploratory nature of this project. We also assumed that the vast majority of the included citations in the reviews were located through screening of titles, abstracts and full papers.

We have also assumed, because the number of studies missed by operating the proposed MEDLINE, EMBASE and reference tracking strategy is so small that the findings of the systematic reviews would not have been greatly affected by their omission. However, this is uncertain and can only be assessed statistically by excluding those particular studies from the many analyses reported in the reviews, although, as noted above, most of these reviews conducted narrative synthesis. Such an analysis is a major task to undertake retrospectively and has therefore not been completed in this exploratory study. Future work should test the findings of this small study in a larger, preferably prospective sample of systematic reviews from multiple institutions. If possible, statistical analysis should also be undertaken to quantify fully the impact of omitting any data from studies that might otherwise be missed.

Limitations stated by the reviewer(s): 
No further limitations were identified.
Study Type: 
Single study

Appraisal of: "Janssens, A. C., Gwinn M. Novel citation-based search method for scientific literature: application to meta-analyses. BMC Medical Research Methodology 2015 15: 84."

Reviewer(s): 
Short description: 

In two independent studies, the authors aimed to reproduce the results of literature searches for sets of published meta-analyses (n = 10 and n = 42). For each meta-analysis, they extracted co-citations for the randomly selected ‘known’ articles from the Web of Science database, counted their frequencies and screened all articles with a score above a selection threshold. In the second study, the authors extended the method by retrieving direct citations for all selected articles.

In the first study, they retrieved 82 % of the studies included in the meta-analyses while screening only 11 % as many articles as were screened for the original publications. Articles that were missed were published in non-English languages, published before 1975, published very recently, or available only as conference abstracts. In the second study, they retrieved 79 % of included studies while screening half the original number of articles.

Citation searching appears to be an efficient and reasonably accurate method for finding articles similar to one or more articles of interest for meta-analysis and reviews.

Limitations stated by the author(s): 

Clearly, the method does not work when the topic of the meta-analysis is heterogeneous and the studies of interest are unlikely to have cited each other. The second study also included several meta-analyses with very small sample sizes, including one in which half of the studies were case reports that had few or no references [21], as well as a meta-analysis for which the ‘known’ studies were cited only four times in total [22]. The percentage of retrieved studies jumped to 89 % when these five metaanalyses were excluded.

The accuracy and efficiency of co-citation searching depends on characteristics of the underlying citation network. By design, our method misses the studies that the collective community of researchers apparently did not find worth citing. In our analysis, these included abstracts, articles in non-English languages, very old articles, and publications in semi-scientific journals, reports, websites, and theses. In addition, some newer and some very old articles were not cited often enough to rank high in our search.

Limitations stated by the reviewer(s): 
The authors performed 2 independent studies with slightly different methods (e.g. thresholds, use of “direct citations”). It is not clear which method the authors recommend for further use.
Study Type: 
Single study

Appraisal of: Westphal A, Kriston L, Holzel LP, Harter M, von Wolff A. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. J Public H

Short description: 

The aim of this investigation was to assess the efficiency and contribution of additional search strategies for identifying randomized controlled trials in conducting a systematic review on the efficacy of psychotherapeutic, pharmacological and combined interventions in the treatment of chronic depression after performing a sensitive electronic database search.

Seven electronic databases, 3 journals and 11 systematic reviews were searched. All first authors of the included studies were contacted; citation tracking and a search in clinical trial registers were performed.

50 studies were included in the systematic review, wherefrom 94% were acquired by the sensitive electronic database search and 16% through additional search strategies. Screening reference lists of related systematic reviews was the most beneficial additional search strategy with a contribution of 5 additional references. Citation tracking did not lead to any further inclusion of primary studies. 

Limitations stated by the author(s): 

Since our investigation is a single case study of a systematic review on interventions of chronic depression, the generalizability of our findings to other research topics may be limited. Above, the presented search strategies were applied in the research area of therapies that focus on chronic depression so that we could draw on a relatively large number of existing systematic reviews, 11 in total, that were rewarding for our analysis. Accordingly, in areas less researched, this beneficial additional search strategy cannot be applied.23 Furthermore, we conducted a sensitive electronic database search which might have limited the gain of additional search strategies. In contrast, the use of only one database (e.g., MEDLINE) and a more restrictive search could have resulted in a significantly larger benefit through additional search strategies.

Another possible limitation of our findings may consist in the choice of the journals that were searched for relevant content since this selection may have influenced the number of identified studies through this strategy as well as the relative efficiency of other strategies conducted subsequently.

In addition, we did not consult an expert, e.g. a librarian, for designing the terms for the electronic database search, so we may have missed a few studies. However, it should be noted that several of the authors have extensive training and experience in performing systematic reviews and searches within them.

Above, the contribution of any additional search strategy strongly depends on the order of its application. In this case contents of relevant journals were searched at first. As we did not investigate any alternative sequence of additional search strategies and thus did not calculate the overlap between the applied additional search strategies, the generalizability of our findings may be limited. Although we applied many additional search strategies, the search for grey literature was limited. As the inclusion of unpublished trials in systematic reviews is a controversially discussed issue that is capable of both reducing and introducing bias, we decided not to search for additional unpublished trials.

Limitations stated by the reviewer(s): 
It is not stated which references and how many were used for citation tracking and screening reference lists of systematic reviews.
Study Type: 
Single study

Appraisal of: Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Info Lib J. 2010; 27(2): 114-122.

Short description: 

A conventional subject search was undertaken as the principal method of identifying the literature for this review. Four supplementary search methods were used including citation searching, reference list checking, contact with experts and pearl growing.

11 of  41 references were identified via citation searching, reference list checking and contact with experts (27%). Pearl growing was suspended as the nominated pearls were dispersed across numerous databases, with no single database indexing more than four pearls. Citation searching identified 3 (5 ?) references, reference list checking 4, and contact with experts 2 additional references. Problems in indexing, ambiguity of terms and limited abstract content resulted in reduced sensitivity for the main subject search.

Limitations stated by the author(s): 

No limitations were stated by the authors.

Limitations stated by the reviewer(s): 
Generalisabilty may be limited as the information retrieval of only one systematic review was assessed.
Study Type: 
Single study

Appraisal of: Chapman AL, Morgan LC, Gartlehner G. Semi-automating the manual literature search for systematic reviews increases efficiency. Health Info Libr J. 2009; 27(1):22-7.

Short description: 

Manual literature searching (or hand-searching) for systematic reviews is often promoted as a way of minimizing retrieval bias (the failure to find relevant studies due to the retrieval capacity of electronic databases), but it can be a lengthy and unsystematic process. This study aimed to determine the validity of a new manual search method to check reference lists of identified studies for relevant articles using the Scopus database by comparing it to the traditional method of manually checking reference lists as the gold standard. Outcome measures included accuracy and completeness of article detection (validity) and personnel time involved (efficiency). The Scopus method identified the same studies as the gold standard, but was much more time efficient (3 hours versus 8 hours, time saving of 62.5%). The study authors concluded that the Scopus method could significantly improve the efficiency of manual searches and thus of systematic reviews.

Limitations stated by the author(s): 

Findings are based on one medium-sized systematic review. Further method studies are needed to validate these findings and identify potential improvement to this approach.

 Scopus requires a paid subscription.

Scopus currently does not include the reference lists of Cochrane reviews, which is a major limitation as Cochrane reviews are a major source of trial information.

Limitations stated by the reviewer(s): 
Findings based on reference checking of only 20 studies.
Study Type: 
Single study

Appraisal of: Horsley, T.; Dingwall, O.; Tetzlaff, J. M.; Sampson, M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev. 2009;(1):MR000026.

Short description: 

The aim of this Cochrane review was to investigate the effectiveness of checking reference lists for the identification of additional, relevant studies for systematic reviews. From the 12 included studies, the authors found that there was some evidence to support the use of checking reference lists for locating additional studies in systematic reviews, although this evidence is mainly derived from heterogeneous studies of poor quality. They conclude that, in situations where the identification of all relevant studies through hand searching and Boolean database searching is difficult, it would seem prudent that authors of reviews check reference lists to supplement their searching. 

Limitations stated by the author(s): 

The authors included 12 studies (in 13 publications) in this review, but interpretability and generalizability of these studies is difficult and the study designs used were at high risk of bias.

Limitations stated by the reviewer(s): 
No other limitations were detected by the reviewers.
Study Type: 
Review

Appraisal of: Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. Pharmacoeconomics 2015; 33(1): 5-11.

Short description: 

This paper discusses the relative merits of a systematic citation searching approach as both an alternative and a concurrent method to keyword searching. A method of citation searching, both forwards and backwards, which is iterated to form a closed loop solution, is discussed. An illustrative example is presented of both methods, applying them to the topic of the UK National Institute for Health and Care Excellence (NICE) cost-effectiveness threshold. The case study finds the citation searching approach dominates the traditional keyword searching approach, finding 76 papers of relevance, including all 15 found by the alternative approach. 

Limitations stated by the author(s): 

No limitations stated regarding study methods.But limitations stated regarding search techniques under investigation:

Citation searching faces two major limitations which, it could be argued, have limited its implementation: (1) the ability to extract a paper’s references (backward citation searching) and identify those that have since cited it (forward); and (2) the completeness of citation networks and low efficiency of the search (p. 8) 

Limitations stated by the reviewer(s): 
Not stated whether study selection was performed by 2 people independent from each other; no information about performance of each search technique separately (forward and backward searching); no information about time; high number of hits (n = 4.529); results only compared to a very simple Boolean search (n = 36).
Study Type: 
Single study

Appraisal of: Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 2005; 331(7524): 1064-1065.

Short description: 

To describe where papers come from in a systematic review of complex evidence. Only 30% of sources were obtained from the protocol defined at the outset of the study (that is, from the database and hand searches). Fifty one per cent were identified by “snowballing” (such as scanning reference lists (44%) and citation tracking (7%)), and 24% by personal knowledge or personal contacts.

Limitations stated by the author(s): 

No limitations were stated by the authors.

Limitations stated by the reviewer(s): 
The article assesses the total contribution of the different sources but it is not possible to identify the additional references which could only be found by the additional search techniques. Besides that it is not mentioned if perhaps the original Boolean search was inadequate as the searches and the results are not discussed in detail. The generalizability of the results is also questionable because the authors results are based only on one review (topic “diffusion of service-level innovations in healthcare organizations”).
Study Type: 
Single study
Syndicate content