Pearl growing

Scope: also includes snowballing – use to index papers providing explorations and descriptions of the technique to develop search strategies

Appraisal of: "Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev, 5(1). doi:10.1186/s13643-016-0249-x"

Reviewer(s): 
Short description: 

This review provides a systematic overview of the available published evidence of searching methods to inform qualitative evidence synthesis (QES). The author sought to assess and identify:

1) the current state of knowledge in relation to searching for qualitative evidence

2) the robustness of the evidence base

3) research gaps and future priorities.

The studies were obtained from the Reference Manager database of the Cochrane Qualitative Methods Group’s study register, of which the author is responsible for updating and maintaining. Supplementary citation searches via Google Scholar was also carried out for 15 key papers. 113 studies were assessed for inclusion. Quality assessment of the included studies was not deemed feasible due to a large proportion of the included studies providing only narrative findings, the lack of a common appraisal instrument and the high levels of heterogeneity across the remaining studies.

The evidence underpinning systematic approaches to searching for qualitative evidence is classified and summarized within one or more of eight headings/ “7 S structure/ framework”: overviews and methodological guidelines, sampling, sources, structured questions, search procedures, search strategies and filters, supplementary strategies and standards. The author summarizes the available evidence and key issues within each section and makes recommendations for further empirical research. Table 7 breaks down the key starting principles in reference to the “7S structure” of searching to inform qualitative evidence synthesis to inform future guidance and Table 8 provides an overview of research priorities.

The review concludes that there is a lack of empirical data to inform information retrieval for QES and that the strength of the evidence is weak and largely based on personal/ professional experience and case studies. Advances have been made in reporting QES, however, validated standards are lacking. 

Limitations stated by the author(s): 

The author states that in order for studies to be included, the references needed to include terms specifically related to searching or retrieval in their titles or abstracts, cite a number of key texts, or be referred to from previously identified items. The full-text of all papers reporting QES were not examined. There is a possibility that potentially relevant reviews reporting emerging information retrieval methods that were not reported in the title or abstract were missed. However, these risks are offset by the sensitive search approach and the currency and comprehensiveness of the Cochrane Qualitative Methods Group study register. The author also notes that some papers were excluded as they did not distinguish between qualitative and quantitative approaches, which could potentially be useful for mixed methods reviews.

Limitations stated by the reviewer(s): 
1. This was a single reviewer methodology review so judgements on eligibility and interpretations of potential significance and contribution of individual studies may not be consistent and/or reproducible. [Study Selection Bias] 2. Although as a methodology review there is no formal requirement to follow PRISMA reporting standards this review may have benefited from more complete and transparent reporting. [Incomplete Reporting Bias] 3. As the reviewer was author on a high proportion of included studies this may have consciously impacted on study identification and subconsciously on study selection and interpretation. [Citation Bias; Observer Bias]
Study Type: 
Single study

Appraisal of: “Belter CW. Citation analysis as a literature search method for systematic reviews. J Assn Inf Sci Tec 2016; 67(11): 2766-2777.”

Short description: 

The author proposed a method of searching the literature by systematically mining the various types of citation relationships between articles (cited by, citing, cociting, cocited articles).

The author tested the method by comparing its precision and recall to that of 14 published systematic reviews. The method successfully retrieved 74% of the studies included in these reviews and 90% of the studies it could reasonably be expected to retrieve. Indirect citation relationships clearly outperformed direct citation relationships in retrieving relevant publications. The method also retrieved fewer than half of the total number of publications retrieved by these reviews and can be performed in substantially less time. 

Limitations stated by the author(s): 

This analysis clearly demonstrated that certain kinds of studies can rarely be identified using citation relationships. This is due both to the citation behavior of authors and the limitations of citation databases. Because authors rarely cite conference proceedings, and even more rarely cite studies in progress, it is extremely difficult for the proposed method to identify studies published or made available in these formats. Further, because these publication formats rarely include indexed cited references, the method is even less likely to identify them. This limitation means that the proposed method is even more prone to publication bias than the traditional method, meaning that its ability to identify descriptions of negative results is minimal. In addition, studies published in journals not indexed by the Web of Science were rarely identified by the proposed method. This means that certain kinds of studies, especially studies published in non-Western journals like the traditional Chinese medicine review excluded from this analysis, will rarely be identified by this method. Although using Scopus instead of, or in addition to, the Web of Science may be able to identify some of these publications, there are still a large number of journals not indexed by either database. The proposed method would be unlikely to retrieve studies published in such journals. Because of these limitations, it is unlikely that the proposed method could ever replace text-based searching for systematic reviews.

Although the method as implemented here performed relatively well, I made several arbitrary choices in implementing it in this analysis. My decision to use two key studies on which to base the method was one of these arbitrary choices.

Other arbitrary choices in this method were the thresholds I used to filter the cociting and cocited articles.

Limitations stated by the reviewer(s): 
No additional limitations were detected by the reviewers.
Study Type: 
Single study

Appraisal of: "Sampson M. Complementary approaches to searching MEDLINE may be sufficient for updating existing systematic reviews. J Clin Epidemiol 2016; 78: 108-115."

Short description: 

The aim of the article was to test whether the combined approach of a focused Boolean search paired with a second search using the similar articles feature of PubMed or support vector machine (SVM) can yield high recall with reasonable precision.

The general approach of a Boolean plus a ranking search is effective in MEDLINE retrieval for systematic reviews. Very high levels of identification of relevant MEDLINE records, with adequate precision, are possible using a focused Boolean search complemented by a document similarity or ranking method.

Limitations stated by the author(s): 

There are two limitations to our proposed strategy. Other databases should be searched in the unusual event that numerous studies, representing more than a small proportion of the total N, are not included in MEDLINE. Second, when it is important to find articles too new to be indexed by MEDLINE, systematic reviewers may wish to conduct a simple PubMed search limited to the nonindexed subsets.

Limitations stated by the reviewer(s): 
It is not possible to reproduce or use the support vector machine functionality.
Study Type: 
Single study

Appraisal of: Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Info Lib J. 2010; 27(2): 114-122.

Short description: 

A conventional subject search was undertaken as the principal method of identifying the literature for this review. Four supplementary search methods were used including citation searching, reference list checking, contact with experts and pearl growing.

11 of  41 references were identified via citation searching, reference list checking and contact with experts (27%). Pearl growing was suspended as the nominated pearls were dispersed across numerous databases, with no single database indexing more than four pearls. Citation searching identified 3 (5 ?) references, reference list checking 4, and contact with experts 2 additional references. Problems in indexing, ambiguity of terms and limited abstract content resulted in reduced sensitivity for the main subject search.

Limitations stated by the author(s): 

No limitations were stated by the authors.

Limitations stated by the reviewer(s): 
Generalisabilty may be limited as the information retrieval of only one systematic review was assessed.
Study Type: 
Single study

Appraisal of: Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 2005; 331(7524): 1064-1065.

Short description: 

To describe where papers come from in a systematic review of complex evidence. Only 30% of sources were obtained from the protocol defined at the outset of the study (that is, from the database and hand searches). Fifty one per cent were identified by “snowballing” (such as scanning reference lists (44%) and citation tracking (7%)), and 24% by personal knowledge or personal contacts.

Limitations stated by the author(s): 

No limitations were stated by the authors.

Limitations stated by the reviewer(s): 
The article assesses the total contribution of the different sources but it is not possible to identify the additional references which could only be found by the additional search techniques. Besides that it is not mentioned if perhaps the original Boolean search was inadequate as the searches and the results are not discussed in detail. The generalizability of the results is also questionable because the authors results are based only on one review (topic “diffusion of service-level innovations in healthcare organizations”).
Study Type: 
Single study

Appraisal of: Hinde S, Spackman E, Claxton K, Sculpher M J. The cost-effectiveness threshold: the results of a novel literature review method. Value in Health 2011;14:A354.

Short description: 
This research is reported as a conference abstract. The authors report a comparison of searching for papers on the cost-effectiveness threshold using key terms with Medical Subject Headings (MeSH) and pearl growing from a pool of relevant papers. Searching using a search strategy identified 34 papers of which 17 were relevant. Pearl growing identified 76 relevant papers including the 17 already identified. Pearl growing identifies papers and chapters not indexed elsewhere.
Limitations stated by the author(s): 
The existing software (unspecified) limits the searching.
Limitations stated by the reviewer(s): 
The limited space provided in a conference abstract does not offer the authors the opportunity to describe the databases they searched and how they conducted the pearl growing. Journal databases do not index reports or books or book chapters so searches limited to journals would not be comparable to the pearl growing approach. The authors do not report the time taken for the two approaches and the number of irrelevant references rejected through the pearl growing.
Study Type: 
Single study
Related Chapters: 
Syndicate content