Hand searching

Scope: use to index papers providing explorations of the techniques and methods of handsearching and the value of handsearching for the HTA process

Appraisal of: "Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev, 5(1). doi:10.1186/s13643-016-0249-x"

Reviewer(s): 
Short description: 

This review provides a systematic overview of the available published evidence of searching methods to inform qualitative evidence synthesis (QES). The author sought to assess and identify:

1) the current state of knowledge in relation to searching for qualitative evidence

2) the robustness of the evidence base

3) research gaps and future priorities.

The studies were obtained from the Reference Manager database of the Cochrane Qualitative Methods Group’s study register, of which the author is responsible for updating and maintaining. Supplementary citation searches via Google Scholar was also carried out for 15 key papers. 113 studies were assessed for inclusion. Quality assessment of the included studies was not deemed feasible due to a large proportion of the included studies providing only narrative findings, the lack of a common appraisal instrument and the high levels of heterogeneity across the remaining studies.

The evidence underpinning systematic approaches to searching for qualitative evidence is classified and summarized within one or more of eight headings/ “7 S structure/ framework”: overviews and methodological guidelines, sampling, sources, structured questions, search procedures, search strategies and filters, supplementary strategies and standards. The author summarizes the available evidence and key issues within each section and makes recommendations for further empirical research. Table 7 breaks down the key starting principles in reference to the “7S structure” of searching to inform qualitative evidence synthesis to inform future guidance and Table 8 provides an overview of research priorities.

The review concludes that there is a lack of empirical data to inform information retrieval for QES and that the strength of the evidence is weak and largely based on personal/ professional experience and case studies. Advances have been made in reporting QES, however, validated standards are lacking. 

Limitations stated by the author(s): 

The author states that in order for studies to be included, the references needed to include terms specifically related to searching or retrieval in their titles or abstracts, cite a number of key texts, or be referred to from previously identified items. The full-text of all papers reporting QES were not examined. There is a possibility that potentially relevant reviews reporting emerging information retrieval methods that were not reported in the title or abstract were missed. However, these risks are offset by the sensitive search approach and the currency and comprehensiveness of the Cochrane Qualitative Methods Group study register. The author also notes that some papers were excluded as they did not distinguish between qualitative and quantitative approaches, which could potentially be useful for mixed methods reviews.

Limitations stated by the reviewer(s): 
1. This was a single reviewer methodology review so judgements on eligibility and interpretations of potential significance and contribution of individual studies may not be consistent and/or reproducible. [Study Selection Bias] 2. Although as a methodology review there is no formal requirement to follow PRISMA reporting standards this review may have benefited from more complete and transparent reporting. [Incomplete Reporting Bias] 3. As the reviewer was author on a high proportion of included studies this may have consciously impacted on study identification and subconsciously on study selection and interpretation. [Citation Bias; Observer Bias]
Study Type: 
Single study

Appraisal of: Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 2007; (2): MR000001.

Short description: 

Aim of the Cochrane review was to compare the technique of handsearching with electronic searching to identify RCT.

Thirty-four studies were included. Handsearching identified between 92% to 100% of the total number of reports of randomized trials found in the various comparisons in this review. Searching MEDLINE retrieved 55%, EMBASE 49% and PyscINFO 67%. The retrieval rate of the electronic database varied depending on the complexity of the search. The Cochrane Highly Sensitive Search Strategy (HSSS) identified 80% of the total number of reports of randomized trials.

Conclusion of the authors is, that hand searching has a valuable role in identifying reports of RCT for inclusion in SR of health care interventions.

Particularly important in identifying trials reported as abstracts, letters and those published in languages other than English.

Limitations stated by the author(s): 

In this review, no attempts were made to assess the importance of the trials missed by either handsearching or electronic searching. There is evidence to suggest that only around half of all trials reported as abstracts are subsequently published in full and that published trials may show a larger treatment effect than ’grey’ trials (for example those published as conference abstracts). Further research is needed to assess the importance of those trials missed by either method of searching.

Limitations stated by the reviewer(s): 
Main problem is, that the review is outdated. The last search was in 2002. Handsearch was mainly compared to “old” 3 phase Highly Sensitive Search Strategy filter.
Study Type: 
Review

Appraisal of: Glanville J, Cikalo M, Crawford F, Dozier M, McIntosh H. Handsearching did not yield additional unique FDG-PET diagnostic test accuracy studies compared with electronic searches: a preliminary investigation. Res Synth Methods. 2012; 3(3): 20

Short description: 

The aim of the investigation was to explore the role and value of handsearching in the identification of diagnostic test accuracy (DTA studies) of PET-CT for colorectal cancer. 12 journals were handsearched for reports of DTA studies. A total of 1147 potential reports of DTA studies were identified. No papers met the systematic review inclusion criteria.

 

Limitations stated by the author(s): 

It is possible that the six seed reviews selected to identify candidate journals to handsearch may not have been the most relevant reviews because no reviews of PET-CT were identified and the reviews explored a range of imaging modalities in CRC. However, subsequent investigation of the journal frequency analysis of the three most relevant seed reviews had little impact on the order of candidate journals in the overall frequency table or on the journals that yieldedmost papers for consideration in the Brush et al. (2011) review following handsearching. In neither the revised (three seed review) nor original (six seed review) frequency list did the Journal of Nuclear Medicine or the European Journal of Nuclear Medicine andMolecular Imaging appear as high yield journals, despite yielding the highest number of included studies for the Brush et al. (2011) systematic review.

This study encompassed the handsearch of 12 journals for 4.5 years. As noted earlier, there are many more journals that have included reports of FDG PET-CT diagnostic studies in CRC.

The analysis of the papers used to construct the journal frequency list by publication year and broad journal discipline category showed no clear trends in the publication of imaging studies in CRC. The journal discipline categories were broad and pragmatic, with no overlap between categories allowed. This may not reflect the exact nature or encompass the true diversity of the journals. This suggests that searches (handsearch or electronic) for DTA studies for imaging CRC cannot easily be limited to specific journal disciplines and means that identifying core journals to handsearch in this topic is problematic.

There were some resource constraints to this project that meant that the handsearch was not comprehensive in terms of searching journal special issues reporting on workshops, meetings and conferences. These issues often included high volumes (in the order of thousands) of abstracts of oral and poster contributions. Where conference abstracts were grouped by session theme, only the relevant groups of abstracts were handsearched: those sessions on CRC or imaging FDG-PET. This means that relevant abstracts may have been missed if they were categorized into different sessions than those searched; non-FDG-PET DTA conference abstracts in other sessions will also not have been captured. In the cases where conference proceedings failed to group the abstracts by session theme, it was often impractical to assess individual abstracts, and again, potentially relevant studies may have been missed.

Limitations stated by the reviewer(s): 
No limitations were detected by the reviewers.
Study Type: 
Single study

Appraisal of: Armstrong R, Jackson N, Doyle J, Waters E, Howes F. It's in your hands: the value of handsearching in conducting systematic reviews of public health interventions. J Public Health (Oxf). 2005; 27(4): 388-391.

Short description: 

Aim of the study was to identify value of handsearching in conducting systematic reviews of public health interventions. 6 journals with the highest yield of RCTs and CCTs in Central and not currently being handsearch by the Cochrane Collaboration were prioritized to be handsearched. All journals´ issues (including supplements) published in 2003 and 2004 were handsearched. A total of 92 issues were handsearched. Of the 131 RCTs and CCTs identifierd by handsearching, 125 would have been identified through a MELDINE search using the PICO framework. If the search included study design limits (PICOt), a further 7 trials would have been missed. The greatest value of handsearching was found to be in supplement editions and abstract sections of journals.

Limitations stated by the author(s): 

No limitations were stated by the authors.

Limitations stated by the reviewer(s): 
It is not clear if the relevant studies identified by hand search were indexed in Medline (2 trials were abstracts only, 3 studies were identified in supplement editions of journals which were not indexed in MEDLINE, 1 trial could not be found in MEDLINE (the entire issue appeared to be missing from MEDLINE). The study did not compare hand searching with a Boolean search or a validated RCT filter. It was only stated that 7 trials provided no information in the abstract, title or subject headings of the random allocation of interventions (randomization was mentioned in the methods section of the article) – meaning study type was not mentioned.
Study Type: 
Single study

Appraisal of: Mattioli S, Farioli A, Cooke RMT, Baldasseroni A, Ruotsalainen J, Placidi D, et al. Hidden effectiveness? Results of hand-searching Italian language journals for occupational health interventions. Occup Environ Med 2012;69(7):522-524.

Short description: 

The study compared the yield of hand-searching with optimised electronic search strategies in retrieving occupational health (OH) intervention studies published in a language other than English.

The authors systematically hand-searched and screened reports of OH intervention studies published in Italian in peer-reviewed scientific journals between 1990 and 2008. The Cochrane Occupational Safety and Health Review Groups (OSHRG) most sensitive search string retrieved all 16 articles published in the Italian language journals that were indexed in MEDLINE.

These findings suggest that a sensitive electronic search strategy may be able to find most of the OH interventions published in languages other than English that are indexed in MEDLINE.

Limitations stated by the author(s): 

No limitations were stated by the authors.

Limitations stated by the reviewer(s): 
The authors hand searched only 3 MEDLINE-indexed journals.
Study Type: 
Single study

Appraisal of: Chapman AL, Morgan LC, Gartlehner G. Semi-automating the manual literature search for systematic reviews increases efficiency. Health Info Libr J. 2009; 27(1):22-7.

Short description: 

Manual literature searching (or hand-searching) for systematic reviews is often promoted as a way of minimizing retrieval bias (the failure to find relevant studies due to the retrieval capacity of electronic databases), but it can be a lengthy and unsystematic process. This study aimed to determine the validity of a new manual search method to check reference lists of identified studies for relevant articles using the Scopus database by comparing it to the traditional method of manually checking reference lists as the gold standard. Outcome measures included accuracy and completeness of article detection (validity) and personnel time involved (efficiency). The Scopus method identified the same studies as the gold standard, but was much more time efficient (3 hours versus 8 hours, time saving of 62.5%). The study authors concluded that the Scopus method could significantly improve the efficiency of manual searches and thus of systematic reviews.

Limitations stated by the author(s): 

Findings are based on one medium-sized systematic review. Further method studies are needed to validate these findings and identify potential improvement to this approach.

 Scopus requires a paid subscription.

Scopus currently does not include the reference lists of Cochrane reviews, which is a major limitation as Cochrane reviews are a major source of trial information.

Limitations stated by the reviewer(s): 
Findings based on reference checking of only 20 studies.
Study Type: 
Single study

Appraisal of: O'Leary N, Tiernan E, Walsh D, Lucey N, Kirkova J, Davis MP. The pitfalls of a systematic MEDLINE review in palliative medicine: symptom assessment instruments. Am J Hosp Palliat Care. 2007; 24(3): 181-184.

Short description: 

The authors describe the usefulness of the related articles link in PubMed as part of a systematic MEDLINE review. Of 1181 citations identified, through a PubMed search, 10 articles met the indusion criteria. as did a further 21 articles identified through hand-searching the references of the 10 articles. The PubMed related articles link of the 10 articles yielded 15% of all reports finally included in the systematic review. The authors suggest that it is a useful tool in PubMed for revietrieving complex evidence.

Limitations stated by the author(s): 

No limitations stated by the authors.

Limitations stated by the reviewer(s): 
Different numbers are described in the result section: references that met the inclusion criteria and references finally included in the systematic review. This selection process is not clearly documented and it is not clear how many references finally included in the systematic review are identified through which source.
Study Type: 
Single study

Appraisal of: Waffenschmidt S, Hausner E, Engel L, Volz F, Kaiser T. Benefit of searching different databases to identify health economic evaluations in German HTA-reports. Abstract presented at: Health Technology Assessment International (HTAi) 7th Annua

Reviewer(s): 
Short description: 
This study assessed the benefits of searching various databases used for identifying health economic evaluations. German language HTA reports that incorporated economic evaluations were consulted. The data sources used in the literature search were analyzed to see where relevant references were retrieved from (standard international, German or health economics databases). The findings showed that 90% of citations were retrieved from searches of MEDLINE, Embase and the Cochrane Library (HTA database). Neither German databases nor health economics databases (such as NHS EED) provided any unique citations. These findings indicate that German HTA reports identify most of their relevant health economics citations through standard international databases. However, if a comprehensive pool of studies is required, searching additional sources beyond bibliographic databases, including hand searching, seems to be required.
Limitations stated by the author(s): 
This is a presentation of preliminary findings.
Limitations stated by the reviewer(s): 
Since results are available only in abstract/poster form, complete study information is not available. The study was limited to German language reports, so results may not be transferable to other countries or languages. We note that health economics databases tend to be populated from MEDLINE or Embase and are therefore unlikely to yield unique references: their value lies in being time saving and value-added.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Golder S, Loke YK. The contribution of different information sources for adverse effects data. Int J Technol Assess Health Care 2012;28(2):133-137.

Short description: 

Research has shown that MEDLINE might not be a comprehensive source on adverse effects information. The aim of this study was to evaluate the contribution of a variety of sources when searching for adverse effects data for a systematic review of thiazolidinedione-related fractures in patients with type 2 diabetes mellitus.

The results showed the potential value of searching a variety of sources for adverse effects data.

In this study, the minimum combination of sources required to identify all the relevant references was:

  • GlaxoSmithKline website,
  • Science Citation Index,
  • Embase,
  • BIOSIS Previews,
  • British Library Direct,
  • Medscape DrugInfo,
  • AHFS First (American Hospital Formulary Service),
  • Thomson Reuters Integrity,
  • Conference Papers Index,
  • Handsearching,
  • Reference checking.
Limitations stated by the author(s): 
The results were based on a single case study and are, not necessarily generalisable. It was also difficult to maintain consistency in search strategies when adapting them to different search interfaces and it was, therefore, difficult to make fair comparisons. It was also not possible to perform a cost analysis of searching each source due to different pricing mechanisms used by the database providers.
Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewer.
Study Type: 
Single study
Related Chapters: 

Comments from the authors:

Syndicate content