Peer reviewing

Scope: peer review of literature search strategies- how to do it, what to report and evaluations of the impact of peer review, resources for peer review

Appraisal of: Relevo R, Paynter R. Peer Review of Search Strategies [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (AHRQ) (US); 2012 Jun. Available from: http://www.ncbi.nlm.nih.gov/books/NBK98353/

Reviewer(s): 
Short description: 

This study was an evaluation of the feasibility of instituting a peer review process for search strategies in HTA organizations.  The aims of the study were to:

1. evaluate whether the PRESS instrument (4) or no-instrument (‘free-form’ evaluations) was preferred by Technical Expert Peer Reviewers (TEPRs) of search strategies.

2. evaluate the usefulness of a peer review process for database search strategies, that is, did peer reviews change search strategies.

3. evaluate the costs of implementing a formal peer review of search strategies programme as a part of the review process.

The investigators undertook a randomized controlled trial of peer review using the PRESS instrument versus “free-form” review of search strategies, at the publication of the research protocol phase of AHRQ reports, by predominantly qualified and experienced information specialists.  Of those participants randomized to the PRESS intervention (20/25) only 11 went on to complete the study and use the PRESS instrument.  Of these, most took less than 2 hours to complete the review using PRESS (91%), some thought it could be incorporated into their workflow (yes: 46%; maybe 46%); some would have been willing to take on peer review duties on a more permanent basis (yes: 37%; maybe 37%); most found the instrument helpful (82%); 54% indicated that they preferred having the checklist but not being required to use it and 27% said that they preferred the PRESS instrument and being required to use it.  The PRESS instrument reviews contained more recommendations overall and had more comments that could be termed error detection.  

Limitations stated by the author(s): 

There was significant dropout in the study (about a third of the peer reviewers failed to complete all reviews, in an already small sample).  The sample of original searchers was also small.  Some of the wording in the questionnaire was ambiguous.  The project focused on pharmacological treatment topics, as they were the more common type of effectiveness review within the AHRQ, and it was unclear whether these results would be generalizable to more diffuse topics.

Limitations stated by the reviewer(s): 
In the Results section the Authors state that “In 97% of cases, the original searcher indicated that the comments did not cause them to alter their search strategies. However, it should be noted that only one original searcher indicated that this was because he or she disagreed with the review. In nearly all other cases, the reason given was simply that the report had already gone forward, and it would be too late to incorporate any changes suggested.” This is not elaborated upon further in the Limitations section of the report.
Study Type: 
Single study

Appraisal of: McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol 2016;75:40-6.

Reviewer(s): 
Short description: 

This article provides the 'PRESS 2015 Guideline Statement' and 'PRESS checklist' resulting from the update of the 2008-2010 CADTH PRESS project: Peer Review of Electronic Search Strategies (2,3,4). The update entailed a systematic review, web-based survey of experts and consensus development forum to evaluate electronic search strategies in order to consolidate and make adjustments to the original PRESS Evidence Based Checklist (4).

The systematic review  identified no new search elements for addition to the existing PRES5 Evidence-Based Checklist; the web-based survey of experts found that most respondents felt that peer review should be undertaken after the MEDLINE search had been prepared but before it had been translated to other databases; and the consensus development forum found that of the seven original PRESS elements, six should be retained and one should be removed (skilled translation of the search strategy to additional databases – not because it was unimportant but rather because it could not easily be ascertained solely via the peer review process).

Limitations stated by the author(s): 

The systematic review for the update was focused on health science databases only. Grading of strength of recommendations was not done. Piloting of the revised PRESS Guideline was undertaken by only one agency (CADTH). 

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Single study

Appraisal of: McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS – Peer Review of Electronic Search Strategies: 2015 Guideline Explanation and Elaboration (PRESS E&E). Ottawa: CADTH; 2016 Jan.

Reviewer(s): 
Short description: 

This guideline updates and expands upon the 2008 CADTH report PRESS: Peer Review of Electronic Search Strategies (2) and the associated Journal of Clinical Epidemiology article in 2009 (3), as well as the Evidence Based Checklist for the Peer Review of Electronic Search Strategies (PRESS EBC), published in the Evidence Based Library and Information Practice journal in 2010 (4).  A systematic review was conducted, along with a web-based survey and consensus forum meeting, to guide peer review of electronic search strategies in order to consolidate and update the original PRESS Evidence Based Checklist.

Limitations stated by the author(s): 

The systematic review searches for the update were focused on the health science databases only. Grading of strength of recommendations was not done. Piloting of the revised PRESS Guideline was undertaken by only one agency (CADTH).

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Single study

Appraisal of: McGowan J, Sampson M, Lefebvre C. An evidence based checklist for the Peer Review of Electronic Search Strategies (PRESS EBC). Evidence Based Library & Information Practice 2010;5(1):149-154.

Reviewer(s): 
Short description: 

This article presents an evidence based assessment checklist to help evaluate the quality of electronic search strategies used in systematic reviews and health technology assessments. A table explaining each of the elements used in the checklist is included. The methods used to produce this checklist are reported elsewhere. [Sampson M, McGowan J, Lefebvre C, Moher D, Grimshaw J. PRESS: Peer Review of Electronic Search Strategies. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2008; Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol 2009;62(9):944-52.]

Limitations stated by the author(s): 

None reported by the authors.

Limitations stated by the reviewer(s): 
No limitations detected by the reviewers.

Appraisal of: Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol 2009;62(9):944-52.

Reviewer(s): 
Short description: 

The objective of the study was to develop a guideline for the peer review of complex and highly sensitive electronic literature search strategies as used in systematic reviews and health technology assessments. This article describes the evidence-based methods used to develop a guideline through the creation of an annotated checklist.

From a systematic review of library and information retrieval literature, and a survey of experienced information specialists, the study found six elements of strong importance and seven additional elements of partial importance that have an impact on the performance of electronic search strategies. The six important elements were: accurate translation of the research question into search concepts; correct use of Boolean operators; correct combination of line numbers; sufficient translation of the search strategy for use in other databases; inclusion of relevant subject headings; and correct spellings. Using the resultant guideline should reduce errors in search strategies and ultimately improve the quality of systematic reviews and health technology assessments.

The checklist itself, ‘Evidence Based Assessment Form for the Peer Review of Electronic Search Strategies (PRESS EBC)’ is published elsewhere [McGowan J, Sampson M, Lefebvre C. An evidence based checklist for the Peer Review of Electronic Search Strategies (PRESS EBC). Evidence Based Library & Information Practice 2010;5(1):149-154.]

Limitations stated by the author(s): 

Several of the elements designated as being important in the search strategy did not have any supporting research evidence, relying instead on survey response.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Single study
Syndicate content