Appraisal of: Relevo R, Paynter R. Peer Review of Search Strategies [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (AHRQ) (US); 2012 Jun. Available from: http://www.ncbi.nlm.nih.gov/books/NBK98353/

Reviewer(s): 
Carol Lefebvre
Steven Duffy
Full Reference: 
Peer Review of Search Strategies. Methods Research Report. (Prepared by the Oregon Evidence-based Practice Center under Contract No. 290-2007-100572.)
Short description: 

This study was an evaluation of the feasibility of instituting a peer review process for search strategies in HTA organizations.  The aims of the study were to:

1. evaluate whether the PRESS instrument (4) or no-instrument (‘free-form’ evaluations) was preferred by Technical Expert Peer Reviewers (TEPRs) of search strategies.

2. evaluate the usefulness of a peer review process for database search strategies, that is, did peer reviews change search strategies.

3. evaluate the costs of implementing a formal peer review of search strategies programme as a part of the review process.

The investigators undertook a randomized controlled trial of peer review using the PRESS instrument versus “free-form” review of search strategies, at the publication of the research protocol phase of AHRQ reports, by predominantly qualified and experienced information specialists.  Of those participants randomized to the PRESS intervention (20/25) only 11 went on to complete the study and use the PRESS instrument.  Of these, most took less than 2 hours to complete the review using PRESS (91%), some thought it could be incorporated into their workflow (yes: 46%; maybe 46%); some would have been willing to take on peer review duties on a more permanent basis (yes: 37%; maybe 37%); most found the instrument helpful (82%); 54% indicated that they preferred having the checklist but not being required to use it and 27% said that they preferred the PRESS instrument and being required to use it.  The PRESS instrument reviews contained more recommendations overall and had more comments that could be termed error detection.  

Limitations stated by the author(s): 

There was significant dropout in the study (about a third of the peer reviewers failed to complete all reviews, in an already small sample).  The sample of original searchers was also small.  Some of the wording in the questionnaire was ambiguous.  The project focused on pharmacological treatment topics, as they were the more common type of effectiveness review within the AHRQ, and it was unclear whether these results would be generalizable to more diffuse topics.

Limitations stated by the reviewer(s): 
In the Results section the Authors state that “In 97% of cases, the original searcher indicated that the comments did not cause them to alter their search strategies. However, it should be noted that only one original searcher indicated that this was because he or she disagreed with the review. In nearly all other cases, the reason given was simply that the report had already gone forward, and it would be too late to incorporate any changes suggested.” This is not elaborated upon further in the Limitations section of the report.
Study Type: 
Single study
Tags:
  • Share this