Appraisal of: "Arber M, Garcia S, Veale T, Edwards M, Shaw A, Glanville JM. Performance of Ovid MEDLINE search filters to identify health state utility studies. Int J Technol Assess Health Care. 2017 Jan;33(4);472-80."

Reviewer(s): 
Short description: 

The retrieval of studies that report health state utility values (HSUVs) is an important aspect of information retrieval for HTA and economic model production.  This study first assessed three MEDLINE search filters designed by the York Health Economic Consortium (YHEC) to identify studies reporting HSUVs. The relative recall method was used to test the sensitivity of each filter. Three quasi gold standard (QSG) sets of relevant studies were compiled from reviews of studies reporting HSUVs. The first QSG (consisting of 294 records) was used to assess the performance of the three initial filters. The best performing of the three filters was further developed using the second QSG (139 records). Ultimately, three final search filters were validated using the third QSG (139 records). The first final search filter is sensitivity maximizing, with 95% sensitivity and a number needed to read (NNR) of 842. The second filter balances sensitivity and precision, and has a 92% sensitivity with an NNR of 502. The third filter is precision maximizing, with 88% sensitivity and an NNR of 383. Real world volume of retrieved records was also tested to illustrate the impact of using the three filters for three example health conditions. Having a range of sensitivity and precision options allows researchers to choose filters based on their search requirements. The authors believe that these are the first validated filters for retrieving HSURs. Search strategies for all three final filters are presented in the article.

Limitations stated by the author(s): 

The robustness of the relative recall method depends on the quality of the searches within each systematic review. Despite passing a quality checklist, some of the systematic reviews employed less than optimal search methods, which may have negatively affected study identification.

Also, a larger quasi-gold standard set may have improved the generalizability of the study findings. During filter development, several terms (utility loss, disutility, short form, SF-12) were removed to increase precision. However, during final validation, it was found that inclusion of these search terms would have retrieved an additional three records, revealing the difficulty in making filter decisions based on a small number of relevant records. These removed terms could be added to the filters to maximize sensitivity. Similarly, some terms added to the filters resulted from an analysis of a very small number of records. These revisions were made to increase sensitivity, but the missed records retrieved through these additions were outliers, and less likely to be representative of the wider body of relevant results. The addition of these terms may increase search result size, but potentially have minimal impact on the total number of relevant results retrieved.

The initial filters created by YHEC were developed through pragmatic and traditional subjective methods, rather than using objective filter design methods. Despite the pragmatic development, these strategies performed well in testing.

The final filters are designed to find reviews, cost-utility analyses and utility elicitation studies that report HSUVs, as well utility mapping studies. Since the filters do not distinguish between these different types of research, precision for research seeking only utility elicitation studies, for example, will be lower than the results reported in the article.

 

 

Limitations stated by the reviewer(s): 
These filters have been validated only for Ovid MEDLINE searches. Translations of these filters to other databases may not produce search results with similar sensitivity or NNR.
Study Type: 
Single study