Cost-benefit analysis

A method of comparing the cost of a program with its expected benefits in dollars (or other currency). The benefit-to-cost ratio is a measure of total return expected per unit of money spent. This analysis generally excludes consideration of factors that are not measured ultimately in economic terms. Cost effectiveness compares alternative ways to achieve a specific set of results.

Appraisal of: "Arber M, Garcia S, Veale T, Edwards M, Shaw A, Glanville JM. Performance of Ovid MEDLINE search filters to identify health state utility studies. Int J Technol Assess Health Care. 2017 Jan;33(4);472-80."

Reviewer(s): 
Short description: 

The retrieval of studies that report health state utility values (HSUVs) is an important aspect of information retrieval for HTA and economic model production.  This study first assessed three MEDLINE search filters designed by the York Health Economic Consortium (YHEC) to identify studies reporting HSUVs. The relative recall method was used to test the sensitivity of each filter. Three quasi gold standard (QSG) sets of relevant studies were compiled from reviews of studies reporting HSUVs. The first QSG (consisting of 294 records) was used to assess the performance of the three initial filters. The best performing of the three filters was further developed using the second QSG (139 records). Ultimately, three final search filters were validated using the third QSG (139 records). The first final search filter is sensitivity maximizing, with 95% sensitivity and a number needed to read (NNR) of 842. The second filter balances sensitivity and precision, and has a 92% sensitivity with an NNR of 502. The third filter is precision maximizing, with 88% sensitivity and an NNR of 383. Real world volume of retrieved records was also tested to illustrate the impact of using the three filters for three example health conditions. Having a range of sensitivity and precision options allows researchers to choose filters based on their search requirements. The authors believe that these are the first validated filters for retrieving HSURs. Search strategies for all three final filters are presented in the article.

Limitations stated by the author(s): 

The robustness of the relative recall method depends on the quality of the searches within each systematic review. Despite passing a quality checklist, some of the systematic reviews employed less than optimal search methods, which may have negatively affected study identification.

Also, a larger quasi-gold standard set may have improved the generalizability of the study findings. During filter development, several terms (utility loss, disutility, short form, SF-12) were removed to increase precision. However, during final validation, it was found that inclusion of these search terms would have retrieved an additional three records, revealing the difficulty in making filter decisions based on a small number of relevant records. These removed terms could be added to the filters to maximize sensitivity. Similarly, some terms added to the filters resulted from an analysis of a very small number of records. These revisions were made to increase sensitivity, but the missed records retrieved through these additions were outliers, and less likely to be representative of the wider body of relevant results. The addition of these terms may increase search result size, but potentially have minimal impact on the total number of relevant results retrieved.

The initial filters created by YHEC were developed through pragmatic and traditional subjective methods, rather than using objective filter design methods. Despite the pragmatic development, these strategies performed well in testing.

The final filters are designed to find reviews, cost-utility analyses and utility elicitation studies that report HSUVs, as well utility mapping studies. Since the filters do not distinguish between these different types of research, precision for research seeking only utility elicitation studies, for example, will be lower than the results reported in the article.

 

 

Limitations stated by the reviewer(s): 
These filters have been validated only for Ovid MEDLINE searches. Translations of these filters to other databases may not produce search results with similar sensitivity or NNR.
Study Type: 
Single study

Appraisal of: Paisley S. Identification of evidence for key parameters in decision-analytic models of cost effectiveness: a description of sources and a recommended minimum search requirement [epub]. PharmacoEconomics. 2016. DOI 10.1007/s40273-015-0372-x

Short description: 

This paper provides recommendations for the minimum level of searching which should be conducted to populate the various parameters  (e.g. treatment effects, costs, resource use) for a health economic decision model. The minimum searching level varies by parameter and the author suggests that the level should be adjusted depending on the importance of the parameter to the model. The paper describes the range of different resources which might need to be searched for each parameter and also indicates where country or region-specific resources might need to be identified. Specific guidance is given for each parameter and where more detailed guidance is available for search approaches for specific parameters, this guidance is signposted. This paper also makes brief recommendations on how to report searches.

Limitations stated by the author(s): 

The author highlights the developmental nature of the searches undertaken to inform decision model searches since modelling is an “iterative and dynamic process” and that this is difficult to capture in procedural guidance such as this.  The author acknowledges that the recommendations in this guidance have “not been tested empirically” and further research is required to support the recommendations.

Limitations stated by the reviewer(s): 
No additional limitations were detected by the reviewers.
Study Type: 
Single study

Appraisal of: Pitt C, Goodman C, Hanson K. Economic evaluation in global perspective: a bibliometric analysis of the recent literature. Health Econ. 2016 Feb;25 Suppl 1:9-28.

Reviewer(s): 
Short description: 

A bibliometric analysis of full economic evaluations of health interventions was conducted. A search of 14 databases identified 2844 economic evaluations published between 1 January 2012 and 3 May 2014. Analyses included: the sensitivity and specificity of searches in all databases, the distribution of the publications between countries, regions and health areas studied, as well as authors' country and institutional affiliations, journal type, language, and type of economic evaluation. The Scopus search identified the largest number of economic evaluations (85%), followed by NHS EED (80%), and MEDLINE (79%). EconLit captured only 1% of the publications identified. The majority of economic evaluations were published in biomedical journals (74%), addressed high-income countries (83%), and had at least one author based in a high-income country (91%). The health areas studied correlated more closely with the burden of diseases in high income countries compared to low-and middle-income countries. 

Limitations stated by the author(s): 

The NHS EED database is no longer being updated as of March 2015. Wiley HEED ceased to be available at the end of 2014. The authors assigned one point per institution per article in the analysis of institutional affiliation, which gives more weight to articles from multiple institutions. A single article could be classified as belonging to multiple health areas, which could be considered double-counting.

Limitations stated by the reviewer(s): 
The search strategies used to identify economic evaluations were unique strategies created for this study and were not established search filters, so they do not reflect current practice of information specialists. Because the search strategies were very specific, they may have missed relevant studies which were not explicit about being an economic evaluation. Additionally, variations in the search strategies between databases may have affected sensitivity and precision of retrieval. For example, in the cross-disciplinary databases strategies included a concept to limit retrieval to health topics. As such, the results presented might not fully represent the content of the databases. Because of these factors, it may be difficult to draw meaningful conclusions based on the data presented in this article.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Arber M, Garcia S, Veale T, Edwards M, Shaw A, Glanville J. Sensitivity of a search filter designed to identify studies reporting health state utility values [poster].

Reviewer(s): 
Short description: 

The retrieval of studies that report health state utility values (HSUVs) is an important aspect of information retrieval for HTA and economic model production.  The York Health Economic Consortium (YHEC) created and tested three search filters designed to identify studies reporting HSUVs. The relative recall method was used to test the sensitivity of each filter. A quasi-gold standard set of 293 studies was compiled by identifying a sample of systematic reviews of studies reporting HSUVs. All three filters tested retrieved the same 267 citations from the quasi-gold standard (0.91 sensitivity). Filter 3 had the highest precision at 0.003. The 26 studies missed by the filters did not explicitly refer to HSUVs within the citations. Inclusion of the “quality of life” subject heading would have retrieved 20 of those studies, but also increased overall results by 116%. Further analysis of the 26 records missed by the search filters is planned to try to improve sensitivity without a significant loss in precision. The search strategy for the most precise search filter is included in the poster.

Limitations stated by the author(s): 

The retrieval of studies that report health state utility values (HSUVs) is an important aspect of information retrieval for HTA and economic model production.  The York Health Economic Consortium (YHEC) created and tested three search filters designed to identify studies reporting HSUVs. The relative recall method was used to test the sensitivity of each filter. A quasi-gold standard set of 293 studies was compiled by identifying a sample of systematic reviews of studies reporting HSUVs. All three filters tested retrieved the same 267 citations from the quasi-gold standard (0.91 sensitivity). Filter 3 had the highest precision at 0.003. The 26 studies missed by the filters did not explicitly refer to HSUVs within the citations. Inclusion of the “quality of life” subject heading would have retrieved 20 of those studies, but also increased overall results by 116%. Further analysis of the 26 records missed by the search filters is planned to try to improve sensitivity without a significant loss in precision. The search strategy for the most precise search filter is included in the poster.

Limitations stated by the reviewer(s): 
Ovid MEDLINE was the only database searched for this study.
Study Type: 
Single study

Appraisal of: Kaunelis D. When everything is too much: development of a CADTH narrow economic search filter. Poster presented at: 2011 CADTH Symposium. 2011 Apr 3-5; Vancouver, BC.

Reviewer(s): 
Short description: 
This poster reported an investigation of the performance of filters to find a gold standard set of economic evaluations with a focus on maximizing the precision of the filter. The best performing filter was further adapted and then tested for performance on the same gold standard and using a series of queries. The ‘CADTH narrow search filter’ found approximately 75% of the gold standard records with better precision than other filters, which tended to maximize sensitivity. The author suggested that this filter is useful for rapid reviews, to identify critiques of models, and to carry out high level reviews of an intervention or reviews of an indication. The filter mostly comprised title and abstract terms and should therefore be usable across a range of databases. It is available for Ovid and for PubMed interfaces.
Limitations stated by the author(s): 
No limitations were identified.
Limitations stated by the reviewer(s): 
The filter was developed and tested in the same gold standard and so will tend to over-perform. The performance of the filter should be tested in other gold standards to see whether performance from the test set is maintained. The sensitivity and precision of the search filters was not provided.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Dakin H. Review of studies mapping from quality of life or clinical measures to EQ-5D: an online database. Health and Quality of Life Outcomes 2013;11(1):151.

Reviewer(s): 
Short description: 
This paper reported the creation of a database of mapping studies. The mapping studies provided mapping algorithms to convert quality of life instrument values to EQ-5D (EuroQoL) instrument values. Following a search of MEDLINE and three websites, the authors identified ninety mapping studies, which reported eighty mapping algorithms. These were described in the paper. The studies can be searched in a free internet database: http://www.herc.ox.ac.uk/downloads/mappingdatabase. The database also contains references to studies validating mapping algorithms and supplementary information including tools to generate predictions. It is intended that the database will be updated. It is possible to register by email to be informed of updates to the database.
Limitations stated by the author(s): 
The author did not search Embase or the grey literature where mapping may also be reported. Some studies might have been excluded because of lack of information but might have proved to be relevant if the authors had been contacted to provide more information.
Limitations stated by the reviewer(s): 
The author did not report the full search terms and search term combinations used to identify studies from MEDLINE, so it is difficult to know whether the search was fit for purpose.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Coyle KB, Trochlil K, Iversen P. MEDLINE and EMBASE for health economic literature reviews [abstract]. Value Health 2012;15(4):A162.

Reviewer(s): 
Short description: 
The authors compared search results in MEDLINE and Embase for the indication chronic pulmonary obstructive disease (COPD) with regard to cost of illness (COI), burden of illness (BOI) and cost benefit analysis (CBA). MEDLINE retrieved a higher number of references for CBA, while Embase retrieved higher results in the areas of COI and BOI. Conference abstracts accounted for the higher Embase totals. The inclusion of conference abstracts in Embase support the practice of conducting economic literature reviews in both MEDLINE and Embase.
Limitations stated by the author(s): 
None stated - this is an abstract.
Limitations stated by the reviewer(s): 
Since the results were published only in abstract form, very limited information was presented on methods, results and conclusions. Specific search strategies were not provided, nor was it stated if the citations retrieved were evaluated for relevance. Only one subject area (COPD) was searched for citations published during 2010-11, resulting in low retrieval results. Differences in retrieval may therefore not have statistical significance or be generalizable to other topic areas.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Royle P, Waugh N. Literature searching for clinical and cost-effectiveness studies used inhealth technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technol Assess 2003;7(34).

Short description: 
This study assessed a sample of 20 NICE technology assessment reports (TARs) to assess optimum literature retrieval strategies for identifying clinical and cost-effectiveness studies using relative recall methods. All sources searched for the TARs were identified and 130 included cost-effectiveness studies were identified. The presence of the included studies within sources was recorded. 73.1% of studies were published and 1.5% were abstracts. 86.6% of those published/abstract studies were indexed in MEDLINE, 86.6% in Embase and 40.2% in NHS EED. Searching SCI and ASCO Online in addition increased retrieval of published studies to 97.9%. Searching non-database sources an identify further studies: 25% of the included studies were ‘unpublished’ or grey literature.
Limitations stated by the author(s): 
The number of TARs assessed was small and may be biased by the large number of cancer topics. Most of the key measurement decisions were made by only one reviewer. The descriptions of study designs and search accuracy were accepted uncritically. The effect of the harder to find studies on the effect size was not assessed. The search strategies were not assessed for relevance and fitness for purpose: their quality is unknown. The TARs may not be generalisable to the future TAR programme in terms of subject focus.
Limitations stated by the reviewer(s): 
The study was conducted on reviews published in 2000-2002. The relative recall approach adopted gives an assessment of the relative yield of databases and their overlap with a perfect search strategy. The study did not rerun TAR search strategies to investigate how many of the studies which are available to be found, would have been found by the strategies used.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Rosen A B, Greenberg D, Stone P W, Olchanski N V, Neumann P J. Quality of abstracts of papers reporting original cost-effectiveness analyses. Medical Decision Making 2005;25:424-428.

Short description: 
This research examined the quality of reporting of key data elements in journal abstracts of cost-effectiveness analyses (CEA). MEDLINE was searched (1998-2001) using three search terms and HEED was searched for additional records. 303 abstracts (from 150 journals) were assessed. Two independent readers found that 94% gave a clear description of the intervention, 71% gave a description of the comparator, 85% reported the target population and 28% the study perspective. The mean number of elements reported was 2.78. Journals which have CEA-specific abstract reporting requirements, structured abstract requirements or high impact factors had significantly more data in abstracts than journals without those features.
Limitations stated by the author(s): 
The study examines a subset of CEAs reporting $/QALY outcomes and may not reflect the quality of abstract reporting in the broader body of CEAs. The study focused on four data elements and other elements might be important. The authors propose the use of standard descriptors but this may not automatically result in improved abstract quality.
Limitations stated by the reviewer(s): 
The MEDLINE search strategy is quite sparse (three terms) and may not have identified all relevant CEAs. The HEED search is not described in detail so an assessment of its quality cannot be made.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Hinde S, Spackman E, Claxton K, Sculpher M J. The cost-effectiveness threshold: the results of a novel literature review method. Value in Health 2011;14:A354.

Short description: 
This research is reported as a conference abstract. The authors report a comparison of searching for papers on the cost-effectiveness threshold using key terms with Medical Subject Headings (MeSH) and pearl growing from a pool of relevant papers. Searching using a search strategy identified 34 papers of which 17 were relevant. Pearl growing identified 76 relevant papers including the 17 already identified. Pearl growing identifies papers and chapters not indexed elsewhere.
Limitations stated by the author(s): 
The existing software (unspecified) limits the searching.
Limitations stated by the reviewer(s): 
The limited space provided in a conference abstract does not offer the authors the opportunity to describe the databases they searched and how they conducted the pearl growing. Journal databases do not index reports or books or book chapters so searches limited to journals would not be comparable to the pearl growing approach. The authors do not report the time taken for the two approaches and the number of irrelevant references rejected through the pearl growing.
Study Type: 
Single study
Related Chapters: 
Syndicate content