Economics

Scope: this includes cost-effectiveness studies, costs and costing studies and utilities

Appraisal of: " Luhnen M, Prediger B, Neugebauer EA, Mathes T. Systematic reviews of health economic evaluations: A structured analysis of characteristics and methods applied. Res Syn Methods. 2019:10;2:195-206."

Short description: 

The review investigates the characteristics and methods, including literature searches, used in 202 systematic reviews of health economic evaluations. A median of 4 electronic databases were searched (range: 1 - 16). The systematic reviewers searched general medical databases (MEDLINE and Embase), 56% of the reviews searched specific economic databases and 33% searched specific systematic review and health technology assessment databases. In terms of the search strategies, the majority of authors did not use published filters and at least half of SRs of economic evaluations applied a limit to the literature search, mostly time limits. 

Limitations stated by the author(s): 

The study authors highlighted that they had not extracted all the data in duplicate and also highlighted that they had only searched MEDLINE to retrieve SRs of economic evaluations and that this may have impacted on the completeness of their search results. However, they suggested that this had probably not negatively affected the findings, as the paper is a representative sample of current SR-HEs and not a complete overview.

Limitations stated by the reviewer(s): 
The review authors stated that ‘the majority of authors did not use filters’, but do not specify the percentage or what filters were used by the authors that did use them. The review authors also do not specify what other limits were used. The authors state that economics search terms were used, but again they do not discuss this in detail.
Study Type: 
Review
Related Chapters: 

Appraisal of: "Gansen FM. Health economic evaluations based on routine data in Germany: a systematic review. BMC Health Serv Res 2018 Apr 10;18(1):268."

Short description: 

This paper investigates health economists’ use of routine data (routine data being defined as “electronically documented information which is generated in the process of administration, provision of services or reimbursement”) by reviewing practice reported in 35 economic evaluations.  In the studies routine data were typically obtained from health insurance funds or other reimbursement data sources rather than from  bibliographic databases. In future, economists may increasingly need to include data from non-traditional bibliographic sources.

 

Limitations stated by the author(s): 

Appraisal of the studies’ quality was restricted to quality of reporting. The authors of studies included in the review did not specify that their studies were economic evaluations.  The review authors categorised the studies as economic evaluations for the purpose of the review.

Limitations stated by the reviewer(s): 
The review does not provide information on how to identify health insurance funds or their databases, nor how to search or query those resources. The study was undertaken from the perspective of Germany, and may not be generalisable to other countries.
Study Type: 
Review

Appraisal of: "Luhnen M, Prediger B, Neugebauer EAM, Mathes T. Systematic reviews of economic evaluations in health technology assessment: a review of characteristics and applied methods. Int J Technol Assess Health Care. 2018(34);6:537–546."

Short description: 

The review investigates the literature searches used in 83 systematic reviews of economic evaluations. A median of 4 electronic databases were searched (range: 1 - 14). The systematic reviewers searched general medical databases (MEDLINE and Embase) and 60% of the reviews also searched specific economic evaluation and health technology assessment databases. In terms of the search strategies Luhnen et al report that economic terms were included in 55% of the literature searches and only 7% reported using a search filter.  48% of reports developed their own Boolean search string with economic evaluations. The authors also state that ‘in most reports’ a limitation was applied such as English language. Additional searches were also reported for online searches (49%), although this was not described further, and reference lists (63%).

Limitations stated by the author(s): 

The study authors highlighted that their definition of a health technology assessment was quite broad but therefore should be inclusive. The authors noted that they may have included reports which were not explicitly intended to be systematic reviews of economic evaluations. Not all data were extracted in duplicate so there may be some errors as a result.

Limitations stated by the reviewer(s): 
The authors stated that 55% of the searches included economic terms. We suspect that some of the other 45% of the searches were in economic databases where economic search terms would not be required. No further details were provided to explain what ‘online searches’ might be.
Study Type: 
Review

Appraisal of: "Arber M, Glanville J, Isojarvi J, Baragula E, Edwards M, Shaw A, Wood H. Which databases should be used to identify studies for systematic reviews of economic evaluations? Int J Technol Assess Health Care. 2018 Jan;34(6):547-54."

Reviewer(s): 
Short description: 

NHS EED and HEED, two key databases for retrieving health economics information, have closed. Based on this changed landscape, this article assesses which databases are now the best sources of information for retrieving economic evaluations to inform systematic reviews. The authors built a quasi-gold standard database of 351 records compiled from 46 systematic reviews of economic evaluations. Nine databases were searched for each record. Embase had the highest yield (89%), followed by Scopus (84%) and MEDLINE and PubMed (both 81%). The HTA database identified the highest number of unique citations (13/351). Embase also uniquely identified two conference abstracts, an important consideration if this type of material is eligible for inclusion in a review. All nine database combined retrieved 337/351 (96%) records. Searching a combination of Embase, the HTA database and either PubMed or MEDLINE identified 95% of the quasi-gold standard records (333/351). The authors concluded that searching additional database outside the core group may be inefficient because of limited incremental yield. Searchers should not rely on PubMed or MEDLINE alone. Searching a multi-disciplinary database may also be useful, especially for non-clinical or public health topics. The authors conclude that searchers should focus on developing suitable search strategies in these key databases to ensure high sensitivity and adequate precision. Supplementary search techniques such as grey literature searching may be more efficient than searching a larger number of databases, as 14/351 (4%) citations were not identified in any of the databases.

MEDLINE search strategies reported in source systematic reviews were also assessed. 10/29 (34.5%) of re-run search strategies missed at least one of the included records found in MEDLINE (with 25 citations missed in total). Weaknesses in the population or intervention concepts, rather than the economics concept, were identified as negatively impacting search retrieval.

Limitations stated by the author(s): 

Candidate reviews were screened by one reviewer only, unless a clear inclusion decision could not be made.

Researchers may have access to more than the nine databases chosen for this review. Other databases were not tested.

The quasi-gold standard database used in this project was compiled from reports commissioned or carried out by a single entity (NICE) and comprised 351 economic evaluations. A larger reference set that included evaluations from more than one entity would have increased generalizability of the findings. Reports focused on clinical medicine or public health in the context of the United Kingdom, so may be less generalizable for other topics such as mental health or health management, or for healthcare in low- and middle-income countries.

The quality of the searches conducted by the systematic reviews from which the quasi-gold standard database was harvested was not assessed. It was assumed that the searches conducted were of sufficient quality to provide a representative set of economic evaluations. Weaknesses in these search strategies could have failed to retrieve relevant economic evaluations, perhaps reducing the representativeness of the quasi-gold standard.

“Efficient” was defined as the fewest number of databases that could be combined to retrieve that largest number of citations. Other types of efficiency such as the time needed to search a database or the number of irrelevant citations retrieved were not considered. Many databases are available on multiple platforms which can impact retrieval and efficiency.

For the evaluation of MEDLINE searches, the sensitivity of each of review’s search strategy in MEDLINE was used as a proxy for search quality. There are other methods available to assess the quality of searches, such as the PRESS checklist, or how the search strategy performs when translated to other databases.

Limitations stated by the reviewer(s): 
While not a particular limitation of this study, the authors do note that the HTA Database is not currently being updated, which may hamper retrieval of recently published studies from the producers of HTA. Until the database is updated, searchers should identify this material by searching websites of individual HTA agencies, or through a general web search. The authors do not mention which databases were searched in the reports used to generate the quasi-gold standard. Since all economic evaluations were drawn from a single organization (NICE), it is possible that they share similar search methodologies, and, thus may search a standard list of databases. If not all of the nine databases were searched for these reports, it may have biased estimates of database yield.
Study Type: 
Single study

Appraisal of: "Brazier J, Ara R, Azzabi I, Busschbach J, Chevrou-Séverac H, Crawford B, Cruz L, Karnon J, Lloyd A, Paisley S, Pickard AS. Identification, review, and use of health state utilities in cost-effectiveness models..."

Short description: 

This report detail the search methods for identifying HSUs in the literature. It describes using iterative searching for identifying HSUs following the process of completing a scoping search. The paper states that the search required for addressing the full range of evidence for a cost-effectiveness model will differ from standard systematic review searches. The paper lists factors to consider when running these iterative searches. Advice is also included for the extent of the searching required and useful search tools.

Limitations stated by the author(s): 

None

Limitations stated by the reviewer(s): 
None

Appraisal of: "Dakin H, Abel L, Burns R, Yang Y. Review and critical appraisal of studies mapping from quality of life or clinical measures to EQ-5D: an online database and application of the MAPS statement. Health Qual Life Outcomes. 2018;16:202."

Short description: 

This paper presents an updated systematic review of studies reporting mapping algorithms from clinical or patient reported outcome measures to EQ-5D-3L or EQ-5D-5L, as well as an assessment of how far mapping studies conform to the MAPS reporting statement.

The review identified 144 mapping studies reporting 190 algorithms mapping from 110 different instruments onto EQ-5D. The 15/17 studies published in 2016 had low adherence to the MAPS checklist.

The mapping studies can be accessed in the Health Economics Research Centre (HERC) Database of Mapping Studies, available at https://www.herc.ox.ac.uk/downloads/herc-database-of-mapping-studies. This database was created in 2013 and is focused on studies mapping to EQ-5D.

Limitations stated by the author(s): 

The review did not include a search of EMBASE. Single reviewer data extraction was undertaken. Only 2016 studies were assessed against the MAPS criteria.

Limitations stated by the reviewer(s): 
The authors do not describe how frequently they update the database.
Study Type: 
Single study

Appraisal of: "Arber M, Wood H, Isojarvi J, Glanville J. Which information sources should be used to identify studies for systematic reviews of economic evaluations in healthcare? Value Health. 2017 Oct/Nov;20(9):A738. Abstract PRM46."

Reviewer(s): 
Short description: 

NHS EED and HEED, two key databases for retrieving health economics information, have closed. Based on this changed landscape, this abstract assesses which database are now the best sources of information for retrieving economic evaluations for models and systematic reviews. The authors built a quasi-gold standard database of 351 records compiled from 46 systematic reviews of economic evaluations. Nine databases were searched for each record. Embase had the highest yield (0.89), followed by Scopus (0.84) and MEDLINE and PubMed (both 0.81). The HTA database identified the highest number of unique citations (13/351). All nine database combined retrieved 337/351 (0.96) records. The authors conclude that for most systematic reviews, Embase, the HTA database and either PubMed or MEDLINE are likely sufficient to identify economic evaluations found in bibliographic databases. Searching a multi-disciplinary database may also be useful, especially in non-clinical topics. Supplementary search techniques may be more efficient than searching a larger number of databases.

MEDLINE search strategies reported in source systematic reviews were also assessed. 10/29 (34.5%) of re-run search strategies missed at least one of the included records found in MEDLINE (with 25 citations missed in total). Weaknesses in the population or intervention concepts, rather than the economics concept, were identified as negatively impacting search retrieval.

Limitations stated by the author(s): 

None stated.

Limitations stated by the reviewer(s): 
Since the results were published only in abstract form, very limited information was presented on methods, results and conclusions.
Study Type: 
Single study

Appraisal of: "Arber M, Garcia S, Veale T, Edwards M, Shaw A, Glanville JM. Performance of Ovid MEDLINE search filters to identify health state utility studies. Int J Technol Assess Health Care. 2017 Jan;33(4);472-80."

Reviewer(s): 
Short description: 

The retrieval of studies that report health state utility values (HSUVs) is an important aspect of information retrieval for HTA and economic model production.  This study first assessed three MEDLINE search filters designed by the York Health Economic Consortium (YHEC) to identify studies reporting HSUVs. The relative recall method was used to test the sensitivity of each filter. Three quasi gold standard (QSG) sets of relevant studies were compiled from reviews of studies reporting HSUVs. The first QSG (consisting of 294 records) was used to assess the performance of the three initial filters. The best performing of the three filters was further developed using the second QSG (139 records). Ultimately, three final search filters were validated using the third QSG (139 records). The first final search filter is sensitivity maximizing, with 95% sensitivity and a number needed to read (NNR) of 842. The second filter balances sensitivity and precision, and has a 92% sensitivity with an NNR of 502. The third filter is precision maximizing, with 88% sensitivity and an NNR of 383. Real world volume of retrieved records was also tested to illustrate the impact of using the three filters for three example health conditions. Having a range of sensitivity and precision options allows researchers to choose filters based on their search requirements. The authors believe that these are the first validated filters for retrieving HSURs. Search strategies for all three final filters are presented in the article.

Limitations stated by the author(s): 

The robustness of the relative recall method depends on the quality of the searches within each systematic review. Despite passing a quality checklist, some of the systematic reviews employed less than optimal search methods, which may have negatively affected study identification.

Also, a larger quasi-gold standard set may have improved the generalizability of the study findings. During filter development, several terms (utility loss, disutility, short form, SF-12) were removed to increase precision. However, during final validation, it was found that inclusion of these search terms would have retrieved an additional three records, revealing the difficulty in making filter decisions based on a small number of relevant records. These removed terms could be added to the filters to maximize sensitivity. Similarly, some terms added to the filters resulted from an analysis of a very small number of records. These revisions were made to increase sensitivity, but the missed records retrieved through these additions were outliers, and less likely to be representative of the wider body of relevant results. The addition of these terms may increase search result size, but potentially have minimal impact on the total number of relevant results retrieved.

The initial filters created by YHEC were developed through pragmatic and traditional subjective methods, rather than using objective filter design methods. Despite the pragmatic development, these strategies performed well in testing.

The final filters are designed to find reviews, cost-utility analyses and utility elicitation studies that report HSUVs, as well utility mapping studies. Since the filters do not distinguish between these different types of research, precision for research seeking only utility elicitation studies, for example, will be lower than the results reported in the article.

 

 

Limitations stated by the reviewer(s): 
These filters have been validated only for Ovid MEDLINE searches. Translations of these filters to other databases may not produce search results with similar sensitivity or NNR.
Study Type: 
Single study

Appraisal of: "Ara R, Brazier J, Peasgood T, Paisley S. The identification, review and synthesis of health state utility values from the literature. Pharmacoeconomics. 2017;35:43–55."

Short description: 

This paper is a guidance document providing an overview of how health state utility values can be identified, reviewed and synthesised when conducting systematic reviews. The paper includes a case study of a review in osteoporosis-related conditions.

In terms of study identification, the authors note that a range of study designs could be relevant and that a number of instruments could be required including condition-specific preference based measures or generic preference based measures. They recommend that a variety of resources and methods should be used to identify studies. As well as electronic databases, searchers should also look at reference lists, conduct key author and citation searches and contact experts.

The authors recommend caution in using filters too early in the search process, to avoid missing potentially relevant studies. The authors also note the absence of dedicated subject headings within MeSH and EMTREE, and that although general subject headings such as ‘Quality of life’ will yield relevant studies, they are likely to demonstrate poor precision. Free text terms should be included in searches and are categorised as general terms (such as QALY), instrument specific terms (such as EQ-5D) and terms describing methods of utility elicitation such as standard gamble.

Limitations stated by the author(s): 

The authors discuss the issues of searching for utility studies including current limitations. They do not discuss limitations of their review and do not provide details of their methods in this publication.

Limitations stated by the reviewer(s): 
The overview and recommendations in relation to searching do not seem controversial.
Study Type: 
Single study

Appraisal of: Shannon M. Sullivan, Kate Tsiplova & Wendy J. Ungar. A scoping review of pediatric economic evaluation 1980-2014: do trends over time reflect changing priorities in evaluation methods and childhood disease? Expert Rev Pharmacoecon Outcomes R

Short description: 

This is a scoping review of the volume, economic analytic techniques and therapeutic areas encompassed by the pediatric economic evaluation studies published between 1980 and 2014.  The review uses records from the Paediatric Economic Database Evaluation (PEDE) database of pediatric economic evaluations to explore trends in analytical techniques, target age groups, interventions, outcomes and disease classifications. The paper describes the eligibility criteria for PEDE. PEDE is populated through searches of  MEDLINE, CINAHL, EMBASE, International Pharmaceutical Abstracts (IPA), EconLit, the Cochrane Library, NHS EED and ERIC, as well as searches of over 73 websites of HTA agencies and research groups. PEDE is updated annually.

Limitations stated by the author(s): 

Studies might have been missed for inclusion in PEDE if adolescents were included in adult populations.  Non-English language papers are not eligible unless they have an English abstract.

Limitations stated by the reviewer(s): 
No other limitations were detected by the reviewers.
Syndicate content