Safety

Appraisal of: Golder S, Wright K, Rodgers M. Failure or success of search strategies to identify adverse effects of medical devices: a feasibility study using a systematic review. Syst Rev 2014;3:113.

Short description: 

A systematic review of the safety of recombinant human bone morphogenetic protein-2 (rhBMP-2) for spinal fusion was used as a case study to investigate whether adverse effects terms were prevalent in the bibliographic records of the included studies. The aim of the study was to assess whether the development of a search filter for medical devices would be feasible.

Out of the 82 publications (49 studies) included in the systematic review, 51 were indexed in MEDLINE and 55 in Embase. The results showed that 94% of the MEDLINE records and 95% of the Embase records contained at least one adverse effects related term. However, the records contained a wide range of adverse effects terms, and no single term was able to retrieve the majority of the included publications. In addition, the most successful search terms were different from search terms shown to be most successful when searching for adverse drug effects. The study concluded that the development of adverse effects search filters for medical devices is feasible, but that such filters should be developed using a large set of relevant records in order to identify the optimal combination of search terms.

Limitations stated by the author(s): 

The main limitation to this study is that because only one systematic review was used as a case study, the generalisability of the study results is limited. This case study is also of a medical device with a pharmaceutical component, while a case study of a different kind of a medical device might have given different results.

In addition, unlike many other HTAs and systematic review authors, the study authors were able to obtain unpublished data directly from the manufacturer. They also included an unusually high number of conference abstracts and multiple publications for the same study in their review. This might have influenced the findings of this study.

Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewers.
Study Type: 
Single study

Comments from the authors:

Appraisal of: Wieseler B, Wolfram N, McGauran N et al. Completeness of reporting of patient-relevant clinical trial outcomes: comparison of unpublished clinical study reports with publicly available data. PLoS Med 2013;10(10):e1001526.

Reviewer(s): 
Short description: 
The authors compared the information available in clinical study reports and in publicly accessible sources (journal articles, registry reports) included in 16 HTA reports for drugs prepared in IQWiG. They assessed the completeness of information on patient-relevant benefit (e.g. symptom relief, health related quality of life) and harm (adverse effects) outcomes reported in each document type. Clinical study reports provided complete information of all outcomes on a higher proportion (86%) than publicly accessible sources (39%). They also provided more information on harms (87% versus 43 %). The authors conclude that clinical study reports should be made publicly available for the benefit of unbiased trial evaluation and informed decision-making in health care.
Limitations stated by the author(s): 
The main limitation was that the authors could only study those clinical study reports that were provided voluntarily upon request by the pharmaceutical companies. Also, the sample of clinical study reports was restricted on studies investigating drugs.
Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewer.
Study Type: 
Single study
Related Chapters: 

Appraisal of: Golder S, Loke YK. Failure or success of electronic search strategies to identify adverse effects data. J Med Libr Assoc 2012;100(2):130-134.

Short description: 
The aim of this study was to assess the prevalence of adverse effects terms in the title, abstract, and / or indexing of adverse effects papers published from 2001 onwards and the implications of this on search strategy approaches. A cohort of 242 papers, indexed in MEDLINE, Embase and / or Science Citation Index (SCI), was identified to investigate this and compare the results with a similar study published in 2001 by Derry, Loke and Aronson. Of the 242 papers identified, 231 were indexed in MEDLINE, 222 were indexed in Embase and 238 were indexed in SCI. Searching with generic or specific named adverse effects terms in the title, abstract, or indexing identify 89% of the references indexed in Embase, 80% of the references indexed in Medline and 70% of the references indexed in SCI. Generic adverse effects terms in the title and abstract in any of the 3 databases, generic and specific indexing terms or adverse effects subheadings in Embase, and subheadings for adverse effects in Medline retrieved the highest proportion of references. Compared to previous research findings, adverse effects terms seemed to be increasingly prevalent in the title, abstract and indexing of adverse effects papers in both MEDLINE and Embase.
Limitations stated by the author(s): 
No limitations stated by the study authors
Limitations stated by the reviewer(s): 
No limitations detected by the reviewer
Study Type: 
Single study
Related Chapters: 

Appraisal of: Golder S, Loke YK. Sensitivity and precision of adverse effects search filters in MEDLINE and EMBASE: a case study of fractures with thiazolidinediones. Health Info Libr J 2012;29(1):28-38.

Short description: 
A systematic review of fracture-related adverse effects associated with the use of thiazolidinediones was used as a case study to calculate the sensitivity, precision and Number Needed to Read (NNR) of published adverse effects search filters in MEDLINE and Embase. 12 MEDLINE filters and 3 Embase filters were tested. The results showed that 4 search filters in MEDLINE achieved high levels of sensitivity (95 or 100%) with improved levels of precision compared with searches without any adverse effects filters. The highest level of precision in MEDLINE (55%) was achieved with search filters that relied only on subject headings (MeSH). No search filter in Embase achieved sensitivity higher than 83% and precision remained low using any of the filters (all under 5%).
Limitations stated by the author(s): 
The main limitation to this study is that because only one systematic review was used as a case study, the generalisability of the study results is limited. In addition, this case study is of a particular named adverse effect (fractures), while a case study of a safety profile systematic review, in which all adverse effects are searched for, might have given different results. Another limitation is the adaptations made to some of the tested search filters. These filters were originally created for use in searches where the adverse effects are not known in advance of searching, while in this case study, these filters were used in addition to search terms for a named adverse effect (fracture terms).
Limitations stated by the reviewer(s): 
In addition to the limitation noted above by the authors, it should be noted that the number of included studies on which the search filters were tested was relatively small (19 records in MEDLINE and 24 records in Embase). It should also be noted that the filters were tested as published. Testing of individual terms within the filters might have resulted in higher precision and better trade-off between sensitivity and precision than the full filters.
Study Type: 
Single study
Related Chapters: 

Comments from the authors:

Appraisal of: Golder S, McIntosh HM, Loke Y. Identifying systematic reviews of the adverse effects of health care interventions. BMC Med Res Methodol 2006;6:22.

Short description: 
The aim of the study was to assess how easily one could identify systematic reviews of adverse effects in the two major databases of systematic reviews: the Database of Abstracts of Reviews of Effects (DARE) and the Cochrane Database of Systematic Reviews (CDSR). A gold standard set of records (270 systematic reviews) was identified from DARE and CDSR by carrying out searches for systematic reviews of adverse effects published from 1994 to 2005. The search strategies used a combination of text words in the title and abstract, subject headings (MeSH) and subheadings. In addition, DARE records in progress were hand searched. The search terms used to identify the systematic reviews were assessed for their usefulness in retrieving relevant records by measuring their sensitivity and precision. The results showed that ‘floating’ subheadings provided the highest sensitivity in both DARE and CDSR (85% and 64% respectively). All single search terms in CDSR resulted in very low precision (0% - 3%). Single MeSH terms provided the highest level of precision in DARE (67%-73%). The most sensitive search strategy in DARE used a combination of text words in the title and abstract, a MeSH term and ‘floating’ subheadings (94%). The precision of this strategy was 16%. In CDSR the most sensitive strategy used the ‘floating’ subheading ‘adverse effects’ and text word search ‘adverse near/20 objectives’ in the abstract field (79%). The precision of this strategy was 3%.
Limitations stated by the author(s): 
Although DARE and CDSR are excellent sources of systematic reviews of adverse effects, not all reviews reported as being systematic are contained in these databases. Also, systematic reviews included in these databases tend to be of higher methodological quality, which may reflect better reporting and therefore better indexing. The low number of systematic reviews of adverse effects in CDSR (14) made the analysis of data and assessment of usefulness of individual search terms difficult. The search terms tested in the study were predefined from previous research and were not obtained by objective methods. However, papers not retrieved with the predefined search terms didn’t reveal many additional terms.
Limitations stated by the reviewer(s): 
Even though this study provides the most up-to-date evidence on the topic of identifying systematic reviews of adverse effects in CDSR and DARE, readers should bear in mind that the study was published in 2006. Adverse effects terms may be increasingly prevalent in the title, abstract and indexing terms of records of studies published more recently compared to those published up until 2006. Readers may, therefore, wish to consider the appraisals of more recent work by Golder et al which address identifying single studies reporting adverse effects. Changes made in the two databases in recent years (such as new user interfaces and search functionality) may also influence the validity of the study findings. In addition to the limitation outlined above regarding the very low number of systematic reviews of adverse effects in CDSR (14), it should also be noted that reviews of adverse effects make up a relatively low percentage of the total number of reviews in DARE and an even lower percentage of the total number of reviews in CDSR and that this might have some bearing on the findings of the study. Also, it should be noted that the same set of search terms was used both to identify the systematic reviews for the gold standard set of records, and to measure the performance of single search terms and their combinations. This might have influenced the findings of this study.
Study Type: 
Single study
Related Chapters: 

Comments from the authors:

Appraisal of: Golder S, Loke YK. Sources of information on adverse effects. Health Info Libr J 2010;27:176-190.

Short description: 
Research has shown that most systematic reviews of adverse effects rely solely on searches of MEDLINE, even though it is unlikely to be a comprehensive source on adverse effects information. The authors aimed to identify and summarize studies that had evaluated sources of information on adverse effects. No date or language restrictions were applied when searching for studies. The results indicated that Embase, Derwent Drug File, MEDLINE and industry submissions might be the sources of the largest number of relevant references for adverse effects information. In addition, they concluded that searching a wide range of sources might be a useful approach when conducting a thorough search.
Limitations stated by the author(s): 
Studies included in the review were inconsistent in their use of outcome measures. They used different information sources which made direct comparisons difficult. Recent research information was lacking and many of the studies were more than 10 years old. Many potentially useful information sources were not covered in the studies identified (e.g. search engines and industry clinical trial registers). Most studies reported only the number of relevant references retrieved for comparison which is not a sufficient criterion. Also, the cost of searching different sources had not been assessed.
Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewer.
Study Type: 
Review
Related Chapters: 
Comments from the authors:
 
This publication is related to Su Golder’s PhD Thesis “Evaluating and Optimising the Retrieval of Research Evidence for Systematic Reviews of Adverse Drug Effects and Adverse Drug Reactions” from 2013. The thesis is available from http://etheses.whiterose.ac.uk/4749/

Appraisal of: Golder S, Loke YK. The contribution of different information sources for adverse effects data. Int J Technol Assess Health Care 2012;28(2):133-137.

Short description: 

Research has shown that MEDLINE might not be a comprehensive source on adverse effects information. The aim of this study was to evaluate the contribution of a variety of sources when searching for adverse effects data for a systematic review of thiazolidinedione-related fractures in patients with type 2 diabetes mellitus.

The results showed the potential value of searching a variety of sources for adverse effects data.

In this study, the minimum combination of sources required to identify all the relevant references was:

  • GlaxoSmithKline website,
  • Science Citation Index,
  • Embase,
  • BIOSIS Previews,
  • British Library Direct,
  • Medscape DrugInfo,
  • AHFS First (American Hospital Formulary Service),
  • Thomson Reuters Integrity,
  • Conference Papers Index,
  • Handsearching,
  • Reference checking.
Limitations stated by the author(s): 
The results were based on a single case study and are, not necessarily generalisable. It was also difficult to maintain consistency in search strategies when adapting them to different search interfaces and it was, therefore, difficult to make fair comparisons. It was also not possible to perform a cost analysis of searching each source due to different pricing mechanisms used by the database providers.
Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewer.
Study Type: 
Single study
Related Chapters: 

Comments from the authors:

Appraisal of: Golder S, Loke YK, Bland M. Unpublished data can be of value in systematic reviews of adverse effects: methodological overview. J Clin Epidemiol 2010;63(10):1071-1081.

Short description: 
The authors assessed the impact of including unpublished data on adverse effects in systematic reviews. They carried out a systematic review of methodological evaluations comparing the quantitative reporting of adverse effects data between published and unpublished sources. The key finding was that unpublished studies could provide additional adverse effects information. However, there was insufficient data to conclude whether including unpublished studies would have major impact on the results of meta-analyses.
Limitations stated by the author(s): 
Methodological evaluations are difficult to retrieve by database searches and it is possible that review articles may have been missed. There is also a possibility of reporting or publication bias with respect to methodological evaluations, as investigators may have chosen not to report their findings if they did not find any significant differences between published and unpublished studies.
Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewer.
Study Type: 
Review
Related Chapters: 

Comments from the authors:

Appraisal of: Golder S, Loke Y. The performance of adverse effects search filters in MEDLINE and EMBASE. Health Info Libr J 2012;29(2):141-151.

Short description: 
The aim of the study was to measure the sensitivity of published adverse effects search filters in MEDLINE and Embase. The performance of all the individual search terms included in the filters was also measured. The results showed that 93% of the MEDLINE records and 97% of the Embase records could be retrieved by using adverse effects search filters. High sensitivity could also be achieved by using individual adverse effects ‘floating’ subheadings such as Adverse Drug Reaction (83%) or Side Effect (83%) in Embase. In MEDLINE the highest sensitivity using individual adverse effects ‘floating’ subheadings was achieved by using Adverse Effects (51%). Some free text terms for adverse effects in the title and / or abstract were also found to be useful, but the authors recommend applying these terms only concurrently with other search terms such as subheadings. The sensitivity of the few existing indexing terms (MeSH and Emtree) for adverse effects was shown to be low.
Limitations stated by the author(s): 
The main limitation of this study was that the precision of search terms and search filters was not measured. The most sensitive search filters or search terms might be those with the lowest precision and might therefore result in very low precision (that is, the retrieval of very large numbers of irrelevant records). Further research that measures the precision of these search terms and filters is required in order to determine the full value and effectiveness of using adverse effects terms in search strategies. Caution should be applied when using the results of this study, especially as some of the search terms with relatively high sensitivity appear to have only a vague connection to adverse effects (such as ‘Therapeutic Use’ and ‘Pharmacology’).
Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewer
Study Type: 
Single study
Related Chapters: 

Comments from the authors:

Appraisal of: Golder S, Loke YK, Bland M. Meta-analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological overview. PLoS Med 2011;8(5):e1001026.

Reviewer(s): 
Short description: 
This methodological overview investigates the impact of study design in meta-analyses of adverse effects data. The authors look at the estimates of the risk of adverse events derived from meta-analyses of randomized controlled trials (RCTs) and from observational studies. The evidence indicates that there is no difference on average between estimates of harm in meta-analyses of RCTs as compared to observational studies. The study suggests that systematic reviews of harms should not be restricted to specific study types.
Limitations stated by the author(s): 
Confounding factors may account for any differences identified when comparing pooled results from various study designs. Due to time constraints, data was not extracted from individual papers; rather the data reported by the authors of the meta-analyses were used. Because the search strategy was designed to identify methodological papers whose primary goal was to assess the influence of study design on adverse effects, systematic reviews with embedded data comparing study designs may have been missed. The considerable heterogeneity between the comparisons of different studies suggests that any differences could be specific to particular types of interventions or certain adverse events. For example, some types of adverse events may be more easily identified by RCTs as opposed to observational studies.
Limitations stated by the reviewer(s): 
No additional limitations detected by the reviewer
Study Type: 
Review
Related Chapters: 

Comments from the authors:

Syndicate content