Skip Navigation Bar
 

National Information Center on Health Services Research and Health Care Technology (NICHSR)

blue arrow
blue arrow
blue arrow
blue arrow
blue arrow
blue arrow
blue arrow
blue arrow
blue arrow
blue arrow
blue arrow
blue arrow
blue arrow
blue arrow
Health Economics Logo

Health Economics Information Resources: A Self-Study Course

Module 3: Identification and Retrieval of Published Health Economic Evaluation Studies

The quality of health economic evaluation studies

In addition to thinking about the consequences of mislabeling, we should also pay attention to the quality of health economic evaluation studies. Why? For two reasons.

The variability in the quality of published health economic evaluation studies is well documented *

This variability has significant implications for the identification and subsequent utilization of information on efficiency in the health care decision-making process

*Jefferson T, Demicheli V, Vale L. Quality of systematic reviews of economic evaluations in health care. JAMA 2002;287:2809-12. [online] Site URL.
*Jefferson T, Demicheli V. Quality of economic evaluations in health care. BMJ 2002;324:313-314. [online] Site URL.

Issues regarding quality of health economic evaluation studies

In this section we look at some of the issues and problem areas in the quality of health economic evaluation studies. We find three major deficiencies: poor methodological design, inadequate reporting, and publication bias.

Poor methodological design includes such concerns as poor quality data sources, failure to define clearly the economic evaluation method used, or use of an inappropriate method if the economic evaluation is to address the research question, and omission of sensitivity analysis to test robustness of modeling.

Inadequate reporting is generally a lack of transparency regarding methodology and/or a lack of stringency of journal editorial policy with regard to acceptance of economic submissions.

Publication bias results from bias in the effectiveness literature (e.g., when studies showing negative results have not been published) or from the motivation for conducting an economic evaluation (e.g., studies sponsored by groups with financial interests in the outcomes of the study).

Guidelines for conduct of an economic evaluation

Guidelines have been developed as a means of addressing the problem of quality variability in health economic evaluation studies. Guidelines may be categorized as those which address the conduct, reporting, or appraisal of economic evaluation studies.

These form two distinct categories: guidelines for use prior to pharmaceutical reimbursement, as, for example, the Ontario Guidelines for Economic Analysis of Pharmaceutical Products, or guidelines aimed at improving health economic methods, such as the Washington Panel*, CADTH (The Canadian Agency for Drugs and Technologies in Health).

Inconsistencies found between guidelines for conduct may be due to differences in purpose or differences in the health care system.

*Weinstein MC. et al. Recommendations of the Panel on Cost-effectiveness in Health and Medicine. JAMA 1996;276:1253-58. [online] Site URL.

Guidelines for reporting economic evaluations

The editorial process employed by medical journals with regard to economic submissions can be less rigorous compared to that process employed for biomedical papers.

There is evidence that economic evaluations published in general clinical journals and in journals that published more of these analyses are of higher quality.*

*Neumann PJ, Stone PW, Chapman RH, Sandberg EA, Bell CM. The quality of reporting in published Cost-Utility Analyses, 1976-1997. Ann Intern Med 2000; 132: 964-972. [online] Site URL.

Some general medical journals have introduced reporting guidelines, such as the British Medical Journal (BMJ), Journal of the American Medical Association (JAMA), and the New England Journal of Medicine.

The BMJ guidelines* are the most explicit of those produced so far.

*Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions. BMJ 1996;313:275-83. [online] Site URL.

Guidelines for appraisal of economic evaluations

Numerous checklists exist for use in the quality assessment of health economic evaluations. These checklists vary in presentation and depth but address common core issues. Reporting guidelines such as those developed for the BMJ can be adopted for use in quality assessment.

The guidance for writing abstracts for the NHS Economic Evaluations Database (NHS EED) provides a quality assessment tool

*Improving access to cost-effectiveness information for health care decision-making: the NHS Economic Evaluation Database. NHSCRD Report no.6(2nd ed). York. NHS Centre for Reviews and Dissemination. 2001. Note: Superseded by the 2007 NHS EED handbook.

Already-appraised economic evaluations: A look at the NHS Economic Evaluation Database

The NHS Economic Evaluation Database (NHS EED) is a value-added source of economic evaluation studies. The studies have been identified from relevant core bibliographic databases and evaluated by specific criteria. NHS EED is a key source in providing structured abstracts of economic evaluation studies that include a critical appraisal, and a commentary on the relevance of study findings to practice.

NHS EED abstracts focus on health care issues of importance to developed countries.

Bibliographic database indexing of economic evaluations

The accuracy of indexing terms that have been applied to individual economic evaluation studies will depend on three things: 1) a clear description of the study and accurate labeling of the methodology at the reporting stage; and, 2) the specificity of terms available within the database controlled vocabulary. This will, in turn, 3) depend on the indexing policy of the database producer.

The structure and scope of indexing terms will often differ between databases; not all bibliographic databases include a controlled vocabulary. If this is the case, reliance must be placed on free text searching.

blue arrow facing left Previous
Test Yourself Nextblue arrow