National Information Center on Health Services Research and Health Care Technology (NICHSR)
Health Economics Information Resources: A Self-Study Course
Module 3: Identification and Retrieval of Published Health Economic Evaluation Studies
The quality of health economic evaluation studies
In addition to thinking about the consequences of mislabeling, we should also pay attention to the quality of health economic evaluation studies. Why? For two reasons.
The variability in the quality of published health economic evaluation studies is well documented *
This variability has significant implications for the identification and subsequent utilization of information on efficiency in the health care decision-making process
*Jefferson T, Demicheli V, Vale L. Quality of systematic reviews of economic evaluations in health care. JAMA 2002;287:2809-12. [online] Site URL.
Issues regarding quality of health economic evaluation studies
In this section we look at some of the issues and problem areas in the quality of health economic evaluation studies. We find three major deficiencies: poor methodological design, inadequate reporting, and publication bias.
These form two distinct categories: guidelines for use prior to pharmaceutical reimbursement, as, for example, the Ontario Guidelines for Economic Analysis of Pharmaceutical Products, or guidelines aimed at improving health economic methods, such as the Washington Panel*, CADTH (The Canadian Agency for Drugs and Technologies in Health).
Inconsistencies found between guidelines for conduct may be due to differences in purpose or differences in the health care system.
*Weinstein MC. et al. Recommendations of the Panel on Cost-effectiveness in Health and Medicine. JAMA 1996;276:1253-58. [online] Site URL.
The editorial process employed by medical journals with regard to economic submissions can be less rigorous compared to that process employed for biomedical papers.
*Neumann PJ, Stone PW, Chapman RH, Sandberg EA, Bell CM. The quality of reporting in published Cost-Utility Analyses, 1976-1997. Ann Intern Med 2000; 132: 964-972. [online] Site URL.
The BMJ guidelines* are the most explicit of those produced so far.
Numerous checklists exist for use in the quality assessment of health economic evaluations. These checklists vary in presentation and depth but address common core issues. Reporting guidelines such as those developed for the BMJ can be adopted for use in quality assessment.
The guidance for writing abstracts for the NHS Economic Evaluations Database (NHS EED) provides a quality assessment tool
*Improving access to cost-effectiveness information for health care decision-making: the NHS Economic Evaluation Database. NHSCRD Report no.6(2nd ed). York. NHS Centre for Reviews and Dissemination. 2001. Note: Superseded by the 2007 NHS EED handbook.
The NHS Economic Evaluation Database (NHS EED) is a value-added source of economic evaluation studies. The studies have been identified from relevant core bibliographic databases and evaluated by specific criteria. NHS EED is a key source in providing structured abstracts of economic evaluation studies that include a critical appraisal, and a commentary on the relevance of study findings to practice.
NHS EED abstracts focus on health care issues of importance to developed countries.
The accuracy of indexing terms that have been applied to individual economic evaluation studies will depend on three things: 1) a clear description of the study and accurate labeling of the methodology at the reporting stage; and, 2) the specificity of terms available within the database controlled vocabulary. This will, in turn, 3) depend on the indexing policy of the database producer.
The structure and scope of indexing terms will often differ between databases; not all bibliographic databases include a controlled vocabulary. If this is the case, reliance must be placed on free text searching.