Skip Navigation Bar
 

National Information Center on Health Services Research and Health Care Technology (NICHSR)

HTA 101: II. FUNDAMENTAL CONCEPTS

Health Technology

Technology is the practical application of knowledge. Three ways to describe health care technology include its material nature, its purpose, and its stage of diffusion.

Material Nature

For many people, the term "technology" connotes "hardware" or other mechanical devices or instrumentation; to others, it is a short form of "information technology" such as computer software. However, the practical application of knowledge in health care is quite broad. Broad categories of health technology include the following.

  • Drugs: e.g., aspirin, beta-blockers, antibiotics, HMG-CoA reductase inhibitors ("statins")
  • Biologics: vaccines, blood products, cellular and gene therapies
  • Devices, equipment and supplies: e.g., cardiac pacemakers, CT scanners, surgical gloves, diagnostic test kits
  • Medical and surgical procedures: e.g., psychotherapy, nutrition counseling, coronary angiography, gall bladder removal
  • Support systems: e.g., electronic patient record systems, telemedicine systems, drug formularies, blood banks, clinical laboratories
  • Organizational and managerial systems: e.g., prospective payment using diagnosis-related groups, alternative health care delivery configurations, clinical pathways, total quality management programs

Purpose or Application

Technologies can also be grouped according to their health care purpose, i.e.:

  • Prevention: protect against disease by preventing it from occurring, reducing the risk of its occurrence, or limiting its extent or sequelae (e.g., immunization, hospital infection control program, fluoridated water supply)
  • Screening: detect a disease, abnormality, or associated risk factors in asymptomatic people (e.g., Pap smear, tuberculin test, mammography, serum cholesterol testing)
  • Diagnosis: identify the cause and nature or extent of disease in a person with clinical signs or symptoms (e.g., electrocardiogram, serological test for typhoid, x-ray for possible broken bone)
  • Treatment: designed to improve or maintain health status, avoid further deterioration, or provide palliation (e.g., antiviral therapy, coronary artery bypass graft surgery, psychotherapy, drugs for cancer pain)
  • Rehabilitation: restore, maintain or improve a physically or mentally disabled person's function and well-being (e.g., exercise program for post-stroke patients, assistive device for severe speech impairment, incontinence aid)

Not all technologies fall neatly into single categories. Many tests and other technologies used for diagnosis also are used for screening. (The probability that a patient has a disease or other health condition is greatly affected by whether these technologies are used for screening asymptomatic patients or diagnosing symptomatic patients.) Some technologies are used for diagnosis as well as treatment, e.g., coronary angiography to diagnose heart disease and to monitor coronary angioplasty. Implantable cardioverter defibrillators detect potentially life-threatening heart arrhythmias and deliver electrical pulses to restore normal heart rhythm. Electronic patient record systems can support all of these technological purposes or applications.

Certain "boundary-crossing" or "hybrid" technologies combine characteristics of drugs, devices or other major categories of technology (Goodman 1993; Lewin Group 2001). Among the many examples of these are: photodynamic therapy, in which drugs are laser-activated (e.g., for targeted destruction of cancer cells); local drug delivery technologies (e.g., implantable drug pumps and drug inhalers); spermicidal condoms; and bioartificial organs that combine natural tissues and artificial components. Examples of hybrid technologies that have complicated regulatory approval and coverage decisions in recent years are gallstone lithotripters (used with stone-dissolving drugs) (Zeman 1991), positron-emission tomography (PET, used with radiopharmaceuticals) (Coleman 1992), and metered-dose inhalers (Massa 2002).

Stage of Diffusion

Technologies may be assessed at different stages of diffusion and maturity. In general, health care technologies may be described as being:

  • Future:  in a conceptual stage, anticipated, or in the earliest stages of development
  • Experimental:  undergoing bench or laboratory testing using animals or other models
  • Investigational:  undergoing initial clinical (i.e., in humans) evaluation for a particular condition or indication
  • Established: considered by providers to be a standard approach to a particular condition or indication and diffused into general use
  • Obsolete/outmoded/abandoned:  superseded by other technologies or demonstrated to be ineffective or harmful

Often, these stages are not clearly delineated, and technologies do not necessarily mature through them in a linear fashion. A technology may be investigational for certain indications, established for others, and outmoded or abandoned for still others, such as autologous bone marrow transplantation with high-dose chemotherapy for certain types of advanced cancers. Many technologies undergo multiple incremental innovations after their initial acceptance into general practice (Gelijns and Rosenberg 1994; Reiser 1994). Further, a technology that was once considered obsolete may return to established use for a better defined or entirely different clinical purpose. A prominent example is thalidomide, whose use as a sedative during pregnancy was halted more than 40 years ago when it was found to induce severe fetal malformation, but which is now used to treat such conditions as leprosy, advanced multiple myeloma, chronic graft vs. host disease, and certain complications of HIV infection (Baidas 2002).  

Health Technology Assessment

Health technology assessment is the systematic evaluation of properties, effects or other impacts of health technology. The main purpose of HTA is to inform policymaking for technology in health care, where policymaking is used in the broad sense to include decisions made at, e.g., the individual or patient level, the level of the health care provider or institution, or at the regional, national and international levels. HTA may address the direct and intended consequences of technologies as well as their indirect and unintended consequences. HTA is conducted by interdisciplinary groups using explicit analytical frameworks, drawing from a variety of methods.  

Purposes of HTA

HTA can be used in many ways to advise or inform technology-related policymaking. Among these are to advise or inform:

  • Regulatory agencies such as the US Food and Drug Administration (FDA) about whether to permit the commercial use (e.g., marketing) of a drug, device or other technology
  • Health care payers, providers, and employers about whether technologies should be included in health benefits plans or disease management programs, addressing coverage (whether or not to pay) and reimbursement (how much to pay)
  • Clinicians and patients about the appropriate use of health care interventions for a particular patient's clinical needs and circumstances
  • Health professional associations about the role of a technology in clinical protocols or practice guidelines
  • Hospitals, health care networks, group purchasing organizations, and other health care organizations about decisions regarding technology acquisition and management
  • Standards-setting organizations for health technology and health care delivery regarding the manufacture, use, quality of care, and other aspects of health care technologies
  • Government health department officials about undertaking public health programs (e.g., vaccination, screening, and environmental protection programs)
  • Lawmakers and other political leaders about policies concerning technological innovation, research and development, regulation, payment and delivery of health care
  • Health care product companies about product development and marketing decisions
  • Investors and companies concerning venture capital funding, acquisitions and divestitures, and other transactions concerning health care product and service companies

HTA contributes in many ways to the knowledge base for improving the quality of health care, especially to support development and updating of a wide spectrum of standards, guidelines, and other health care policies. For example, the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) and the National Committee for Quality Assurance (NCQA) set standards for measuring quality of care and services of hospitals, managed care organizations, long-term care facilities, hospices, ambulatory care centers, and other health care institutions. Health professional associations (e.g., American College of Cardiology, American College of Physicians, American College of Radiology, American Medical Association) and special panels (e.g., the US Preventive Services Task Force of the Agency for Healthcare Research and Quality) develop clinical practice guidelines, standards, and other statements regarding the appropriate use of technologies. Standards-setting organizations such as the American National Standards Institute and the American Society for Testing and Materials coordinate development of voluntary national consensus standards for the manufacture, use and reuse of health devices and their materials and components.

As noted above, HTA can be used to support decisionmaking by clinicians and patients. The term evidence-based medicine refers to the use of current best evidence from scientific and medical research, and the application of clinical experience and observation, in making decisions about the care of individual patients. This has prompted the appearance of many useful resources, including:

  • Evidence-Based Medicine (Sackett 1997), a guide to the field
  • Evidence-Based Medicine (a joint product of the American College of Physicians and the BMJ Publishing Group), a journal digest of articles selected from international medical journals
  • "Users' guides to the medical literature," a series of more than 30 articles by the Evidence-Based Medicine Working Group published in the Journal of the American Medical Association, ranging from (Oxman 1993) to (Guyatt 2000
  • Centre for Evidence-Based Medicine [http://www.cebm.net/]

Basic HTA Orientations

The impetus for an HTA is not necessarily a technology. Three basic orientations to HTA are as follows.

  • Technology-oriented assessments are intended to determine the characteristics or impacts of particular technologies. For example, a government agency may want to determine the clinical, economic, social, professional, or industrial impacts of population-based cancer screening, cochlear implants, or other particular interventions.
  • Problem-oriented assessments focus on solutions or strategies for managing a particular problem for which alternative or complementary technologies might be used. For example, clinicians and providers concerned with the problem of diagnosis of dementia may call for the development of clinical practice guidelines involving some combination or sequence of clinical history, neurological examination, and diagnostic imaging using various modalities..
  • Project-oriented assessments focus on a local placement or use of a technology in a particular institution, program, or other designated project.  For example, this may arise when a hospital must decide whether or not to purchase a magnetic resonance imaging (MRI) unit, considering the facilities, personnel, and other resources needed to install and operate an MRI unit; the hospital's financial status; local market potential for MRI services; competitive factors; etc.

These basic assessment orientations can overlap and complement one another. Certainly, all three types could draw upon a common body of scientific evidence and other information. A technology-oriented assessment may address the range of problems for which the technology might be used and how appropriate the technology might be for different types of local settings (e.g., inpatient versus outpatient). A problem-oriented assessment, examining the effects or other impacts of alternative technologies on a given problem, may incorporate multiple, focused (i.e., on the problem at hand) technology-oriented assessments. A project-oriented assessment would consider the range of impacts of a technology or its alternatives in a given setting, as well as the role or usefulness of that technology for various problems. Although the information used in a project-oriented assessment by a particular hospital may include findings of pertinent technology- and problem-oriented assessments, local data collection and analysis may be required to determine what is sensible for that hospital. Thus, many HTAs will blend aspects of all three basic orientations.

Timing of Assessment

There is no single correct time to conduct an HTA. It is conducted to meet the needs of a variety of policymakers seeking assessment information throughout the lifecycles of technologies. Investors, regulators, payers, hospital managers and others tend to make decisions about technologies at particular junctures, and each may subsequently reassess technologies. Indeed, the determination of a technology's stage of diffusion may be the primary purpose of an assessment. For insurers and other payers, technologies that are deemed experimental or investigational are usually excluded from coverage, whereas those that are established or generally accepted are usually eligible for coverage (Newcomer 1990; Reiser 1994; Singer 2001).

There are tradeoffs inherent in decisions regarding the timing for HTA. On one hand, the earlier a technology is assessed, the more likely its diffusion can be curtailed if it is unsafe or ineffective (McKinlay 1981). From centuries' old purging and bloodletting to the more contemporary autologous bone marrow transplantation with high-dose chemotherapy for advanced breast cancer, the list of poorly evaluated technologies that diffused into general practice before being found to be ineffective and/or harmful continues to grow. Box 3 shows examples of health care technologies found to be ineffective or harmful after being widely diffused.

On the other hand, to regard the findings of an early assessment as definitive or final may be misleading. An investigational technology may not yet be perfected; its users may not yet be proficient; its costs may not yet have stabilized; it may not have been applied in enough circumstances to recognize its potential benefits; and its long-term outcomes may not yet be known (Mowatt 1997). As one technology assessor concluded about the problems of when-to-assess: "It's always too early until, unfortunately, it's suddenly too late!" (Buxton 1987). Further, the "moving target problem" can complicate HTA (Goodman 1996). By the time a HTA is conducted, reviewed and disseminated, its findings may be outdated by changes in a technology, in how it is used, or in its technological alternatives for a given problem.

Some payers provide conditional coverage for selected investigational technologies in order to compile evidence on safety, effectiveness, cost, etc., for making more informed coverage policies. In these instances, payers cover the use of a technology only under certain conditions, such as where patients are enrolled in an RCT at certain participating medical centers. This arrangement offers a way to balance the need for evidence with the demand for access and financially compensated care. Depending on the type of technology involved, it further enables refinement of technique or delivery, and building of experience and expertise among physicians and other providers (Beebe 1997; Brenner 2002; McGivney 1992; Sheingold 1998; Medical Technology Leadership Forum 1999; Wood 2001).  

Box 3
Examples of Health Care Technologies Found to be Ineffective or Harmful After Being Widely Diffused

  • Autologous bone marrow transplant with high-dose chemotherapy for advanced breast cancer
  • Colectomy to treat epilepsy
  • Diethylstilbestrol (DES) to improve pregnancy outcomes
  • Electronic fetal monitoring during labor without access to fetal scalp sampling
  • Episiotomy (routine or liberal) for birth
  • Extracranial-intracranial bypass to reduce risk of ischemic stroke
  • Gastric bubble for morbid obesity
  • Gastric freezing for peptic ulcer disease
  • Hormone replacement therapy for healthy menopausal women
  • Hydralazine for chronic heart failure
  • Intermittent positive pressure breathing
  • Mammary artery ligation for coronary artery disease
  • Optic nerve decompression surgery for nonarteritic anterior ischemic optic neuropathy
  • Quinidine for suppressing recurrences of atrial fibrillation
  • Radiation therapy for acne
  • Sleeping face down for healthy babies
  • Supplemental oxygen for healthy premature babies
  • Thalidomide for sedation in pregnant women
  • Thymic irradiation in healthy children
  • Triparanol (MER-29) for cholesterol reduction

Sources: Coplen 1990; Enkin 1995; Feeny 1986; Fletcher 2002; Grimes 1993; Mello 2001; The Ischemic Optic Neuropathy Decompression Trial Research Group 1995; Passamani 1991; Rossouw 2002; US DHHS 1990, 1993; others.

Despite the value of conditional coverage in principle, some observers have raised practical and ethical concerns about their implementation. Among these are that: (1) if conditional coverage is initiated after a technology has diffused, some patients who had expected to get a procedure may be denied it if they are not enrolled in a trial; (2) some patients who would be interested in enrolling in a covered trial are not located near a participating center and are therefore denied access; (3) patients and physicians who believe in the effectiveness of the technology may be unwilling to be involved in an RCT, including some who decide to finance the technology outside of the trial and therefore diminish enrollment, (4) the indications for using the technology in the conditional coverage trial may be too broad or too narrow to properly reflect the potential safety and effectiveness of the technology; and (5) the technology continues to evolve during the conditional coverage process, to the point where the trial findings are of diminished relevance (Berger 2001; Cooper 2001).

Properties and Impacts Assessed

What is assessed in HTA? HTA may involve the investigation of one or more properties, impacts, or other attributes of health technologies or applications. In general, these include the following.

  • Technical properties
  • Safety
  • Efficacy and/or effectiveness
  • Economic attributes or impacts
  • Social, legal, ethical and/or political impacts

Technical properties include performance characteristics and conformity with specifications for design, composition, manufacturing, tolerances, reliability, ease of use, maintenance, etc. Safety is a judgment of the acceptability of risk (a measure of the probability of an adverse outcome and its severity) associated with using a technology in a given situation, e.g., for a patient with a particular health problem, by a clinician with certain training, and/or in a specified treatment setting.

Efficacy and effectiveness both refer to how well a technology works to improve patient health, usually based on changes in one or more pertinent health outcomes or "endpoints" as described below. A technology that works under carefully controlled conditions or with carefully selected patients under the supervision of its developers does not always work as well in other settings or as implemented by other practitioners. In HTA, efficacy refers to the benefit of using a technology for a particular problem under ideal conditions, e.g., within the protocol of a carefully managed randomized controlled trial, involving patients meeting narrowly defined criteria, or conducted at a "center of excellence." Effectiveness refers to the benefit of using a technology for a particular problem under general or routine conditions, e.g., by a physician in a community hospital for a variety of types of patients.  

Clinicians, patients, managers and policymakers are increasingly aware of the practical implications of differences in efficacy and effectiveness. Researchers delve into registers, databases (e.g., of third-party payment claims and administrative data) and other epidemiological and observational data to discern possible associations between the use of technologies and patient outcomes in general or routine practice settings. The validity of any findings regarding causal connections between interventions and patient outcomes may be weakened to the extent that these data are not derived from prospective, randomized, controlled studies (US Congress, OTA 1994). As discussed below, some newer prospective trials are designed to incorporate varied groups of patients and settings.

Box 4 shows certain distinctions in efficacy and effectiveness for diagnostic tests. Whereas the relationship between a preventive, therapeutic, or rehabilitative technology and patient outcomes is typically direct (though not always easy to measure), the relationship between a technology used for diagnosis or screening and its patient outcomes is typically indirect. Also, diagnostic and screening procedures can have their own short-term and long-term adverse health effects, e.g., biopsies and certain radiological procedures.

Health technologies can have a wide range of microeconomic and macroeconomic attributes or impacts. Microeconomic concerns include costs, prices, charges, and payment levels associated with individual technologies. Other concerns include comparisons of resource requirements and outcomes (or benefits) of technologies for particular applications, such as cost effectiveness, cost utility, and cost benefit. (Methods for determining these are described below.)

Box 4
Efficacy vs. Effectiveness for Diagnostic Tests

 

 

Efficacy

Effectiveness

Patient Population

Homogeneous; patients with coexisting illness often excluded

Heterogeneous; includes all patients who usually have test

Procedures

Standardized

Often variable

Testing Conditions

Ideal

Conditions of everyday practice

Practitioner

Experts

All users

Adapted from: Institute of Medicine 1989.

Examples of macroeconomic impacts of health technologies are the impact of new technologies on: national health care costs, resource allocation among different health programs or among health and other sectors, and shifts in the site of care, such as from inpatient to outpatient settings. Other macroeconomic issues that pertain to health technologies include the effects of intellectual property policies (e.g., for patent protection), regulation, third-party payment, and other policy changes on technological innovation, investment, competitiveness, technology transfer, and employment.

A variety of technologies raise social and ethical concerns. Such technologies as genetic testing, use of stem cells to grow new tissues, allocation of scarce organs for transplantation, and life-support systems for the critically ill challenge certain legal standards and societal norms. For example, the small and slowly increasing supply of donated kidneys, livers, hearts, and other organs for transplantation continues to fall behind the rapidly expanding need for them, raising ethical, social, and political concerns about allocation of scarce, life-saving resources (Miranda 1998; Yoshida 1998). In dialysis and transplantation for patients with end-stage renal disease, ethical concerns arise from patient selection criteria, termination of treatment, and managing non-compliant and other problem patients (Rettig 1991).

Ethical questions continue to prompt improvement in informed consent procedures for patients involved in clinical trials. Allocation of scarce resources to technologies that are expensive, inequitably used, or non-curative raises broad social concerns (Gibson 2002). Ethical considerations arise in HTA in the form of normative concepts (e.g., valuation of human life); applications of technology (prevention, screening, diagnosis, therapy, etc.); research and the advancement of knowledge; allocation of resources; and the integrity of HTA processes themselves (Heitman 1998). Methods for assessing ethical and social implications of health technology remain relatively underdeveloped, and the means of translating these implications into policy are often unclear (Van der Wilt 2000). Even so, greater efforts are being made to involve different perspectives in the HTA process in order to better account for identification of the types of effects or impacts that should be assessed, and for values assigned by these different perspectives to life, quality of life, privacy, choice of care, and other matters (Reuzel 2001).  

The terms "appropriate" and "necessary" often are used to describe whether or not a technology should be used in particular circumstances. For example, the appropriateness of a diagnostic test may depend on its safety and effectiveness compared to alternative available interventions for particular patient indications, clinical settings, and resource constraints. A technology may be considered necessary if withholding it would be deleterious to the patient's health (Hilborne 1991; Kahan 1994; Singer 2001).

The properties, impacts, and other attributes assessed in HTA pertain across the wide range of types of technology. Thus, for example, just as drugs, devices, and surgical procedures can be assessed for safety, effectiveness, and cost effectiveness, so can hospital infection control programs, computer-based drug-utilization review systems, and rural telemedicine networks.

Measuring Health Outcomes

Health outcome variables are used to measure the safety, efficacy and effectiveness of health care technologies. Health outcomes have been measured primarily in terms of changes in mortality (death rate) or morbidity (disease rate). For a cancer treatment, the main outcome of interest may be five-year survival; for treatments of coronary artery disease, the main endpoints may be incidence of fatal and nonfatal acute myocardial infarction and recurrence of angina. Increasingly, health outcomes are being measured in the form of health-related quality of life and functional status.  

In a clinical trial comparing alternative treatments, the effect on health outcomes of one treatment relative to another (e.g., a control treatment) can be expressed using various measures of treatment effect. These measures compare the probability of a given health outcome in the treatment group with the probability of the same outcome in a control group. Examples are absolute risk reduction, odds ratio, number needed to treat, and effect size. Box 5 shows how choice of treatment effect measures can give different impressions of study results.  

Health-Related Quality of Life Measures

Although mortality and morbidity are usually the outcomes of greatest concern, they are not the only outcomes of importance to patients nor to others. Many technologies affect patients, family members, providers, employers, and other interested parties in ways that are not reflected in mortality or morbidity rates; this is particularly true for many chronic diseases.

Health-related quality of life (HRQL) measures (or indexes) are increasingly used along with more traditional outcome measures to assess health care technologies, providing a more complete picture of the ways in which health care affects patients. HRQL measures capture such dimensions as: physical function, social function, cognitive function, anxiety/distress, bodily pain, sleep/rest, energy/fatigue and general health perception. HRQL measures may be disease-specific (e.g., heart disease or arthritis) or general (covering overall health). They may be one-dimensional (concerning one aspect such as distress) or multidimensional (Patrick and Deyo 1989). They may provide a single aggregate score or yield a set of scores, each for a particular dimension. HRQL measures are increasingly used by health product companies to differentiate their products from those of competitors, which may have virtually indistinguishable effects on morbidity for particular diseases (e.g., hypertension and depression) but may have different profiles of side effects that affect patients' quality of life (Gregorian 2003).

Box 5
Choice of Treatment Effect Measures Can Give Different Impressions

A study of the effect of breast cancer screening can be used to contrast several treatment effect measures and to show how they can give different impressions about the effectiveness of an intervention (Forrow 1992). In 1988, Andersson (1988) reported the results of a large RCT that was conducted to determine the effect of mammographic screening on mortality from breast cancer. The trial involved more than 42,000 women who were over 45 years old. Half of the women were invited to have mammographic screening and were treated as needed. The other women (control group) were not invited for screening.

The report of this trial states that "Overall, women in the study group aged >55 had a 20% reduction in mortality from breast cancer." Although this statement is true, calculation of other types of treatment effect measures provides important additional information. The table below shows the number of women aged >55 and breast cancer deaths in the screened group and control group, respectively. Based on these figures, four treatment effect measures are calculated.

For example, absolute risk reduction is the difference in the rate of adverse events between the screened group and the control group. In this trial, the absolute risk reduction of 0.0007 means that the absolute effect of screening was to reduce the incidence of breast cancer mortality by 7 deaths per 10,000 women screened, or 0.07%.

Group

Number of Patients

Deaths from breast cancer

Probability of death from breast cancer

Absolute risk reduction1

Relative risk reduction2

Odds ratio3

Number needed to screen4

Screened

13,107

35

Pc = 0.0027

0.0007

21%

0.79

1,429

Control

13,113

44

Pc = 0.0034





Group

Number of Patients

Deaths from breast cancer

Probability of death from breast cancer

Absolute risk reduction1

Relative risk reduction2

Odds ratio3

Number needed to screen4

Screened

13,107

35

Pc = 0.0027

0.0007

21%

0.79

1,429

Control

13,113

44

Pc = 0.0034





Women in the intervention group were invited to attend mammographic screening at intervals of 18-24 months. Five rounds of screening were completed. Breast cancer was treated according to stage at diagnosis. Mean follow-up was 8.8 years.

  1. Absolute risk reduction: Pc - Ps
  2. Relative risk reduction: (Pc - Ps) ÷ Pc
  3. Odds ratio: [Ps ÷ (1 - Ps)] ÷ [Pc ÷ (1 - Pc)]
  4. Number needed to screen: 1 ÷ (Pc - Ps)

Source of number of patients and deaths from breast cancer: Andersson 1988.

HRQL measures can be used to determine the effects of a technology on patients, to compare alternative technologies for their effects on patients with a particular problem or disability, or to compare different technologies' respective abilities to improve the quality of life of patients with different problems. Reflecting, in part, the need to demonstrate the effectiveness of many new technologies for chronic conditions such as rheumatoid arthritis, migraine, and depression, considerable advances have been made in the development and validation of these measures in the last 25 years. Box 6 shows dimensions of general HRQL measures that have been used extensively and that are well validated for certain applications. Box 7 shows aspects of selected disease-specific HRQL measures.

Quality-Adjusted Life Years

A unit of health care outcome that combines gains (or losses) in length of life with quality of life is the quality-adjusted life year (QALY). QALYs represent years of life subsequent to a health care intervention that are weighted or adjusted for the quality of life experienced by the patient during those years (Torrance and Feeny 1989). QALYs provide a common unit for multiple purposes, including: estimating the overall burden of disease; comparing the relative impact of specific diseases, conditions, and health care interventions; and making economic comparisons, such as of the cost-effectiveness (in particular the cost-utility) of different health care interventions. Health economists have proposed setting priorities among alternative health care interventions by selecting among these so as to maximize the additional health gain in terms of QALYs. This is intended to optimize allocation of scarce resources and thereby maximize social welfare. Other units that are analogous to QALYs include disability-adjusted life years (DALYs) and healthy-years equivalents (HYEs). As a group, these types of measures are sometimes known as health-adjusted life years (HALYs) (Gold 2002; Johannesson 1993; Mehrez and Gafni 1993; World Development Report 1993).

The scale of quality of life used for QALYs can be based on general HRQL indexes or other methods of eliciting patient utility for certain states of life. This dimension is typically standardized to a scale ranging from 0.0 (death) to 1.0 (perfect health). A scale may allow for ratings below 0.0 for states of disability and distress that some patients consider to be worse than death (Patrick 1994). QALYs can be useful for making comparisons among alternative technologies because they are generic units that can reflect changes brought about by different health care interventions for the same or different health problems. Box 8 shows how QALYs were used to compare the cost utilities of three alternative therapies for end-stage heart disease. Box 9 lists the cost utility of different interventions for different health problems according to the amount of money that must be invested per QALY gained. The CEA Registry is a continuously updated, detailed set of standardized cost-utility analyses, including tables of cost-utility ratios for many types of health care interventions [https://research.tufts-nemc.org/cear4/default.aspx].

Certain methodological aspects and the proposed use of QALYs or similar units in setting health care priorities remain controversial (Arnesen 2000; Gerard and Mooney 1993; Mason 1994; Nord 1994; Richardson 1994; Ubel 2000). Research on public perceptions of the value of health care programs indicates that health gain is not necessarily the only determinant of value, and that the rule of maximizing QALYs (or similar measures) per health expenditure to set priorities may be too restrictive, not reflecting public expectations regarding fairness or equity. For example, because people who are elderly or disabled may have a lower "ceiling" or potential for gain in QALYs or other measure of HRQL than other people would for the same health care expenditure, making resource allocation decisions based on cost-utility is viewed by some as inherently biased against the elderly and disabled.

Box 6
Domains of Selected General Health-Related Quality of Life Indexes

 

Sickness Impact Profile (Bergner 1981; de Bruin 1992)

This table lists the domains of the sick impact profile index.

• Body care and movement

• Emotional behavior

• Ambulation

• Alertness behavior

• Mobility

• Communication

• Sleep and rest

• Social interaction

• Home management

• Work

• Recreation and pastimes

• Eating

Nottingham Health Profile (Doll 1993; Jenkinson 1988)

This table lists thd domains of the Nottingham health profile index.

• Physical mobility

• Energy

• Pain

• Social isolation

• Sleep

• Emotional reactions

Quality of Well-Being Scale (Kaplan 1988; Kaplan 1989)

This table lists the domains of the Quality of Well-Being Scale Index.

• Symptom-problem complex

• Physical activity

• Mobility

• Social activity

Functional Independence Measure (Bunch 1994; Linacre 1994)

This table lists the domain of the Functional Independence Measure Index.

• Self-care

• Locomotion

• Sphincter control

• Communication

• Mobility

• Social cognition

Short Form (SF)-36 (McHorney 1994; Ware 1992)

This table lists the domains of the Short Form (SF)-36 index.

• Physical functioning

• General mental health

• Role limitations due to physical problems

• Role limitations due to emotional

problems

• Social functioning

• Vitality

• Bodily pain

• General health perceptions

EuroQol Descriptive System (Essink-Bot 1993; EuroQol Group 1990)

This table lists the domains of the EuroQol descriptive system index.

• Mobility

• Pain/discomfort

• Self-care

• Anxiety/depression

• Usual activities

 

Activities of Daily Living (Katz 1970; Lazaridis 1994)

This table lists the domains of the Activities of daily living index.

• Bathing

• Mobility

• Dressing

• Continence

• Toileting

• Eating

 

Box 7
Domains of Selected Disease-Specific Health-Related Quality of Life Indexes

New York Heart Association Functional Classification (O'Brien 1993; van den Broek 1992)

Class I: Patients with cardiac disease but without resulting limitations of physical activity. Ordinary physical activity does not cause undue fatigue, palpitation, dyspnoea, or anginal pain.

Class II: Patients with cardiac disease resulting in slight limitation of physical activity. They are comfortable at rest. Ordinary physical activity results in fatigue, palpitation, dyspnoea or anginal pain.

Class III: Patients with cardiac disease resulting in marked limitation of physical activity. They are comfortable at rest. Less than ordinary physical activity causes fatigue, palpitation, dyspnoea or anginal pain.

Class IV: Patients with cardiac disease resulting in inability to carry on any physical activity without discomfort. Symptoms of cardiac insufficiency or of anginal syndrome may be present even at rest. If any physical activity is undertaken, discomfort is increased.

Arthritis Impact Measurement Scales (Kazis 1989; Meenan 1992)

This table lists the domains of the Arthritis Impact Measurement Scales Index.

• Mobility

• Social activities

• Walking and bending

• Support from family and friends

• Hand and finger function

• Arthritis pain

• Arm function

• Work

• Self care

• Level of tension

• Household tasks

• Mood

Visual Functioning (VF)-14 Index (Steinberg 1994)

  • reading small print, such as labels on medicine bottles, a telephone book, or food labels
  • reading a newspaper or book
  • reading a large-print book or newspaper or the numbers on a telephone
  • recognizing people when they are close to you
  • seeing steps, stairs, or curbs
  • reading traffic, street, or store signs
  • doing fine handwork such as sewing, knitting, crocheting, or carpentry
  • writing checks or filling out forms
  • playing games such as bingo, dominos, card games, or mahjong
  • taking part in sports such as bowling, handball, tennis, or golf
  • cooking
  • watching television
  • daytime driving
  • nighttime driving

 

Box 8
Cost-Utilities for Alternative Therapies for End-Stage Heart Disease

 

Therapy

Life years gained (yr)

Mean utility

QALY (yr)

Aggregate cost ($)

Cost per QALY ($/yr)

Conventional medical treatment

0.50

0.06

0.03

$28,500

$950,000

Heart transplantation

11.30

0.75

8.45

$298,200

$35,290

Total artificial heart

4.42

0.65

2.88

$327,600

$113,750

Notes: Costs and outcomes discounted at three percent per year; 20-year horizon. Mean utilities derived using time-tradeoff method on scale for which 1.0 was well, 0.0 was death, and states worse than death were valued between 0.0 and -1.0.

This table indicates that, although the cost of conventional medical treatment is the lowest, its cost per QALY is the highest, as the life-years gained and the patient utility of those years are low compared to the alternatives. The costs of heart transplantation and total artificial heart are of similar magnitude, but the cost per QALY is much lower for heart transplantation, as the life-years gained and the patient utility of those years are higher compared to the total artificial heart.

Source: Hogness 1991.

Box 9
Cost per QALY for Selected Health Care Technologies

 

Cost per QALY

(£ 1990)

 

Cholesterol testing and diet therapy (all 40-69 yrs)

220

Neurosurgery for head injury

240

General practitioner advice to stop smoking

270

Neurosurgery for subarachnoid hemorrhage

490

Antihypertensive therapy to prevent stroke (45-64 yrs)

940

Pacemaker implantation

1,100

Hip replacement

1,180

Valve replacement for aortic stenosis

1,140

Cholesterol testing and treatment

1,480

Coronary artery bypass graft surgery (left main disease, severe angina)

2,090

Kidney transplant

4,710

Breast cancer screening

5,780

Heart transplantation

7,840

Cholesterol testing and treatment (incremental) (all 25-39 yrs)

14,150

Home hemodialysis

17,260

Coronary artery bypass graft surgery (one-vessel disease, moderate angina)

18,830

Continuous ambulatory peritoneal dialysis

19,870

Hospital hemodialysis

21,970

Erythropoietin for dialysis anemia (with 10% reduction in mortality)

54,380

Neurosurgery for malignant intracranial tumors

107,780

     

 

Erythropoietin for dialysis anemia (with no increase in survival) 126,290 This table ranks selected procedures for a variety of health problems according to their cost utility, (i.e., the amount of money that must be spent on each procedure to gain one more QALY). There were some methodological differences in determining costs and QALYs among the studies from which these results were derived. Nonetheless, giving considerable latitude to these figures, the range in the magnitude of investment required to yield the next QALY for these treatments is great. This type of "bucks for the bang" (here, British pounds for the QALY) analysis helps to illustrate implicit choices made in allocating scarce health care resources, and suggests how decision makers might move toward reallocating those resources if societal gain in net health benefits (e.g., as measured using QALYs) is used as an allocation criterion.

Source: Maynard 1991.

Some work has been done recently to capture more dimensions of public preference and to better account for the value attributed to different health care interventions (Dolan 2001; Schwappach 2002). HRQL measures and QALYs continue to be used in HTA while substantial work continues in reviewing, refining and validating them.  

Performance of Diagnostic Technologies

The relationships between most preventive, therapeutic, and rehabilitative technologies and health outcomes can be assessed as direct cause and effect relationships. The relationship between the use of diagnostic and screening technologies and health outcomes is typically indirect, as these technologies provide information that may be used to inform providers concerning the use of interventions that may in turn affect health outcomes.

Many tests and other technologies used for diagnosis are also used for screening, and most of the concepts discussed here for diagnostic technologies pertain as well to screening technologies. A basic difference between screening and diagnosis is that diagnosis is done in symptomatic patients and screening is typically done in asymptomatic patient groups. For a given test used for either screening or diagnosis, this difference has a great effect on the probability that a patient has a disease or other health condition.

The immediate purpose of a diagnostic test is to provide information about the presence (and, less often, the extent) of a disease or other health condition. That is, the diagnostic test should be able to discriminate between patients who have a particular disease and those who do not have the disease (or discriminate among different extents of disease in a given patient).

The technical performance of a diagnostic test depends on a number of factors. Among these are the precision and accuracy of the test, the observer variation in reading the test data, and the relationship between the disease of interest and the cutoff level of the marker or surrogate used in the diagnostic test to determine the presence or absence of that disease. These factors contribute to the ability of a diagnostic test to detect a disease when it is present and to not detect a disease when it is not present.

The marker for a disease or condition is typically defined as a certain cutoff level of a variable such as blood pressure (e.g., for hypertension), glucose level (e.g., for diabetes), or prostate specific antigen level (e.g., for prostate cancer). Disease markers have distributions in non-diseased as well as in diseased populations. For most diseases, these distributions overlap, so that a single cutoff level does not clearly separate non-diseased from diseased people. For instance, in the case of hypertension, a usual marker for the disease is diastolic blood pressure, the cutoff level of which is often set at 95mm Hg. In fact, some people whose diastolic blood pressure is above 95mm will not be hypertensive (false positive result), and some people with diastolic blood pressure below 95mm will be hypertensive (false negative result). Lowering the cutoff to 90mm will decrease the number of false positives, but increase the number of false negatives.

A diagnostic test can have four basic types of outcomes, as shown in Box 10. A true positive diagnostic test result is one that detects a marker when the disease is present. A true negative test result is one that does not detect the marker when the disease is absent. A false positive test result is one that detects a marker when the disease is absent. A false negative test result is one that does not detect a marker when the disease is present.

Box 10
Possible Outcomes of Diagnostic Tests

 

"This tables explains the possible outcomes of negative and positive test results.

Test Result

 

Disease Status

 

Present

Absent

Positive (+)

True +

False +

Negative (-)

False -

True -

Operating characteristics of diagnostic tests and procedures are measures of the technical performance of these technologies. These characteristics are based on the probabilities of the four possible types of outcomes of a diagnostic test. The two most commonly used operating characteristics of diagnostic tests are sensitivity and specificity. Sensitivity measures the ability of a test to detect disease when it is present. Specificity measures the ability of a test to correctly exclude disease in a non-diseased person. One graphical way of depicting these operating characteristics for a given diagnostic test is with a receiver operating characteristic (ROC) curve, which plots the relationship between the true positive ratio (sensitivity) and false positive ratio (1 - specificity) as a function of the cutoff level of a disease (or condition) marker. ROC curves help to demonstrate how raising or lowering the cutoff point for defining a positive test result affects tradeoffs between correctly identifying people with a disease (true positives) and incorrectly labeling a person as positive who does not have the condition (false positives).

Taken alone, sensitivity and specificity do not reveal the probability that a given patient really has a disease if the test is positive, or the probability that a given patient does not have the disease if the test is negative. These probabilities are captured by two other operating characteristics. Predictive value positive is the proportion of those patients with a positive test result who actually have the disease. Predictive value negative is the proportion of patients with a negative test result who actually do not have the disease. (See Box 11.) Unlike sensitivity and specificity, predictive value positive and predictive value negative are not constant performance characteristics of a diagnostic test; they change with the prevalence of the disease in the population of interest. For example, if a disease is very rare in the population, even tests with high sensitivity and high specificity can have low predictive value positive, generating more false-positive than false negative results.

Beyond technical performance of diagnostic technologies, the effect of diagnostic technologies on health outcomes or health-related quality of life is less obvious than for other types of technologies. As health care decisionmakers increasingly demand to know how health care interventions affect health care outcomes, diagnostic technologies will have to demonstrate their efficacy/effectiveness accordingly.

The efficacy (or effectiveness) of a diagnostic technology can be determined along a chain of inquiry that leads from technical capacity of a technology to changes in patient health outcomes to cost effectiveness, as follows.

  1. Technical capacity. Does the technology perform reliably and deliver accurate information?
  2. Diagnostic accuracy. Does the technology contribute to making an accurate diagnosis?
  3. Diagnostic impact. Do the diagnostic results influence use of other diagnostic technologies, e.g., does it replace other diagnostic technologies?
  4. Therapeutic impact. Do the diagnostic findings influence the selection and delivery of treatment?
  5. Patient outcome. Does use of the diagnostic technology contribute to improved health of the patient?
  6. Cost effectiveness. Does use of the diagnostic technology improve the cost effectiveness of health care compared to alternative interventions?

Box 11
Operating Characteristics of Diagnostic Tests

 

"This table shows the operating characteristics and their corresponding formulas and definitions.

Characteristic

Formula

Definition

Sensitivity

True Positives

Proportion of people with

 

True positives + False negatives

condition who test positive

Specificity

True Negatives

Proportion of people without

 

True negatives + False positives

condition who test negative

     

Predictive value

True Positives

Proportion of people with positive

positive

True positives + False positives

test who have condition

Predictive value

True Negatives

Proportion of people with negative

negative

True negatives + False negatives

test who do not have condition

If a diagnostic technology is not efficacious at any step along this chain, then it is not likely to be efficacious at any later step. Efficacy at a given step does not imply efficacy at a later step (Feeny 1986; Fineberg 1977; Institute of Medicine 1985). Box 12 shows a hierarchy of studies for assessing diagnostic imaging technologies that is consistent with the chain of inquiry noted above. Some groups have developed standards for reporting studies of the accuracy of diagnostic tests (Bossuyt 2003).

For diagnostic technologies that are still prototypes or in other early stages of development, there are limited data upon which to base answers to questions such as these. Even so, investigators and advocates of diagnostic technologies should be prepared to describe, at least qualitatively, the ways in which the technology might affect diagnostic accuracy, diagnostic impact, therapeutic impact, patient outcomes and cost effectiveness; how these effects might be measured; approximately what levels of performance would be needed to successfully implement the technology; and how further investigations should be conducted to make these determinations.

Types of Organizations That Conduct HTA

The types of organizations that undertake some form of HTA include:

  • Regulatory agencies
  • Government and private sector payers
  • Managed care organizations
  • Health professions organizations
  • Standards setting organizations
  • Hospitals and health care networks
  • Group purchasing organizations
  • Patient and consumer organizations
  • Government policy research agencies
  • Private sector assessment/policy research organizations
  • Academic health centers
  • Biomedical research agencies
  • Health product companies
  • Venture capital groups and other investors

Box 12
Hierarchical Model of Efficacy for Diagnostic Imaging:
Typical Measures of Analysis

Level 1. Technical efficacy

Resolution of line pairs
Modulation transfer function change
Gray-scale range
Amount of mottle
Sharpness

Level 2. Diagnostic accuracy efficacy

Yield of abnormal or normal diagnoses in a case series
Diagnostic accuracy (% correct diagnoses in case series)
Sensitivity and specificity in a defined clinical problem setting
Measures of area under the ROC curve

Level 3. Diagnostic thinking efficacy

Number (%) of cases in a series in which image judged "helpful" to making the diagnosis
Entropy change in differential diagnosis probability distribution
Difference in clinicians' subjectively estimated diagnosis probabilities pre- to post-test information
Empirical subjective log-likelihood ratio for test positive and negative in a case series

Level 4. Therapeutic efficacy

Number (%) of times image judged helpful in planning management of patient in a case series
% of times medical procedure avoided due to image information
Number (%) of times therapy planned before imaging changed after imaging information
obtained (retrospectively inferred from clinical records)
Number (%) of times clinicians' prospectively stated therapeutic choices changed after
information obtained

Level 5. Patient outcome efficacy

% of patients improved with test compared with/without test
Morbidity (or procedures) avoided after having image information
Change in quality-adjusted life expectancy
Expected value of test information in quality-adjusted life years (QALYs)
Cost per QALY saved with imaging information
Patient utility assessment; e.g., Markov modeling; time trade-off

Level 6. Societal efficacy

Benefit-cost analysis from societal viewpoint
Cost-effectiveness analysis from societal viewpoint

Source: Thornbury 1992.

The purposes, scope, methods, and other characteristics of HTAs that are conducted or sponsored by these organizations vary widely. Examples of these organizations are noted in this document. As in other fields, professional societies and organizational consortia exist in HTA. At the international level, HTA International (HTAi) [www.htai.org] has members from HTA agencies, academic institutions, health professions, hospitals and other health care providers, payers, industry, and others from more than 40 countries. The International Network of Agencies for Health Technology Assessment (INAHTA) [www.inahta.org] is a network of about 40 organizations (including government agencies and non-profit private sector organizations) that generate a shared HTA report database and engage in related collaborative activities. Examples of other professional organizations whose interests include areas related to HTA include:

Expertise for Conducting HTA

Given the variety of impacts addressed and the range of methods that may be used in an assessment, multiple types of experts are needed in HTA. Depending upon the topic and scope of assessment, these may include a selection of the following:

  • Physicians, nurses, dentists, and other clinicians
  • Managers of hospitals, clinics, nursing homes, and other health care institutions
  • Radiology technicians, laboratory technicians and other allied health professionals
  • Biomedical and clinical engineers
  • Pharmacologists
  • Patients and patient affairs representatives
  • Epidemiologists
  • Biostatisticians
  • Economists
  • Lawyers
  • Social scientists
  • Ethicists
  • Decision scientists
  • Computer scientists/programmers
  • Librarians/information specialists

Certain individuals have expertise in more than one area. The set of participants in an assessment depends upon its purpose, available resources and other factors. For example, the standing members of a hospital technology assessment committee might include: the chief executive officer, chief financial officer, physician chief of staff, director of nursing, director of planning, materials manager and director of biomedical engineering (Sadock 1997; Taylor 1994). Physician specialists and marketing, legal, patient affairs and additional analytical support staff could be involved as appropriate.

Ten Basic Steps of HTA

There is great variation in the scope, selection of methods and level of detail in the practice of HTA. Nevertheless, most HTA activity involves some form of the following basic steps.  

  1. Identify assessment topics
  2. Specify the assessment problem
  3. Determine locus of assessment
  4. Retrieve evidence
  5. Collect new primary data (as appropriate)
  6. Appraise/interpret evidence
  7. Integrate/synthesize evidence
  8. Formulate findings and recommendations
  9. Disseminate findings and recommendations
  10. Monitor impact

Not all assessment programs conduct all of these steps, and they are not necessarily conducted in a linear manner. Many HTA programs rely largely on integrative methods of reviewing and synthesizing data from existing primary data studies (reported in journal articles or from epidemiological or administrative data sets), and do not collect primary data. Some assessment efforts involve multiple cycles of retrieving/collecting, interpreting, and integrating evidence before completing an assessment. For example, to gain regulatory approval (e.g., by the US FDA) to market a new drugs, pharmaceutical companies typically sponsor several iterations of new data collection: preclinical testing in the laboratory and in animals and phase I, II, and III studies in humans; additional phase IV post marketing studies may be a condition of approval. The steps of appraising and integrating evidence may be done iteratively, such as when a group of primarily data studies are appraised individually for quality, then are integrated into a body of evidence, which in turn is appraised for its overall quality. Depending upon the circumstances of an HTA, the dissemination of findings and recommendations and monitoring of impact may not be parts of the HTA itself, although they may be important responsibilities of the sponsoring program or parent organization.  

Another framework for HTA is offered by the European Collaboration for Health Technology Assessment (Busse 2002), as follows.

  • Submission of an assessment request/identification of an assessment need
  • Prioritization
  • Commissioning
  • Conducting the assessment
    • Definition of policy question(s)
    • Elaboration of HTA protocol
    • Collecting background information/determination of the status of the technology
    • Definition of the research questions
    • Sources of data, appraisal of evidence, and synthesis of evidence for each of:
      • Safety
      • Efficacy/effectiveness
      • Psychological, social, ethical
      • Organizational, professional
      • Economic
    • Draft elaboration of discussion, conclusions, and recommendations
    • External review
    • Publishing of final HTA report and summary report
  • Dissemination
  • Use of HTA
  • Update of the HTA

As indicated by various chapter and section headings, all ten of the basic steps of HTA listed above are described in this document.


Previous Section Next Section Table of Contents NICHSR Home Page