NLM logo


HTA 101: X. Selected Issues in HTA

A. Barriers to HTA

Although the general trend in health care is toward an increased role of HTA, improved HTA processes, greater transparency of HTA, and greater involvement of stakeholders in HTA, various countervailing forces to HTA remain.  Some stakeholders may oppose HTA due to their inherent interests, some of which may conflict with evidence-based processes.  Innovators seek to maintain incentives to develop new technologies and lower hurdles to market access.  Investors in health care products and services seek high, short-term returns on their investments.  Health technology companies seek to maximize profit through increased sales of their products at high prices.  Physicians and other clinicians seek to maintain their decision-making autonomy regarding ordering tests, performing procedures, prescribing therapies, and making referrals to other clinicians.  Managers of hospitals and other health care facilities seek to improve their competitive status in their markets.  Patient interest groups seek to retain unlimited choice of treatment options (Fineberg 2009).  Among the barriers to HTA are the following: 

Technological imperative.  Particularly in the US and other wealthy countries, there is a “technological imperative” comprising an abiding fascination with technology, the expectation that new is better, and the inclination to use a technology that has potential for some benefit, however marginal or even poorly substantiated (Deyo 2002).  Some argue that the increased potential of technology only raises the imperative to conduct HTA (Hoffman 2002). 

Limited resources for HTA.  As is so for other efforts, resources for HTA are limited.  Although some HTA programs and certain HTA findings are nationally or internationally recognized, the resources allocated for HTA, even in the wealthy nations, are vanishingly small compared to national health care spending.

Insufficient primary data.  Lack of primary studies and other data sources limits the evidence base for HTA.  This also includes lack of local or regional data to conduct HTA pertaining to a particular nation, region, or health care facility and lack of data pertaining to particular population subgroups for which a technology may be applicable. 

Timing misalignment.  The timing of HTA may be misaligned with decision-making needs and other events.  This may arise in delays in requesting or allocating funding for an HTA, ongoing adoption and use of technologies while HTA is being conducted, delays between release of HTA findings and their adoption in policy and practice, and the “moving target problem,” in which the relevance of HTA findings is diminished by changes in technologies, their comparators, or how they are used.   

Prestigious proponents of technology.  The opinions of highly regarded or powerful proponents or “champions” of adopting a technology may prevail, even in the absence of credible supporting evidence. 

Marketing.  Increasingly effective and targeted marketing and promotion of health technologies, including short courses sponsored by health care product companies to train physicians in using these products and direct-to-consumer advertising (where this is permitted) can weigh against HTA findings. 

Financial incentives.  Health care systems that reimburse hospitals, physicians, and other providers every time a technology is provided, i.e., “fee-for-service” health care, tend to increase the volume of technology use, even when supporting evidence is lacking.  Hospitals and physician groups that have invested in major capital equipment and supporting infrastructure such as for diagnostic radiology, radiation oncology, and robotic surgery, have incentives to use these technologies (Garrison 2011; Jacobs 2013).  Also, patients with little or no exposure to costs tend to seek more health care.  These financial incentives can contribute to the inertia of existing payment systems that reward use of technologies despite lack of supporting evidence and related policies and clinical practice guidelines. 

Political actions.  HTA may be circumvented by political actions, often prompted by “lobbying” or “pressure groups.”  This occurs, for example, when laws are passed to mandate (or eliminate) coverage by government or private sector payers for certain technologies, in contrast to findings based on available evidence, or in the absence of rigorous evidence. 

Implementation barriers.  There are various barriers to implementing some HTA findings and recommendations by decision makers and policymakers for whom HTA reports are intended.  Among these are:  lack of access to HTA reports, complex and technical formats of HTA reports, questionable data quality, absence of real-world applications, and narrow focus (Henshall 2002).  HTA findings and recommendations may be difficult to implement given clinicians’ and other providers’ reluctance to change long-standing practice routines, inertia of existing payment policies, and rapidly outdated education and training in some instances.  Insufficient opportunity or encouragement for scientific inquiry and skepticism in clinical education contributes to this inertia.  Implementation of HTA findings also may be limited due to practical external constraints, such as when adopting a new technology requires a particular environment (e.g., special shielded rooms, instrumentation, and related facilities for diagnostic and therapeutic procedures using ionizing radiation), professional training, or other resources that are unavailable in a particular facility. 

As noted above, political forces can circumvent evidence-based processes (Fletcher 1997).  One of the main applications of HTA is informing coverage policies.  While many payers use HTA findings to inform coverage policies, they are also subject to laws in their respective countries, states, provinces, and other jurisdictions.  Legislative bodies at these levels can mandate that health programs provide certain services.  In the US, historical examples of technologies that have been mandated by Congress for the Medicare program or by state legislatures for public or private payers have included autologous bone marrow transplant with high-dose chemotherapy (ABMT-HDC) for advanced breast cancer, bone densitometry screening for osteoporosis, screening mammography, prostate cancer screening, and treatment for temporomandibular joint disorder.  Such mandates, including the ones noted here, were not based on the types of evidence-based methods preferred in HTA, including some instances where the evidence was sufficient for some populations with a given disease or risk factors but not for others.  These mandates were influenced by pressure groups representing certain patients, physicians, health product makers, and others (Deyo 1997; Sheingold 1998). 

In some instances, legislative mandates arise through frustration with slowed or delayed HTA processes.  A notable instance was the mandate by the US Congress for Medicare coverage of dual energy x-ray absorption (DEXA) for bone mineral density measurement, which had been subject to an assessment involving two federal agencies over a seven-year period (Lewin Group 2000).  Mandating coverage of a technology, rather than subjecting it to HTA, can mask more complex clinical consequences.  As noted above, in the 1990s, many health plans in the US reluctantly agreed to cover ABMT-HDC in response to state legislative mandates brought about by intensive political pressure, and the threat of litigation (legal action in courts).  It was not until 1999, after tens of thousands of women were subjected to the procedure, that results of five well-conducted RCTs, along with revelations of falsified data by a researcher who reported early findings in support of the procedure, demonstrated that the procedure provided no benefit compared to standard-dose treatment for breast cancer, and caused unnecessary suffering in some women (Berger 1999; ECRI 1995; Mello 2001; Sharf 2001). 

B. Quality of Care and HTA

HTA is an important source of information for quality of care.  Although a thorough discussion of this relationship is not within the scope of this document, the following are some definitions and fundamental relationships concerning these concepts.

Quality of care is a measure or indicator of the degree to which health care is expected to increase the likelihood of desired health outcomes and is consistent with prevailing standards of health care.  HTA and quality assurance are distinct yet interdependent processes that contribute to quality of care.

HTA generates findings that add knowledge about the relationship between health care interventions and outcomes.  This knowledge can be used to develop and update a range of standards and guidelines for improving health care quality, including clinical practice guidelines, manufacturing standards, clinical laboratory standards, adverse event reporting, architecture and facility design standards, and other criteria, practices, and policies regarding the performance of health care.  Participation of clinicians, and particularly opinion leaders, in HTA and in developing evidence-based clinical practice guidelines can improve the acceptance and adoption of those guidelines and, thereby, quality of care. 

The purpose of quality assurance activities is to ensure that the best available knowledge concerning the use of health care to improve health outcomes is properly used.  It involves the implementation of health care standards, including activities to correct, reduce variations in, or otherwise improve health care practices relative to these standards.  Continuous quality improvement (CQI) and total quality management (TQM) (Gann 1994; Wakefield 1993) are among the systematic approaches to implementing quality assurance that have been adapted for hospitals and other health care institutions.  Such approaches include, for example, the identification of “best practices” and the use of benchmarking to develop improved clinical pathways or disease management for medical and surgical procedures, administrative operations, etc. (Kim 2003; Kwan 2003; Pilnick 2001).  For example, CQI has been evaluated in a multicenter RCT as a means improve the adoption of two process-of-care measures for coronary artery bypass graft surgery (CABG):  preoperative β–blockade therapy and internal mammary artery grafting (Ferguson 2003).  Notably, in this RCT, the intervention being tested was not those two health care interventions, but CQI. 

Quality assurance involves a measurement and monitoring function, i.e., quality assessment.  Quality assessment is, primarily, a means for determining how well health care is delivered in comparison with applicable standards or acceptable bounds of care.  These standards or bounds may be grouped according to the structure of care (e.g., institutional, professional, and physical characteristics), the process of care (content or nature of the health care delivered) and the outcomes of care (health status and well-being of patients) (Donabedian 1988).  Increasingly, quality assurance involves studies of effectiveness data, including health outcomes and the determinants of those outcomes from the perspectives of clinicians, patients, administrators, and policymakers (McDonald 2000).  In detecting these differences between how well health care is delivered and applicable standards, quality assessment can also call attention to the need for further HTA or other investigations. 

In summary, HTA is among the sources of knowledge used to set standards for health care, and quality assurance is used to determine the extent to which health care providers adhere to these standards (Lohr 1990; Lohr and Rettig 1988).  Major reorganization of health care systems may be required to ensure that stronger evidence is generated systematically for setting standards of care, and that standards of care are broadly implemented (Institute of Medicine, Committee on Quality of Health Care in America 2001).

C. Comparative Effectiveness Research and HTA

Comparative effectiveness research (CER) is the generation and synthesis of evidence comparing the benefits and harms of alternative technologies to prevent, diagnose, treat, and monitor diseases and other health care conditions in “real-world” settings in order to improve the delivery of health care (Federal Coordinating Council on Comparative Effectiveness Research 2009; Institute of Medicine 2009).  The purpose of CER is to strengthen the evidence base that is used for assisting patients, other consumers, clinicians, health care managers, policymakers and others to make more informed health care decisions for individuals and populations. 

Various attributes of what is known today as CER have been incorporated into research on the impact of health care technologies over the last several decades or more.  Indeed, CER draws selected attributes from such inquiries and evaluations as RCTs, HTA, outcomes research, effectiveness research, and evidence based medicine.   The emergence of CER as an explicit, coherent field of research in the early 2000s arose from a growing recognition of such factors as:

  • Evidence of inappropriate use of health care technologies, including over-use, under-use, and improper use
  • Evidence of large variations (geographic and other) in practice
  • Insufficiency of evidence developed for market approval/clearance by regulatory agencies (e.g., FDA) to also support clinical and policy decisions; this is typically because such evidence:
    • emphasizes efficacy rather than effectiveness
    • is often not derived from controlled clinical trials (especially for many medical devices)
    • often lacks active comparators (e.g., where placebos are the only controls)
    • tends to exclude certain patient populations (those who are elderly, have multiple co-morbidities, etc.)
    • often is not derived from studies that enable subgroup analyses
  • Insufficiently rigorous or absent evidence for the many technologies that are not subject to oversight by regulatory agencies (e.g., surgical procedures)
  • Lack of evidence from “head-to-head” comparisons of alternative interventions for particular health problems
  • Lack of evidence in “real-world” practice (efficacy vs. effectiveness)
  • Continued steep increases in health care costs prompting interest in more efficient care delivery

The main attributes of CER are:

  • Direct (“head-to-head”) comparisons of alternative interventions (rather than comparison with placebo or indirect comparisons)
  • Applies to all types of technologies
  • Measures effectiveness in real-world populations and health care settings
  • Emphasizes health care outcomes (e.g., morbidity, mortality, symptoms, quality of life, adverse events) rather than surrogate or other intermediate endpoints
  • Draws on variety of complementary research methods, data sources, and analytical tools
  • Enables subgroup analyses to yield findings about different responses across patient types
  • Includes emphasis on priority diseases and priority populations


In the US, there is no consensus on the role of economics, such as determining value for money, in the conduct of CER itself.  Indeed, there are legislative constraints on the use of CER and cost-effectiveness analysis in making coverage decisions for the Medicare program and other health programs administered by the Department of Health and Human Services (Neumann 2012; Pearson 2010).  Even so, the findings of CER and other evidence that may involve health economic analyses, such as cost-effectiveness analysis, is available for use by various analysts and decision makers.  While the term “CER” is most often used in the US, other countries and regions use related, though not necessarily synonymous terms.  For example, in Europe, there has been growing interest in “relative efficacy” and “relative effectiveness” trials for new drugs.  This interest derives in part from differing evidence requirements that sometimes arise between regulatory agencies, particularly the European Medicines Agency (EMA), and payment authorities.  Approaches under consideration involve identifying circumstances in which pre-licensing (pre-marketing) efficacy trials should use active comparators, and ways to close gaps between efficacy and effectiveness, including whether to conduct pre-licensing or post-licensing practical clinical trials (with more heterogeneous patient populations in real-world settings), or to better understand how such extrinsic factors as physician prescribing or patient adherence affect variability in drug response (Eichler 2009; Glaeske 2012).  These proposals may affect the relative roles and relationships of regulators, payers, and HTA agencies. 


CER is generating more of certain types of evidence that have been of increasing relevance to HTA.  Among the broad set of attributes or impacts assessed in HTA, those that CER emphasizes are effectiveness and safety in real-world patients and health care settings, patient outcomes, and direct comparisons to standards of care.  Further, CER is contributing to tools and methods that will broaden the scope and strengthen the quality of available evidence, including the development of better health outcome and quality of life measures, observational data sources such as registries and insurance claims, and alternative designs for clinical trials.  As such, when HTA develops evidence questions, conducts systematic literature searches, conducts meta-analyses and other evidence syntheses, and develops findings and recommendations, it will draw on an evidence base enriched by CER findings.

D. Patient-Centered Outcomes Research and HTA

HTA is directly and indirectly related to certain patient-oriented concepts, including patient-centered care, patient-centered outcomes, patient-reported outcomes, and patient-centered outcomes research.

The term “patient-centered medicine” was introduced more than 40 years ago by Balint and colleagues (Balint 1970) in contrast to what they called “illness-centered medicine.”  Whereas “traditional diagnosis” was based on disease-centered thinking, an “overall diagnosis” involved an understanding of the patient’s complaints based on patient-centered thinking (Stewart 2013).  A contemporary definition of patient-centered care is:

The experience (to the extent the informed, individual patient desires it) of transparency, individualization, recognition, respect, dignity, and choice in all matters, without exception, related to one’s person, circumstances, and relationships in health care (Berwick 2009).

The main dimensions of patient-centered care include:

  • Disease and illness experience (patient-as-person)
  • Whole person (biopsychosocial perspective)
  • Common ground (sharing power and responsibility)
  • Patient-doctor relationship (therapeutic alliance) (Hudon 2011; Mead 2000; Stewart 2013)

As a particular form or variant of health care, patient-centered care can have a favorable impact on safety, effectiveness, and related health care outcomes, although the evidence to date of its impact is mixed (Street 2012; Zandbelt 2007).  As such, patient-centered care can be a topic of HTA. 

Patient-centered care can be evaluated using such measures as:

  • Patient Perception of Patient-Centeredness (PPPC): patient perceptions of patient-centered care during the last visit with a family physician; uses 14 items
  • Consultation Care Measure (CCM): patients’ perceptions of patient-centered care during the last visit with a family physician; 5 subscales: communication and partnership, personal relationship, health promotion, positive and clear approach to the problem, interest in effect on life
  • CAHPS (Consumer Assessment of Healthcare Providers and Systems):  brief general measure comparing overall quality of interpersonal care across health care settings; includes some patient-centered care domains:  access (getting care quickly, getting needed care), provider communication (Epstein 2011; Hudon 2011)

In addition, patient-centeredness may be considered a dimension of health care quality (Berwick 2009), i.e., a standard attribute of practice. 

Patient-centered outcomes (or patient-oriented outcomes) comprise a subset of the broad array of health outcomes noted above (mortality, morbidity, adverse events, quality of life, etc.).  Patient-centered outcomes refer to outcomes that patients experience across the variety of real-world settings, including:  survival, functional status, quality of life, quality of death, symptoms, pain, nausea, psychosocial well-being, health utility (patient-perceived value of particular states of health), and patient satisfaction.  Excluded are outcomes or other endpoints that patients do not experience directly, e.g., blood pressure, lipid levels, bone density, viral load, or cardiac output.  Patient-centered outcomes can be assessed at a generic level or a disease/condition-specific level.  Examples of generic instruments for assessing patient-centered outcomes include:

There are patient-centered outcomes instruments for such diseases and conditions as angina, asthma, epilepsy, kidney disease, migraine, multiple sclerosis (MS), and vision.  For example, standard outcomes for assessing MS activity and severity include annualized relapse rate and the Expanded Disability Status Scale.  Patient-centered outcomes instruments that may better reflect patient experience with MS include MS-specific instruments and hybrid instruments (i.e., generic instruments with additional MS items).  Examples of MS-specific patient centered outcome instruments are:  the Multiple Sclerosis Quality of Life Inventory (MSQLI), Multiple Sclerosis Quality of Life-54 scale (MSQoL-54), Functional Assessment of Multiple Sclerosis (FAMS), Multiple Sclerosis Impact Scale (MSIS-29), and Leeds Multiple Sclerosis Quality of Life scale (LMSQoL) (Miller 2010). 

Patient-reported outcomes (PROs) are those patient-centered outcomes that are self-reported by patients (or reported on their behalf by their caregivers or surrogates) or obtained from patients by an interviewer without interpretation or modification of the patient’s response by other people, including clinicians (Rothrock 2010; US FDA 2009).  Patient ratings of health status, adverse events, functional status, quality of life, and other outcomes may differ from measures recorded by clinicians (see, e.g.,  Ebell 2004; Nakamura 2012), and patients may weigh benefits and risks differently among themselves as well as differently than clinicians when considering treatment alternatives (Wong 2012).  Non-patient reported outcomes are those that matter to patients but are not reported directly by them. 

Patient-centered outcomes research (PCOR) generates evidence comparing the impact of health care on patient-centered outcomes.  PCOR can draw on a wide variety of methods, including, e.g., practical or pragmatic RCTs, cluster randomized trials, and other trial designs; registries, insurance claims data, systematic reviews, and others (Methodology Committee 2012).  As described by the recently established Patient-Centered Outcomes Research Institute (PCORI) in the US, PCOR answers these patient-focused questions:

  • Given my personal characteristics, conditions and preferences, what should I expect will happen to me?
  • What are my options and what are the benefits and harms of those options?
  • What can I do to improve the outcomes that are most important to me?
  • How can clinicians and the care delivery systems they work in help me make the best decisions about my health and healthcare? (Patient-Centered Outcomes Research Institute 2013)

In summary, HTA can assess patient-centered care as a type of intervention.  HTA can assess the impact of health care interventions on patient-centered outcomes, including patient-reported outcomes, of the full range of health technologies.  Using a variety of primary and secondary data methods, patient-centered outcomes research generates data and evidence that can be used in HTA.

E. Personalized Health Care and HTA

Clinical trials and other studies that report only average treatment effects may be misleading.  Therapies that yield a statistically significant treatment effect across a study population may not necessarily work for all treated patients; they may be ineffective for some patients and harmful for others.  Also, therapies that do not yield a statistically significant treatment effect across a study population―and that may be dismissed as ineffective―may work for certain subsets of the population.

Personalized health care (a broader concept that personalized medicine) refers to the tailoring of health care to the particular traits (or circumstances or other characteristics) of a patient that influence response to a heath care intervention.  These may include genetic, sociodemographic, clinical, behavioral, environmental, and other personal traits, as well as personal preferences.  Personalized health care does not mean the creation of interventions that are unique to a patient; rather, it recognizes differences in how patient subgroups respond to particular interventions, and uses that information to treat individual patients. 

Some examples of technologies used in personalized health care are:

  • CYP2C9 and VKORC1 genetic testing for warfarin anticoagulation response for patients with atrial fibrillation, mechanical heart valves, deep vein thrombosis, etc.
  • HER-2/neu receptor testing for trastuzumab for breast cancer
  • BRCA 1,2 testing for pharmaceutical and surgical prevention options for and surveillance for breast cancer
  • KRAS testing for use of EGFR inhibitors (e.g., cetuximab, panitumumab) for colon cancer
  • Socioculturally-tailored therapy to treat certain ethnic minority patients with diabetes and depression (Ell 2011)
  • Alternative procedure techniques (gastric banding, gastric bypass, etc.) for bariatric (morbid obesity) surgery
  • Alternative types of coronary artery revascularization (e.g., coronary artery bypass graft surgery, percutaneous coronary interventions) for symptomatic ischemic coronary artery disease
  • Alternative regimens to treat infertility

In order to support personalized health care, information is needed about how alternative technologies affect not just the average patient with a given health problem, but how those technologies affect various subgroups of patients with that health problem.  For example, more specific information about how response differs by age, sex, comorbidities (other health problems), or genetic traits can be used by patients with a certain health problem who share those characteristics.  Heterogeneity of treatment effects (HTEs) refers to the variation in patient responses observed across levels or types of patient characteristics such as these (Kravitz 2004). 

HTA is increasingly oriented to identifying, assembling, and evaluating evidence on HTEs.  This applies to all types of technologies, including screening, diagnostic, therapeutic, palliative, and others.  Deriving findings about effects on subgroups depends in large part on the availability of data from studies that have been designed to detect such subgroup differences.  This depends not only on how well the study population represents various subgroups, but whether the study designs and methods of subgroup analysis are capable of detecting HTEs for the patient subgroups of interest.  For example, prospective subgroup analyses (i.e., identification of subgroups prior to rather than after data collection) tend to be more reliable than retrospective ones, and sample sizes for the subgroups under study must be large enough to detect true subgroup differences where they exist (Oxman 1992; (Wang 2007)  Meta-analyses and other integrative methods may be used to pool subgroup data from different studies.  HTA can also help to strengthen the evidence base for personalized health care by encouraging the development, validation, and use of patient-centered (including patient-reported) outcome measures; involvement of patients in planning clinical trials; and use of alternative data sources, such as health services utilization data (including from insurance claims) and patient registries, to help identify potential subgroup effects.

F. Patient and Consumer Involvement in HTA

The field of HTA increasingly recognizes the importance of involving patients and other “non-professional” or “non-expert” individuals.  Indeed, these individuals often have developed expertise about their disease, condition, and care.  Their involvement can provide perspectives and experiences that are not available from others, but are integral to HTA.  Their participation in HTA also helps to fulfill HTA good practices for openness, transparency, and accountability.  The value placed on patient participation and input should be commensurate with that of other participants in HTA. 

In this context, the terms “patients” and “patient representatives” may also be used to mean patient advocates, consumer representatives, family members, non-professional caregivers, citizens, and the public.  These individuals and groups may have different interests and bring different perspectives.  While recognizing these differences, this section generally uses the broadly inclusive term “patients” to refer to one or more of these groups.  Also, the terms “involvement,” “engagement,” and “participation” of patients are sometimes used interchangeably, although some users consider the term “involvement” to be more inclusive or integral to the HTA process than the other terms.  This section uses all of those terms, and “involvement” is used broadly to encompass various levels of inclusiveness in the HTA process.

In most countries, patient involvement in HTA is still emerging, with considerable variation in how patients are recruited and supported for involvement, the HTA processes in which they are involved, the role they have in determining HTA findings, and how potential conflicts of interest are addressed (Kreis 2013).  Across HTA organizations, patients are participating in more aspects of HTA, including various combinations of the following (see, e.g., Facey 2010; HTAi Patient and Citizen Involvement Interest Sub-Group, Good Practice Examples 2012):

  • Servings as members of HTA boards, committees, and workgroups
  • Identifying potential topics for HTA
  • Early identification of potential target groups for HTA reports
  • Setting priorities among HTA topics
  • Identifying health outcomes and other impacts (economic, social, etc.) to be assessed
  • Reviewing proposals or bids by outside organizations/contractors to conduct HTAs
  • Submitting evidence for HTAs
  • Reviewing draft HTA reports and recommendations
  • Helping to design and prepare patient-friendly HTA report summaries
  • Dissemination of HTA findings to policymakers, patient groups, and other target groups
  • Evaluating the use of HTA recommendations

Patients are also increasingly involved in helping to set research agendas and design clinical trials and other primary data collection (e.g., selecting health outcomes and other endpoints for study that are of particular relevance to patients). 

A systematic review of 142 studies published through 2011 of patient engagement in health care research more broadly found that most patient engagement occurred in agenda setting and protocol development, with less occurring during the conduct and translation of research.  The review reported that patient engagement improved clinical trial enrollment and retention, design of study protocols, selection of relevant outcomes, and securing research funding.  This review found no studies comparing methods of patient engagement.   The main challenges identified included the time and funding to implement patient engagement and concerns by some stakeholders that patient engagement is “tokenistic,” i.e., not genuine (Domecq 2014). 

According to a 2013 report of an international web-based survey, a majority of HTA organizations have formal public involvement or engagement activities, most of which involve lay (i.e., non-professional) representatives of organized groups or, to a lesser extent, individual patients, consumers, and community members.  Most public involvement or engagement activities have entailed communication or consultation with those members of the public, while some activities have entailed direct participation in the form of citizens’ juries, consensus councils, and similar groups (Whitty 2013). 

Through direct involvement in the HTA process, patients can share experiences, perspectives, and evidence that complement or even contradict evidence from such traditional sources as clinical trials and clinicians’ observations.  Certainly, direct patient involvement can improve the understanding and accuracy of patients’ needs and preferences.  Further, patient experience, perspectives, and evidence might address the burden of a disease or condition on patients, family members, and caregivers, including dimensions of health, pain, quality of life, social and economic burdens.  Such patient input might also address the impacts on patients and others of health technologies, such as benefits and harms compared to alternative therapies, ability to work, difficulties in access or adherence, impact on their life, and economic impact, including any out-of-pocket costs, associated travel costs, income loss, and caregiver costs.  Formal patient evidence can be obtained via qualitative or quantitative means through primary data collection or secondary analyses (Bastian 2011; Danner 2011; Facey 2010; Health Equality Europe 2008).

An analysis published in 2009 of a random selection of HTA reports (published in English or Scandinavian languages) from INAHTA-member agencies found that about one-third addressed patient-related issues (i.e., concerning patient views on the use or impact of technologies).  These mainly addressed psychological issues such as patient worry and discomfort, ethical issues such as patient acceptance of the technology, social issues such as impact on patient daily life, effect on quality of life, and patient information.  There was great variation in the comprehensiveness and methodological transparency of assessment of patient-related matters; only a minority of reports had explicit questions about patient issues (Lee 2009). 

One of the more comprehensive efforts to gain patient input is that used for technology appraisals of the UK National Institute for Health and Care Excellence (NICE).  Patients and caregivers provide input on:  the personal impact of their condition; outcomes patients want the technology to improve; the impact of a technology on health outcomes, symptoms, physical and social functioning, quality of life; impact on family, friends and employers; ease of use, side effects and costs of the technology; patient preferences; subgroups who might benefit more or less from the technology; challenges to professional or researcher views; and areas needing further research (HTAi Patient and Citizen Involvement Interest Sub-Group, Good Practice Examples 2012; NICE Guide to the Methods 2013).  Among other forms of patient and citizen input, NICE has a Citizens Council of 30 members that is demographically representative of the UK.  The Citizens Council provides NICE with public perspectives on moral and ethical issues of which NICE takes account when producing its guidances (NICE Citizens Council 2014).

The Consumer Network of the Cochrane Collaboration involves health care consumers in the production of Cochrane systematic reviews.  The most common role is consumer referee, which involves, e.g., commenting on protocols (plans) for new Cochrane systematic reviews, commenting on Cochrane reviews prior to publication, and helping to prepare or provide consumer perspectives on the plain language summaries of Cochrane reviews.  Consumers may also help to search journals, disseminate information about particular Cochrane reviews, co-author reviews, and translate reviews and plain language summaries (Cochrane Consumer Network 2014). 

Based on recent survey findings from HTA organizations, some of the main challenges to patient engagement in HTA include:  lack of organizational capacity to involve patients, extensive time requirements, lack of agreed-upon methods for involving patients in HTA, lack of clarity about when in the HTA process to involve patients, lack of funding, and concerns regarding the scientific credibility of patient evidence (European Patients’ Forum; undated). 

Greater experience with patient involvement in HTA and information sharing about these are leading to good practices for involvement of patients as members of boards, committees, and working groups as well as involvement and communications with patient groups and the broader public.  These emerging good practices address such areas as those shown in Box X-1.  While it may be impractical for most HTA programs to adopt all or even most of these immediately, those programs seeking to benefit from greater patient involvement can adopt more of them as time and resources permit.  Also, greater involvement of patients and other target populations is gradually extending to development of evidence-based clinical practice guidelines, including to incorporate information about PROs, values, and preferences (see, e.g., Basch 2011; Brouwers 2010). 

The HTAi Interest Sub-Group on Patient and Citizen Involvement in HTA has developed a set of resource materials, including examples of good practices for patient and citizen involvement from various HTA agencies, video and slide presentations introducing HTA and patient roles in it, a glossary, answers to frequently-asked questions, and others (see:

Box X-1. Emerging Good Practices for Patient Involvement in HTA

  • Ensure that all HTA processes are appropriately accessible, transparent, fairly applied, documented, and accountable
  • Gain commitment of HTA organization to patient involvement
  • Establish, review, and update policies for patient involvement in HTA
  • Establish a dedicated unit or staff function for patient involvement, with explicit channels of communication and personal points of contact in the HTA organization
  • Plan and provide for adequate budgets and other resources for patient involvement
  • Conduct broadly accessible, transparent, and equitable (including relative to other types of participants) recruitment of patients to HTA boards, committees, and working groups
  • Provide training and education for patient involvement in HTA, including appropriate materials, handbooks, reviewer/evaluation forms, etc.
  • Provide training for HTA staff and other participants on working with patients and groups that represent them
  • Ensure appropriately equitable rights (e.g., in voting) and responsibilities for patient members of HTA boards, committees, and working groups
  • Provide adequate notice of meetings, due dates for input, and related scheduling information to patient participants to enable their sufficient preparation and timely participation
  • Inform the variety of patient groups and other potentially interested groups about planned and ongoing HTA reports and ways to provide input to these reports
  • Communicate in plain, patient-appropriate language, including in background materials and other documentation to ensure that patients are adequately informed for participation in HTA
  • Provide patient participants with adequate financial support, including fees commensurate with those for other participants and reimbursement for travel and other out-of-pocket expenses (e.g., for child and other dependent care)
  • Ensure that the context for patient involvement is welcoming and is not intimidating or coercive
  • Make reasonable accommodations to ensure accessibility to HTA processes for patients and other participants who may have activity limitations (or are otherwise differently-abled)
  • Provide HTA reports (and/or summaries of these) and other outputs in plain, patient-appropriate language and format, including for users with sensory (e.g., visual or hearing) limitations
  • Provide for active, targeted dissemination of HTA reports and related products to patient groups
  • Provide links from HTA organization web sites to those of relevant patient groups
  • Organize special/targeted events, communications, or other activities for engaging individual patients and patient groups
  • Seek to involve patient groups that are traditionally difficult to reach (e.g., ethnic minorities, underserved populations, stigmatized populations)
  • Document/report patient involvement in HTA, including, e.g., levels of participation, impact on HTA deliberations, findings, and recommendations
  • Provide recognition and feedback to patients regarding their participation

Sources: European Patients’ Forum (undated); Health Equality Europe 2008; HTAi Patient and Citizen Involvement Interest Sub-Group PIE Good Practice Principles 2012, Good Practice Examples 2013.


G. Rapid HTA

In recent years, the demand for HTA by health care decision makers has increasingly involved requests for faster responses to help inform emergent decisions.  This has led to development of “rapid HTAs” (or “rapid assessments” or “rapid reviews”).  Among the reported purposes of rapid HTAs are to inform coverage decisions, capital funding, formulary decisions, treatment referrals, guideline formulation, and indications for further research (Hailey 2009).  For these, HTA organizations generate more focused assessments in the course of, e.g., four-to-eight weeks.  Rapid HTAs offer a tradeoff:  provide less-than-comprehensive and less certain information in time to act on a decision versus comprehensive and more certain information when the opportunity to make an effective decision may have passed.  In addition to the shorter completion time, rapid HTAs can differ from full HTAs in such ways as:  limiting scope to fewer types of impact or evidence questions, focusing searches on fewer bibliographic databases, relying on fewer types of studies (e.g., only systematic reviews or only RCTs), use of shorter and more qualitative syntheses with categorization of results without meta-analyses, and more limited or conditional interpretation of findings or recommendations (Khangura 2012). 

Despite the increased use of rapid HTAs on diverse topics and a growing literature on their responsiveness and utility for decision makers, there is no consensus regarding methodology or guidance for rapid HTAs (Harker 2012).  EUnetHTA has a version of rapid HTA that focuses on four of the nine total domains of its HTA core model, i.e., health problem and current use of technology, description and technical characteristics, safety, and clinical effectiveness (EUnetHTA Joint Action WP5 2013).  The Ottawa Hospital Research Institute has developed an eight-step approach to rapid reviews, including:  needs assessment, question development and refinement, proposal development and approval, systematic literature search, screening and selection of studies, narrative synthesis of included studies (including assignment of evidence levels), report production, and ongoing follow-up and dialog with knowledge users (Khangura 2012).  Given the need to tailor rapid HTAs to the particular needs and time constraints of decision-makers, providing transparency of various rapid HTA methodologies is likely to be more important that achieving consensus on any standard approach (Watt 2008).

H. Decentralization of HTA

Although technology assessment originated as a mostly centralized function conducted by government agencies and other national- or regional-level organizations, HTA evolved into a more decentralized function, conducted by a wide variety of organizations in the public and private sectors (Goodman 1998; Rettig 1997).  As noted above, an HTA done from a particular perspective may not serve the policymaking needs of other perspectives.  Even for the same technology or clinical problem, there can be widely different assessment needs of regulatory agencies, health technology companies, hospitals, payers, physicians, policymakers, and others. 

The growth in decentralized HTA activity has arisen less from a reduction in the level of centralized activity than expansion of HTA programs for particular decision-making needs.  In the US, for example, there remain multiple government centers with ongoing HTA or other technology evaluation responsibilities to fulfill particular purposes, e.g., FDA regulation of drugs, biologics, and medical devices; CMS coverage policies for the Medicare program; and the Effective Health Care Program, Technology Assessment Program, and US Preventive Services Task Force of AHRQ.  There has been considerable expansion in activities elsewhere, particularly in the private sector, as well as greater reliance by centralized sources on HTA inputs from outside sources.  Large health care provider institutions and major health care product companies have established groups or units devoted to “technology assessment,” “pharmacoeconomics,” “clinical effectiveness,” and “health economics and outcomes research,” and related areas.  More health plans (including various managed care organizations and insurance companies) have established formal programs to assess new pharmaceuticals, procedures, and other technologies in support of payment decisions.  The number and magnitude of private firms and university centers involved in HTA and related technology evaluation functions continue to increase.  HTA and evidence-based clinical practice guideline committees (with various names) are now common among medical specialty and subspecialty societies.  Hospital networks, managed care organizations and other large health care providers in the private sector have HTA programs to support acquisition and management of pharmaceuticals (e.g., P&T committees and formularies), equipment and other technologies, and other technology-related needs throughout their systems (Kaden 2002). 

Aside from the growth of HTA in the private sector, even HTA conducted by government agencies is drawing upon more decentralized resources.  In the US, the FDA has long relied on advisory panels comprising outside experts to examine clinical trial findings and other evidence to provide recommendations regarding market approval of new drugs, biologicals, and medical devices.  CMS has a Medicare Evidence Development & Coverage Advisory Committee (MEDCAC) of independent experts that provides findings and recommendations to CMS pertaining to the quality of available evidence to help inform national coverage policies for health technologies, based on the clinical literature, and HTAs and “evidence reports” prepared by selected AHRQ Evidence-based Practice Centers. 

AHRQ’s Evidence-based Practice Centers (EPC) program has contracts with 11 EPCs, most of which are affiliated with academic health centers, that generate HTAs and evidence reports in support of clinical practice guidelines, coverage policies, and other practices and policies.  Some EPC reports are conducted at the request, via AHRQ, from CMS and other government agencies; other requests are made by other organizations in the private sector, such as health professional organizations.  In this manner, AHRQ provides a portal for decentralized HTA, via the EPCs, on behalf of government and non-government organizations.  While the AHRQ EPC program is a decentralized model, it also benefits from collaborative efforts to develop and use standardized approaches, such as for evidence appraisal and other aspects of conducting systematic reviews (see, e.g., Methods Guide for Effectiveness 2014).  As noted above, AHRQ also administers the US Preventive Services Task Force, an independent panel of experts in primary care and prevention that systematically reviews evidence of effectiveness and develops recommendations for a broad range of clinical preventive services. 

The Cochrane Collaboration, another highly decentralized model for evaluating health care, involves 53 workgroups of volunteer experts from more than 100 countries coordinated through 14 centers with additional branches based in about 30 countries that conduct systematic reviews of a diverse variety of health care interventions.  As do the AHRQ EPCs, the Cochrane Collaboration develops standardized approaches to conducting systematic reviews and related methods development (Higgins 2011).

Decentralization of HTA and related functions widens the expertise available to HTA and brings broader perspectives to the process and diminishes or balances potential conflicts of interest.  Together, these generally add to the credibility of HTA processes and findings, and diminish concerns that assessments reflect narrow or self-serving interests of a particular agency or organization. 

Certain changes in the health care market are prompting greater balance between centralized and decentralized HTA.  Hospital networks, large managed care systems and other large systems such as the US Department of Veterans Affairs (VA) continually seek to build economies of scale and buying leverage for health care products, ranging from drugs to surgical gloves to hip joint implants.  With HTA units that are centralized yet responsive to needs of individual facilities, these large organizations can consolidate their HTA efforts and support system-wide acquisition of health technologies and related services.

As health care providers and payers understand the resource requirements for conducting evaluations of health care technologies, they consider the respective benefits of conducting their own HTAs and subscribing to HTA reports from outside assessment groups.  Assessment requirements vary widely depending on the type of technology involved.  Acquisition of commodity products such as most types of syringes, surgical gloves, and generic drugs is based largely on price, whereas acquisition of the latest deep-brain stimulator for movement disorders such as Parkinson’s disease requires a more considered evaluation of safety, effectiveness, cost, patient preferences, and other attributes.  Nearly all hospitals and health care networks in the US rely on group purchasing organizations (GPOs) that use economies of scale to acquire most of their products.  The large GPOs have their own technology evaluation or clinical review committees that examine available evidence on technologies such as implantable cardiac defibrillators and MRI units, whose acquisition depends on factors other than price alone.  In turn, many GPOs also subscribe to HTA report services (Lewin Group 2002; Hu 2012).

I. Locus of Assessment: Make or Buy?

Health care decision makers can “make or buy” HTAs.  The nature of an assessment problem will affect the determination of the most appropriate organization to conduct it.  A comprehensive HTA addressing multiple attributes of a technology can be very resource intensive, requiring considerable and diverse expertise, data sources, and other resources. 

Some ministries of health and national health services, major insurance companies, health plans, and integrated health systems have their own internal HTA programs.  For example, in a large hospital or health plan, this might include a core staff and a multidisciplinary HTA committee representing major clinical departments, nursing, pharmacy, allied health, biomedical engineering.  This committee might interact with other committees such as pharmacy and therapeutics (P&T), strategic planning, and capital planning committees (Kaden 2002). 

Other organizations rely on HTA reports acquired from organizations that have devoted functions or otherwise specialize in HTA.  As noted above, AHRQ commissions evidence reports, HTAs, and other reports from its EPC, most of which are affiliated with academic medical centers.  In the UK, NICE commissions one internal center and four external centers (the  National Clinical Guideline Centre, National Collaborating Centre for Cancer, National Collaborating Centre for Women’s and Children’s Health, and National Collaborating Centre for Mental Health) to produce its clinical guidelines.  NICE also commissions designated academic centers to develop assessments/evidence reports in support of its various types of technology appraisals and guidance.

Other US-based vendors of HTA include, e.g., Blue Cross and Blue Shield Association Center for Clinical Effectiveness (CCE) (formerly the Technology Evaluation Center), Cochrane Collaboration, ECRI Institute, Hayes Inc., Institute for Clinical and Economic Review (ICER), Center for Medical Technology Policy (CMTP), Institute for Clinical Systems Improvement (ICSI), and the Oregon Drug Effectiveness Review Project (DERP).  Depending upon the producing HTA organization, these reports may be available at no cost, for members only, on a subscription basis, or for a specific price per report. 

Determining the responsibility for sponsoring or conducting an assessment depends on the nature of the problem, financial resources available, expertise of available personnel, time constraints, and other factors.  For any assessment, an organization must determine to what extent it will conduct the HTA itself or commission it from other sources.  Some organizations commission selected components of an HTA, such as evidence retrieval and synthesis, and perform the other steps in-house. 

One of the advantages of requesting or commissioning an outside group to conduct HTAs is to gain an independent, outside view in instances where a requesting agency might have a perceived conflict of interest.  Thus, a major health care payer might seek an HTA from an outside group to inform its coverage decision about a costly new technology in order to diminish perceptions of a potential bias regarding its willingness to pay for the use of the technology. 

Factors that influence the "make or buy" decision include the following (Goodman, Snider 1996).

  • Is an existing assessment available?  If an existing assessment is available, does it address the specific assessment problem of interest, including the technology or intervention, patient population, and impacts of interest?  Does it have a compatible perspective?  Is the assessment still current?  Is the methodology used sufficiently credible?  Is the report worth its price?
  • If an existing assessment needs to be updated or is not available, do people in the organization have the time and expertise to perform the required data collection and analyses?  If a synthesis of existing information is needed, does the organization have database searching capabilities and staff to review and interpret the literature?  If new data are needed, does the organization have the requisite resources and expertise?
  • What methodology will be used?  If, for example, a consensus development approach is involved, does that consensus need to incorporate and reflect the opinions of an organization's own clinicians?  Will local clinicians accept the results and report recommendations if they do not participate in the assessment?

J. Underused Technologies and HTA

When used properly, HTA can result in the reduction or elimination of the use of technologies that are not safe and effective, or whose costs are too high relative to their benefits.  Less attention is given to the ability of HTA to identify technologies that are underused and to determine why they are underused (Asch 2000; McNeil 2001).  Underuse is prevalent in preventive, acute, and chronic care; applies to care for children, youth, and adults; and contributes to tens of thousands of deaths and billions of dollars of losses to the economy and unnecessary health care costs (GAO 2002; Mangione-Smith 2007; McGlynn 2003).  According to one estimate, replacing more costly, non-superior health care with proven cost-effective care would save approximately one-fourth of US health care spending (Fuchs 2011). 

Underuse of some proven therapies affects large proportions of at-risk populations.  For example, there is high-quality evidence that smoking cessation interventions, including nicotine replacement therapy, the antidepressant bupropion, and counseling, are safe, effective, and cost effective (Anderson 2002; Foulds 2002; Stead 2008; Woolacott 2002).  However, in Europe, North America, and elsewhere, these interventions are used far less than is indicated.  Two further examples in the US are management of hypertension and childhood vaccinations, as follows. 

Based on data from the US National Health and Nutrition Examination Survey (NHANES) 2003-2010, the Centers for Disease Control and Prevention (CDC) estimates that the prevalence of hypertension among US adults was 30.4%, or about 66.9 million people.  (Hypertension was defined as an average systolic blood pressure ≥140 mmHg or an average diastolic blood pressure ≥90 mmHg, or currently using blood pressure-lowering medication.)  Among those with hypertension, 53.5% did not have their hypertension controlled.  Among these people, 39.4% were not aware of their hypertension, 15.8% were aware of their hypertension but were not receiving medication for it, and 44.8% were aware of their hypertension and were being treated with medication (CDC 2012).  Hypertension accounted for an estimated $131 billion in direct US health care costs in 2010, including its contribution to costs of complications associated with other cardiovascular conditions (Heidenreich 2011). 

Vaccination rates in the US remain highly variable by vaccine type and age group, although these rates have generally increased in recent years.  For example, according to 2012 National Immunization Survey data of the CDC, vaccination rates (with minimum appropriate number of doses) for children 19-35 months old were at or near the US objective of 90% for poliovirus (92.8%); measles, mumps, and rubella (90.8%); varicella (90.2%), and hepatitis B (89.7%).  Rates were lower for other vaccines, including diphtheria, tetanus and pertussis (82.5%); pneumococcal conjugate vaccine (81.9%); Haemophilus influenzae b (80.9%); and hepatitis B birth dose (71.6%).  Rates were low for hepatitis A (53.0%, vs. target of 85%), and rotavirus (68.6%, vs. target of 80%).  Rates of vaccination coverage were generally lower for children living below poverty level, including 6.0% to 9.5% lower for several of these vaccination types (CDC, National, state, and local area vaccination coverage among children, 2013).

Underuse of particular technologies is attributed to various reasons.  For smoking cessation, these reasons include:  lack of insurance coverage, concerns about short-term costs without regard to cost-effectiveness in the short-term (e.g., for pregnant women and infants) or the long-term; lack of smoker awareness of effective interventions; insufficient demand by patients, physicians, and the tobacco-control community; and the influence of the tobacco industry on policymaking (Schauffler 2001).  Available evidence suggests, for example, that greater insurance coverage of smoking cessation interventions increases the likelihood that smokers will stop smoking (Reda 2012). 

Rates of underuse can be highly variable in a country or region.  In the instance of pediatric vaccinations in the US, underuse is affected by such factors as differences in socioeconomic status, payment mechanisms, and shortages of certain vaccines (CDC, National, state, and local area vaccination coverage among children, 2013; CDC, National and state vaccination coverage among adolescents, 2013; Hinman 2006).  In developing countries, the major cause of vaccine shortages is lack of resources to purchase them (Hinman 2006).

Box X-2. Some Underused Health Care Technologies (US)

  • ACE inhibitors for treatment of heart failure
  • ACE inhibitors for prevention of renal deterioration in insulin-dependent diabetics
  • Ambulation aids (canes, crutches, walkers)
  • Antibiotics for gastrointestinal ulcers
  • Beta blockers for survivors of acute myocardial infarction
  • Cholesterol-lowering drugs for patients at risk of coronary artery disease
  • Childhood vaccinations
  • Cochlear implants for severe-to-profound deafness
  • Colorectal cancer screening
  • Corticosteroid inhalants for treating asthma
  • Corticosteroid therapy for fetuses at risk of preterm delivery
  • Depression diagnosis and treatment
  • Diabetic retinopathy screening
  • Hemoglobin A1c testing every 6 months in diabetic patients
  • Hepatitis B virus vaccination of infants
  • HIV testing and treatment with antiretroviral therapy
  • Hypertension management
  • Implantable cardioverter-defibrillators for survivors of cardiac arrest
  • Incontinence diagnosis and treatment
  • Influenza immunization
  • Inhaled corticosteroids in adults with asthma
  • Intraocular pressure screening for glaucoma
  • Mammography (especially women age 50+)
  • Oral rehydration therapy for dehydrated children
  • Organ transplantation
  • Pain management
  • Pap smears
  • Pneumococcal vaccine for high-risk patients
  • Prenatal care
  • Smoking cessation interventions
  • Thrombolytic therapy for acute myocardial infarction
  • Thrombolytic therapy for ischemic stroke

Box X-2 shows examples of health technologies for which good evidence exists of safety, effectiveness, and cost-effectiveness, but that are used significantly less than is indicated, even where they are affordable.  Although this list applies primarily to the US, many of these technologies are underused elsewhere in North America, Western Europe, and other wealthy countries.

The reasons that worthy technologies are underused are diverse, and include the following.

  • Lack of awareness on the part of patients, physicians, and others
  • Inadequate or ineffective information dissemination
  • Limited coverage policies (e.g., for preventive services)
  • Limited access to primary care
  • Poor socioeconomic status
  • Fee-for-service payment systems that reward use of high-paying services without regard for cost-effectiveness
  • Concerns about short-term cost without regard for cost savings and cost-effectiveness in the short- and long-terms
  • Inappropriate or unsubstantiated concerns about improper use (e.g., pain therapy)
  • Inconvenience and misperceptions on the part of clinicians, patients, or the public (e.g., false information suggesting that vaccines increase the risk of other disorders)
  • Clinical inertia, i.e., unwillingness to change practice habits
  • Insufficient supply (e.g., organs for transplantation)
  • Health technology companies’ marketing efforts to emphasize brand name products rather than less-expensive technologies that are at least as effective and safe
  • Disproportionate concerns about adverse effects (e.g., warfarin to reduce risk of stroke)
  • Concerns about adherence to therapies by some patient subgroups
  • Fear of stigma (e.g., treatment of mental health disorders)
  • Professional conflicts and “turf battles” on the part of physician specialists, provider institutions, industry, and others

Merely generating better evidence is not enough to meet the decision-making needs of consumers, patients, health care providers, and purchasers. To maximize its impact on the quality and value of health care, these parties must cite and use evidence when making clinical and policy decisions. Disseminating evidence into clinical practice must be accompanied by ongoing evaluation and feedback to decision makers, the key characteristic of a true learning health care system. However, this is not happening consistently.  In a review of adherence to 439 indicators of health care quality for 30 acute and chronic conditions as well as preventive care, McGlynn and colleagues concluded that American adults received only 55% of recommended care (McGlynn 2003). The same study found that children and youth received only 46.5% of recommended care (Mangione-Smith et al., 2007).  

These gaps between evidence and execution underscore the need to identify more effective tools to help patients, providers, and policy makers to use the available evidence.  Those in HTA must consider how implementation of HTA findings and recommendations, including the extent to which technologies may be underused, overused, or inappropriately used, may be affected by such factors as those listed above.  Data suggesting underuse of health technologies known to be safe, effective, and cost-effective may be used to develop priorities for HTA and evidence questions.  Explicit attention to underuse of proven technologies conveys that HTA is concerned with optimal use of proven health technologies, not just reducing use of inappropriate ones or the mistaken view that it seeks to diminish innovation or diffusion of health technology. 

K. Managed Entry and HTA

When HTA is used to inform reimbursement decisions about a new technology (or new application of an existing technology), it may encounter promising, yet non-definitive evidence regarding effectiveness, safety, economic impacts, or other attributes that are important to payers as well as other stakeholders.  In these instances, rather than delaying any coverage until stronger evidence is available, payers may seek an arrangement for managed entry of the technology.  These arrangements offer several forms of potential societal benefit.  They can enable access for certain types of patients for whom existing evidence suggests net health benefit, provide some financial compensation for generating better evidence sooner than in the absence of reimbursement, enable refinement of clinical technique and services delivery, and build expertise and experience among physicians and other providers.

Some government and private sector payers have provided certain forms of coverage for selected “investigational” or “experimental” technologies since the 1990s in order to compile evidence for making more informed coverage policies (Beebe 1997; Brenner 2002; McGivney 1992; Medical Technology Leadership Forum 1999; Sheingold 1998; Wood 2001).  More recently, such forms of “conditional coverage” have been subsumed under the broader term, “managed entry.” 

Intentionally or otherwise, payers have long been providing reimbursement for investigational technologies, including prior to the 1990s in some instances.  In the US, such instances have included reimbursement for many off-label uses of drugs approved for other indications, drugs with “treatment investigational new drug” (IND) designations by the FDA, certain devices being evaluated under investigational device exemptions (IDE) designations by the FDA, certain medical and surgical procedures being provided in qualifying clinical trials, and certain technologies whose coverage has been legislatively mandated (Steinberg, Tunis 1995). 

Managed entry refers to a range of innovative payment approaches that provide patient access under certain conditions.  Three main purposes are to manage:  uncertainty about safety, effectiveness, or cost effectiveness; budget impact; and technology utilization for optimizing performance (Klemp 2011).  In one general framework of managed entry (Carlson 2010), two main types that are based, at least in part, on health outcomes are conditional coverage and performance-linked reimbursement.  Conditional coverage includes coverage with evidence development (CED) and conditional treatment continuation. 

CED, a broad category of managed entry, refers to limited coverage for a new technology in parallel with specified data collection to provide stronger evidence about the effectiveness, safety, or other impacts of the technology of interest to payers.  This enables some patient access to the technology while reducing uncertainty through real-world experience and generating evidence to inform revised coverage policies as well as clinical practice decisions (Trueman 2010).  CED includes coverage “only in research” (i.e., coverage for a technology only when used in members of the payer’s patient population who are participating in clinical trials of the technology) and coverage “only with research” (i.e., coverage for a technology only when also being used contemporaneously in a sample of the payer’s patient population participating in clinical trials of the technology).  Two types of performance-linked reimbursement are outcomes guarantee, where payment (or rebates) depend on achievement of predetermined health outcomes targets, and pattern or process of care, such as demonstrated impact on clinical decisions or patient adherence to prescribed regimens (Carlson 2010). 

Certainly, various forms of managed entry can help to optimize use of a technology.  CED and other forms of managed entry have had various levels of success to date, and face certain practical challenges (Berger 2001; Cooper 2001; Hutton 2007; Miller 2008).  These arrangements can be difficult to negotiate and manage, and further evaluation is needed to assess their ability to achieve their purposes (Klemp 2011).

L. Innovation and HTA

Innovation generally refers to creation or development of something that is new or different.  It is something that addresses the demand of, is accepted or adopted by, or is otherwise beneficial to users or markets.  Innovations usually are in some way claimed to be more effective or otherwise better than existing technologies, thereby offering a meaningful desirable change.  In health care and other fields, meaningful change is true (i.e., statistically significant) and of practical importance (i.e., clinically significant).  To be successful in a market, innovation must be replicable at an acceptable cost. 

Innovation in health care shares attributes of other fields.  Examples of such attributes include:  better (more effective), safer, faster, cheaper, easier to use, smaller, portable, more standardized, more reliable, and more environmentally friendly (“greener”).  In recent years, such attributes as targeted, personalized, less-invasive, and secure (or confidential) have been increasingly sought in health care.  HTA can account for all of these attributes of innovation.  In particular, though, as a form of inquiry intended to support decision making and policymaking, HTA is especially oriented toward discerning improvements in health care outcomes, i.e., “better” and “safer.”  Also, where decision makers and policymakers seek such information, HTA is oriented toward meaningful improvements in health care outcomes per incremental health expenditure.  (In some decision-making contexts, the inverse is of interest, i.e., whether the cost per incremental improvement in health outcomes is acceptable.)  In health care, this is innovation of value (Porter 2010). 

Certainly, HTA is subject to ongoing controversy pertaining to the relative value of investing in socially desirable goals when resources are scarce.  For example, a goal of allocating resources efficiently for the broad population may compete with the goal of allocating resources for those who are most severely affected by ill health.  Such tradeoffs or conflicting goals can affect what is considered to be innovation of value.  The value of innovation may be considered to extend beyond health care impacts to sustaining the progress of science and technology and to employment, balance of trade, and other economic attributes.  These issues pertaining to the value of innovation have arisen in various national HTA programs, such as for NICE in the UK (Green 2010, Kennedy 2009).

HTA could be viewed as yet another hurdle for innovation, beyond proof-of-concept, regulatory approval, and coverage (based merely on whether a technology qualifies as a covered benefit).  Indeed, “new or different” may suffice for those hurdles without HTA; in most major regulatory schemes, a new technology can be approved or cleared for market as long as it functions as labeled, or is efficacious at an acceptable level of safety for a particular patient population, without regard to “value.”  When HTA inquires about tradeoffs of health outcomes and costs, it supports alignment of payment with innovation of value.  Further, it sends signals that may help to improve the efficiency of innovation by focusing efforts on developing and validating innovations that will be valued by purchasers on behalf of clinicians, patients, and provider institutions.  This emphasis is shifting the direction of innovation, including diminishing or eliminating development with little prospect of value as well as creating new opportunities.  Some technologies that would have been “innovative” may no longer be developed because they will not provide sufficient value; their reimbursement outlook will not justify further investment.  At the same time, the ability to demonstrate comparative effectiveness at a lower cost or superiority at an acceptably higher cost can confer market advantages.  As the “blockbuster” model of drugs and other technologies for broad populations (e.g., statins for lowering blood cholesterol and certain antidepressants) fades, there are opportunities to develop technologies targeted to patient subgroups (e.g., identified using genomics) that may merit high prices. 

Some innovations are “disruptive.”  Disruptive innovations alter and even displace existing systems, networks, or markets, and may create new business models and lead to emergence of new markets (Bower 1995).  The disruption may not be inherent in a new technology itself, but in how it is marketed.  Examples in health care are:  diagnostic imaging replacing exploratory surgery, ultrasound replacing certain x-ray imaging, percutaneous coronary interventions substituting for some coronary artery bypass graft surgeries, and gene therapy substituting for some pharmaceutical or biological therapies (Wilson 2012).  Disruptive innovations can present challenges to existing systems of regulation, payment, health care delivery, and professional training.  Consistent with the original scope of technology assessment to consider the broad array of potential impacts of technology, HTA should consider not just whether a new technology is safe, effective, or cost-effective, but how its adoption might affect systems of care; where, how, and by whom care is delivered; and what direct and indirect effects it may have on patients, at-risk populations, use of other technologies, access to care, and broader societal impacts. 

Various forms of process innovation in clinical trials, such as adaptive and Bayesian trial designs, and in manufacturing technologies also contribute to value by increasing standardization, improving product quality, and lowering production costs.  As health technology companies and other innovators better perceive the need to demonstrate the value of their innovations for regulators, payers, clinicians, and other gatekeepers, some are reorganizing and better integrating their technology R&D, validation, and commercialization functions accordingly. 

M. Managing Individual Bias and Conflict of Interest

HTA should seek to ensure that the credibility of its reports is not compromised by any significant biases or conflicts of interest.  Bias and conflict of interest are distinct yet related concepts.

As described in chapter III, bias generally refers to any systematic deviation in an observation from the true nature of an event (e.g., a treatment effect in a clinical trial).  Further, individual bias can refer to factors that might affect one’s interpretation of evidence or formulation of findings and recommendations.  This form of bias has been defined as “views stated or positions taken that are largely intellectually motivated or arise from close identification or association of an individual with a particular point of view or the positions or perspectives of a particular group” (National Academies 2003).  This may include positions taken in public statements, publications, or other media; institutional or professional affiliations; recognition for personal achievement; intellectual passion; political or ideological beliefs; or personal relationships (Knickrehm 2009).  As long as such positions have some recognized scientific or policy-related merit, they need not disqualify a person from participating in research or participating in HTA.  Indeed, to provide for a competent expert review panel or set of advisors for an assessment, it may be useful to represent a balance of potentially biased perspectives. 

Although such stated views or positions are a potential source of bias, they do not necessarily pose a conflict of interest.  However, when an individual also has a significant, directly related interest or role, such as leading a professional society, industry association, or advocacy organization that has taken the same fixed position, this may pose a conflict of interest (National Academies 2003).  Conflict of interest guidelines often address matters of individual bias as well. 

Conflict of interest (or competing interest) refers to “any financial or other interest which conflicts with the service of the individual [person] because it (1) could significantly impair the individual's objectivity or (2) could create an unfair competitive advantage for any person or organization” (National Academies 2003).  Conflict of interest policies typically apply to current, rather than past or expired interests, or possible interests that may arise in the future.  In HTA, a conflict of interest could cause an individual to be biased in interpreting evidence or formulating findings and recommendations.  In most instances, the existence of a significant conflict of interest pertaining to an HTA topic should disqualify an individual from participating in that HTA as a staff person, expert panel member, or consultant.  However, persons with conflicts of interest may provide information to an HTA process, including relevant evidence, background information, other perspectives, or comments on draft reports. 

Biases and conflicts of interest are conditions, not behaviors (Smith 2006; Thompson 1993).  That is, an individual does not have to act on a bias or conflict of interest for it to exist.  The existence of a bias or conflict of interest is reason for an HTA process to address them, e.g., whether to seek a balance of reasonable biases on an expert panel or to disqualify individuals with significant conflicts of interest from participating in an HTA, as appropriate. 

HTA should consider the potential for conflict of interest on at least three levels:

  • Sponsors of clinical trials and other studies that are part of the body of evidence under review
  • Investigators who conducted and reported on the clinical trials and other studies that are part of the body of evidence under review
  • Health technology assessors, including staff members , expert panel members, or other experts involved in reviewing the evidence and making findings and recommendations

1. Sponsors

Health technology companies and other sponsors of primary research typically determine or influence what research is conducted as well as such aspects as designation of the intervention and control treatments, endpoints, and follow-up periods, and whether research results are submitted for publication. 

Financial conflicts of interest are common in clinical trials and other biomedical research.  Industry sponsorship of research has been found to be associated with restrictions on publication and data sharing (Bekelman 2003).  Clinical trials and cost-effectiveness analyses that are sponsored by industry yield positive (favorable) results more often than studies that are funded or conducted by others (Barbieri 2001; Chopra 2003; Friedberg 1999; Jang 2010).  Among the potential reasons for this discrepancy are that industry is more likely to withhold reports of studies with negative results (e.g., that do not demonstrate a treatment effect).  Another is that industry is more likely to sponsor studies (including RCTs) designed to increase the likelihood of positive results, i.e., where there is an expectation that one intervention (e.g., a new drug or diagnostic test) is superior to the alternative intervention (Polyzos 2011).  In the case of RCTs, this latter tendency could undermine the principle of equipoise for enrolling patients in an RCT, although some contend that this principle can be counterproductive to progress in clinical research (Djulbegovic 2009; Fries 2004; Veatch 2007). 

An analysis of clinical trials listed in database found that health technology companies sponsor trials that are largely focused on their own products, while head-to-head comparisons with different active interventions from other companies are rare.  This diminishes the evidence base for assessing the relative benefits and harms of technologies for the same diseases (Lathyris 2010) and is one of the main reasons for the increased interest in comparative effectiveness research. helps protect against publication bias.  Effective July 2005, the International Committee of Medical Journal Editors (ICMJE) established a requirement that, as a condition of consideration for publication, all clinical trials be entered in a public registry (not necessarily that meets specified criteria before the onset of patient enrollment.  As such, a sponsor cannot wait to see the final results of a trial before deciding to submit a manuscript about it to participating journals (International Committee of Medical Journal Editors 2013).

2. Investigators

For study investigators, conflicts of interest may arise from having a financial interest (e.g., through salary support, ongoing consultancy, owning stock, owning a related patent) in a health care company (or one of its competitors) that may be affected by the results of a study or being an innovator of a technology under study.  Investigator conflict of interest is reported to be prevalent among clinical trials in various fields of pharmaceutical therapy and to be associated with a greater likelihood of reporting a drug to be superior to placebo (Perlis 2005).  A systematic review of research on financial conflicts of interest among biomedical researchers found that approximately one-fourth of investigators had industry affiliations, and two-thirds of academic institutions held equity in start-up companies that sponsored research performed at the same institutions (Bekelman 2003).

Investigators with conflicts are more likely to report positive findings.  This may arise from such factors as preferential funding of research that is likely to report positive findings, biased study designs, investigators’ biased interpretation of results, or suppression of negative results (Okike 2008).  As this research often appears in influential, “high-impact” journals, editors have adopted more systematic requirements for disclosure by investigators of their financial interests and the funding sources of studies, and applying greater scrutiny when potential conflicts arise (International Committee of Medical Journal Writers 1993; Kassirer 1993; Lo 2000; Jagsi 2009).  Such requirements also have been applied to economic analyses (Kassirer 1994), although accompanied by controversy regarding whether certain sponsors (e.g., for-profit vs. not-for-profit) or methods (e.g., pharmacoeconomic modeling) are more acceptable than others (Schulman 1995; Steinberg 1995). 

3. Health Technology Assessors

When interpreting the available evidence, health technology assessors should consider the existence of potential conflicts of interest that may have affected the conduct of a study or presentation of results.  In addition, those participating in HTA should be subject to provisions that protect against their own potential conflicts of interest. 

When interpreting evidence, HTA programs should consider information about sponsorship of a study, investigators, or other factors that suggest the potential for conflict of interest.  Studies that are subject to potential conflicts of interest may have to be given less weight or dropped from the body of evidence under consideration. 

For purposes of those conducting or otherwise involved in HTA, INAHTA defines conflict of interest as:

A situation in which the private interests of someone involved in the assessment or evaluation process (e.g. interviewer, rater, scorer, evaluator) have an impact (either positive or negative) on the quality of the evaluation activities, the accuracy of the data, or the results of the evaluation (INAHTA 2006).

Financial conflicts may include holding stock in, serving as a consultant to, or receiving honoraria from health technology companies or other organizations (e.g., medical professional groups) with financial interests in particular medical procedures or other technologies.  Conflicts may be personal, i.e., apply to individuals associated with the HTA program and their immediate family members.   Conflicts may also be non-personal, e.g., financial benefits to one’s organization (e.g., university) or an industry-endowed fellowship held by an individual.  Conflicts may be specific to a given technology or non-specific, such as a different technology made by the same company that makes the one being assessed. 

HTA programs should take active measures, including adoption and implementation of formal guidelines or requirements, to protect against potential conflicts of interest among their managers, analysts, and expert panel members (Fye 2003; Phillips 1994).  Similar measures should apply, as appropriate, to HTA program consultants, contractors, and outside reviewers of draft HTA reports.  For example, as part of its extensive conflict of interest policy, ECRI Institute, a US-based independent nonprofit organization that conducts HTA, examines each employee’s federal income tax return forms after they are filed to ensure that its employees do not own stock shares in medical device or pharmaceutical firms (ECRI Institute 2014).  In addition to minimizing potential conflicts of interest, HTA programs should take active measures to minimize or balance bias among assessment teams and panel members. 

HTA programs may have guidelines regarding when certain types of conflict affecting an individual require withdrawal (recusal) from the assessment process and when disclosure of the conflict is sufficient and participation is still permitted.  This can involve various aspects or stages of HTA, including priority setting of HTA topics, selecting literature and data sources (including confidential versus open access data) for assessment, and preparing the assessment report.  The INAHTA Checklist for HTA Reports includes a question regarding whether an HTA report provides a statement regarding conflict of interest on the part of those who prepared an HTA report or if funding for the HTA was provided by sources other than those responsible for the HTA agency’s usual budget (INAHTA 2007).

References for Chapter X

Anderson JE, Jorenby DE, Scott WJ, Fiore MC. Treating tobacco use and dependence: an evidence-based clinical practice guideline for tobacco cessation. Chest. 2002;121(3):932-41. PubMed

Asch SM, Sloss EM, Hogan C, Brook RH, Kravitz RL. Measuring underuse and necessary care among elderly Medicare beneficiaries using inpatient and outpatient claims. JAMA. 2000;284(18):2325-33. PubMed | PMC free article

Balint M, et al. Treatment or Diagnosis: A Study of Repeat Prescriptions in General Practice. Philadelphia, PA: JB Lippincott; 1970.

Barbieri M, Drummond MF. Conflict of interest in industry-sponsored economic evaluations: real or imagined? Curr Oncol Rep. 2001;3(5):410-3. PubMed

Basch E, Prestrud AA, Hesketh PJ, Kris MG, et al.; American Society of Clinical Oncology. Antiemetics: American Society of Clinical Oncology clinical practice guideline update. J Clin Oncol. 2011;29(31):4189-98. PubMed

Bastian H, Scheibler F, Knelangen M, Zschorlich B, et al. Choosing health technology assessment and systematic review topics: the development of priority-setting criteria for patients' and consumers' interests. Int J Technol Assess Health Care. 2011;27(4):348-56. PubMed

Beebe DB, Rosenfeld AB, Collins N. An approach to decisions about coverage of investigational treatments. HMO Practice. 1997;11(2):65-7. PubMed

Bekelman JE, Li Y, Gross CP.  Scope and impact of financial conflicts of interest in biomedical research: a systematic review. JAMA. 2003;289(4):454-65. PubMed

Berger A. High dose chemotherapy offers little benefit in breast cancer. BMJ. 1999 May 29;318(7196):1440. PubMed

Berger RL, Celli BR, Meneghetti AL, Bagley PH, et al. Limitations of randomized clinical trials for evaluating emerging operations: the case of lung volume reduction surgery. Ann Thorac Surg. 2001;72(2):649-57. PubMed

Berwick DM. What 'patient-centered' should mean: confessions of an extremist. Health Aff (Millwood). 2009;28(4):w555-65. PubMed

Bower JL, Christensen CM.  Disruptive technologies: catching the wave. Harv Bus Rev. 1995;73:43-53.

Brenner M, Jones B, Daneschvar HL, Triff S. New National Emphysema Treatment Trial paradigm of health care financing administration-sponsored clinical research trials: advances and dilemmas. J Investig Med. 2002;50(2):95-100. PubMed

Brouwers MC, Kho ME, Browman GP, Burgers JS, et al.; AGREE Next Steps Consortium. AGREE II: advancing guideline development, reporting and evaluation in health care. J Clin Epidemiol. 2010;63(12):1308-11. PubMed

Carlson JJ, Sullivan SD, Garrison LP, Neumann PJ, Veenstra DL. Linking payment to health outcomes: a taxonomy and examination of performance-based reimbursement schemes between healthcare payers and manufacturers. Health Policy. 2010;96(3):179-90. PubMed

Centers for Disease Control and Prevention (CDC). National and state vaccination coverage among adolescents aged 13 through 17 years — United States, 2012. MMWR Morb Mortal Wkly Rep. 2013;62(34):685-93. PubMed | Publisher online copy

Centers for Disease Control and Prevention (CDC). National, state, and local area vaccination coverage among children aged 19-35 months — United States, 2012. MMWR Morb Mortal Wkly Rep. 2013 Sep 13;62(36):733-40. PubMed | Publisher online copy

Centers for Disease Control and Prevention (CDC). Vital signs: awareness and treatment of uncontrolled hypertension among adults – United States, 2003-2010. MMWR Morb Mortal Wkly Rep. 2012;61(35):703-9. PubMed | Publisher online copy

Based on data from the US National Health and Nutrition Examination Survey (NHANES) 2003-2010. Publisher online copy

Chopra SS. Industry funding of clinical trials: Benefit or bias? JAMA. 2003;290(1):113. PubMed

Cochrane Consumer Network. About the Cochrane Consumer Network (CCNet). Accessed Jan. 2, 2014 at:

Cooper JD. Paying the piper: the NETT strikes a sour note. National Emphysema Treatment Trial. Ann Thorac Surg. 2001; Aug;72(2):330-3. PubMed

Danner M, Hummel JM, Volz F, van Manen JG, et al. Integrating patients' views into health technology assessment: Analytic hierarchy process (AHP) as a method to elicit patient preferences. Int J Technol Assess Health Care. 2011;27(4):369-75. PubMed

Deyo RA. Cascade effects of medical technology. Annu Rev Public Health. 2002;23:23-44. PubMed

Deyo RA, Psaty BM, Simon G, Wagner EH, Omenn GS. The messenger under attack − intimidation of researchers by special-interest groups. N Engl J Med. 1997;336(16):1176-80. PubMed

Domecq JP, Prutsky G, Elraiyah T, Wang Z, et al. Patient engagement in research: a systematic review. BMC Health Serv Res. 2014;14:89. PubMed | PMC free article

Djulbegovic B. The paradox of equipoise: the principle that drives and limits therapeutic discoveries in clinical research. Cancer Control. 2009;16(4):342-7. PubMed | PMC free article

Donabedian A. Quality assessment and assurance: unity of purpose, diversity of means. Inquiry. 1988;25(1):173-92. PubMed

Ebell MH, Siwek J, Weiss BD, Woolf SH, et al. Strength of recommendation taxonomy (SORT): a patient-centered approach to grading evidence in the medical literature. J Am Board Fam Pract. 2004;17(1):59-67. PubMed | Publisher free article

ECRI Health Technology Assessment Information Service. High-dose chemotherapy with autologous bone marrow transplantation and/or blood cell transplantation for the treatment of metastatic breast cancer. Plymouth Meeting, PA : ECRI, 1995.

ECRI Institute. Policies and Mission Statement. Accessed February 1, 2014 at:

Eichler HG, Bloechl-Daum B, Abadie E, Barnett D, et al. Relative efficacy of drugs: an emerging issue between regulatory agencies and third-party payers. Nat Rev Drug Discov. 2010;9(4):277-91. PubMed

Ell K, Katon W, Xie B, Lee PJ, et al. One-year postcollaborative depression care trial outcomes among predominantly Hispanic diabetes safety net patients. Gen Hosp Psychiatry. 2011;33(5):436-42. PubMed PMC free article

Epstein RM, Street RL Jr. The values and value of patient-centered care. Ann Fam Med 2011;9(2):100-3. PubMed | PMC free article

EUnetHTA Joint Action WP5 – Relative Effectiveness Assessment (REA) of Pharmaceuticals – Model for Rapid Relative Effectiveness Assessment of Pharmaceuticals, 1 March 2013– V3.0. Accessed March 8, 2018 at:

European Patients’ Forum. Patient Involvement in Health Technology Assessment. (Undated) Accessed Dec. 1, 2013 at:

Facey K, Boivin A, Gracia J, Hansen HP, et al. Patients' perspectives in health technology assessment: a route to robust evidence and fair deliberation. Int J Technol Assess Health Care. 2010;26(3):334-40. PubMed

Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. Washington, DC: US Department of Health and Human Services, June 2009. Accessed August 5, 2017 at:

Ferguson TB Jr, Peterson ED, Coombs LP, Eiken MC, et al. Use of continuous quality improvement to increase use of process measures in patients undergoing coronary artery bypass graft surgery: A randomized controlled trial. JAMA. 2003;290(1):49-56. PubMed

Fineberg HV. Keynote Address. Health Technology Assessment International  2009 Annual Meeting, Singapore; June 26, 2009.

Fletcher SW. Whither scientific deliberation in health policy recommendations? N Engl J Med. 1997;336(16):1180-3. PubMed

Foulds J. Effectiveness of smoking cessation initiatives. Smoking cessation services show good return on investment. BMJ. 2002 Mar 9;324(7337):608-9. PubMed

Friedberg M, Saffran B, Stinson TJ, et al. Evaluation of conflict of interest in economic analyses of new drugs used in oncology. JAMA. 1999;282(15):1453-7. PubMed

Fries JF, Krishnan E. Equipoise, design bias, and randomized controlled trials: the elusive ethics of new drug development. Arthritis Res Ther. 2004;6(3):R250-5. PubMed | PMC free article

Fuchs VR. The doctor's dilemma--what is "appropriate" care? N Engl J Med. 2011;365(7):585-7. PubMed

Fye WB. The power of clinical trials and guidelines, and the challenge of conflicts of interest. J Am Coll Cardiol. 2003;41(8):1237-42. PubMed

Gann MJ, Restuccia JD. Total quality management in health care: a view of current and potential research. Med Care Rev. 1994;51(4):467-500. PubMed

GAO (General Accounting Office). Medicare: Beneficiary use of clinical preventive services. Report to the Chairman, Subcommittee on Oversight and Investigations, Committee on Energy and Commerce, House of Representatives. GAO-02-422. Washington, DC; 2002.

Garrison LP Jr, Bresnahan BW, Higashi MK, et al. Innovation in diagnostic imaging services: assessing the potential for value-based reimbursement. Acad Radiol. 2011;18(9):1109-14. PubMed

Glaeske G. The dilemma between efficacy as defined by regulatory bodies and effectiveness in clinical practice. Dtsch Arztebl Int. 2012;109(7):115-6. PubMed | PMC free article

Goodman CS. Healthcare technology assessment: methods, framework, and role in policy making. Am J Manag Care. 1998;4:SP200-14. PubMed | Publisher free article

Goodman C, Snider G, Flynn K. Health Care Technology Assessment in VA. Boston, Mass: Management Decision and Research Center. Washington, DC: Health Services Research and Development Service; 1996.

Green C. Considering the value associated with innovation in health technology appraisal decisions (deliberations): a NICE thing to do? Appl Health Econ Health Policy. 2010;8(1):1-5. PubMed

Hailey D. A preliminary survey on the influence of rapid health technology assessments. Int J Technol Assess Health Care. 2009;25(3):415-8. PubMed

Harker J, Kleijnen J. What is a rapid review? A methodological exploration of rapid reviews in Health Technology Assessments. Int J Evid Based Healthc. 2012;10(4):397-410. PubMed

Health Equality Europe. Understanding Health Technology Assessment (HTA). July 2008. Accessed Jan. 2, 2014 at:

Heidenreich PA, Trogdon JG, Khavjou OA, et al. Forecasting the future of cardiovascular disease in the United States: a policy statement from the American Heart Association. Circulation. 2011;123(8):933–44. PubMed | Publisher free article

Henshall C, Koch P, von Below GC, Boer A, et al. Health technology assessment in policy and practice. Int J Technol Assess Health Care. 2002;18(2):447-55. PubMed

Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Accessed Sept. 1, 2013 at:

Hinman AR, Orenstein WA, Santoli JM, Rodewald LE, Cochi SL. Vaccine shortages: history, impact, and prospects for the future. Annu Rev Public Health. 2006;27:235-59. PubMed

Hoffman B. Is there a technological imperative in health care? Int J Technol Assess Health Care. 2002;18(3):675-89.

HTAi Patient and Citizen Involvement Interest Sub-Group. Good Practice Examples of PPI. 2015. Accessed Jul. 3, 2018 at:

HTAi Patient and Citizen Involvement Interest Sub-Group. Values and Quality Standards for Patient Involvement in HTA. June 2014. Accessed Jul. 3, 2018 at:

HTAi Patient and Citizen Involvement Interest Sub-Group. Good Practice Examples of Patient and Public Involvement in Health Technology Assessment. Sept. 2013. Accessed Jan. 2, 2014 at:

Hu Q, Schwarz LB, Uhan NA. The impact of group purchasing organizations on healthcare-product supply chains. MSOM. 2012;14(1):7-23. PubMed

Hudon C, Fortin M, Haggerty JL, et al. Measuring patients' perceptions of patient-centered care: a systematic review of tools for family medicine. Ann Fam Med. 2011;9(2):155-64. PubMed | PMC free article

Hutton J, Trueman P, Henshall C. Coverage with evidence development: an examination of conceptual and policy issues. Int J Technol Assess Health Care. 2007;23(4):425-32. PubMed

International Committee of Medical Journal Editors.  Uniform Requirements for Manuscripts Submitted to Biomedical Journals: Publishing and Editorial Issues Related to Publication in Biomedical Journals: Obligation to Register Clinical Trials. 2013. Accessed June 20, 2014 at:

INAHTA. International Network of Agencies in Health Technology Assessment Secretariat. A Checklist for Health Technology Assessment Reports. Version 3.2. August 2007. Accessed February 29, 2016 at:

INAHTA. International Network of Agencies for Health Technology Assessment. Health Technology Assessment (HTA) Glossary. First Edition. INAHTA Secretariat, c/o SBU, Stockholm, July 5, 2006. Accessed June 20, 2013 at:

Institute of Medicine. Committee on Comparative Effectiveness Prioritization. Initial National Priorities for  Comparative Effectiveness Research. Washington, DC: National Academies Press; 2009.

Institute of Medicine, Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001.

International Committee of Medical Journal Writers. Conflict of interest. Lancet. 1993;341(8847):742-3. PubMed

Jacobs BL, Zhang Y, Schroeck FR, et al. Use of advanced treatment technologies among men at low risk of dying from prostate cancer. JAMA. 2013;309(24):2587-95. PubMed | PMC free article

Jagsi R, Sheets N, Jankovic A, Motomura AR, Amarnath S, Ubel PA. Frequency, nature, effects, and correlates of conflicts of interest in published clinical cancer research. Cancer. 2009;115(12):2783-91. PubMed | Publisher free article

Jang S, Chae YK, Haddad T, Majhail NS. Conflict of interest in economic analyses of aromatase inhibitors in breast cancer: a systematic review. Breast Cancer Res Treat. 2010;121(2):273-9. PubMed

Kaden RJ, Vaul JH, Palazola PA. Negotiating payment for new technology purchases. Healthc Financ Manage. 2002;56(12):44-8. PubMed

Kassirer JP, Angell M. Financial conflicts of interest in biomedical research. N Engl J Med. 1993;329(8):570-1. PubMed | Publisher free article

Kassirer JP, Angell M. The journal's policy on cost-effectiveness analyses. N Engl J Med. 1994;331(10):669-70. PubMed | Publisher free article

Kennedy I. Appraising the Value of Innovation and Other Benefits. A Short Study for NICE. July 2009. Accessed February 29, 2016 at:

Khangura S, Konnyu K, Cushman R, et al. Evidence summaries: the evolution of a rapid review approach. Syst Rev. 2012;1:10. PubMed | PMC free article

Kim S, Losina E, Solomon DH, Wright J, Katz JN. Effectiveness of clinical pathways for total knee and total hip arthroplasty: literature review. J Arthroplasty. 2003;18(1):69-74. PubMed | PMC free article

Klemp M, Frønsdal KB, Facey K; HTAi Policy Forum. What principles should govern the use of managed entry agreements? Int J Technol Assess Health Care 2011;27(1):77-83. PubMed

Knickrehm S. Non-Financial Conflicts of Interest. Slide Presentation from the AHRQ 2009 Annual Conference. December 2009. Agency for Healthcare Research and Quality, Rockville, MD. Accessed February 29, 2016 at: //

Kravitz RL, Duan N, Braslow J. Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages. Milbank Q 2004;82(4):661-87. PubMed | PMC free article

Kreis J, Schmidt H. Public engagement in health technology assessment and coverage decisions: a study of experiences in France, Germany, and the United Kingdom. J Health Polit Policy Law. 2013;38(1):89-122. PubMed

Kwan J, Sandercock P. In-hospital care pathways for stroke: a Cochrane systematic review. Stroke. 2003;34(2):587-8. PubMed | Publisher free article

Lathyris DN, Patsopoulos NA, Salanti G, Ioannidis JP. Industry sponsorship and selection of comparators in randomized clinical trials. Eur J Clin Invest. 2010;40(2):172-82. PubMed

Lee A, Skött LS, Hansen HP. Organizational and patient-related assessments in HTAs: state of the art. Int J Technol Assess Health Care. 2009;25(4):530-6. PubMed

The Lewin Group. The Clinical Review Process Conducted by Group Purchasing Organizations and Health Systems. Prepared for the Health Industry Group Purchasing Association, April 2002.

The Lewin Group. Outlook for Medical Technology Innovation. Report 2: The Medicare Payment Process and Patient Access to Technology. Washington, DC: AdvaMed; 2000.

Lo B, Wolf LE, Berkeley A. Conflict-of-interest policies for investigators in clinical trials. N Engl J Med. 2000;343(22):1616-20. PubMed

Lohr KN, ed. Institute of Medicine. Medicare: a Strategy for Quality Assurance. Volume I. Washington, DC. National Academy Press; 1990.

Lohr KN, Rettig RA, eds. Quality of Care and Technology Assessment. Report of a Forum of the Council on Health Care Technology. Washington, DC: National Academy Press; 1988.

Mangione-Smith R, DeCristofaro AH, Setodji CM, Keesey J, et al. The quality of ambulatory care delivered to children in the United States. N Engl J Med. 2007;357(15):1515-23. PubMed | PMC free article

McDonald IG. Quality assurance and technology assessment: pieces of a larger puzzle. J Qual Clin Pract. 2000;20
(2-3):87-94. PubMed

McGivney WT. Proposal for assuring technology competency and leadership in medicine. J Natl Cancer Inst. 1992;84(10):742-5. PubMed

McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, Kerr EA. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348(26):2635-45. PubMed

McNeil BJ. Shattuck Lecture −Hidden barriers to improvement in the quality of care. N Engl J Med. 2001;345(22):1612-20. PubMed

Mead N, Bower P. Patient-centredness: a conceptual framework and review of the empirical literature. Soc Sci Med 2000;51(7):1087-110. PubMed

Medical Technology Leadership Forum. MTLF Summit: Conditional Coverage of Investigational Technologies. Prepared by The Lewin Group. Washington, DC; October 1999.

Mello MM, Brennan TA. The controversy over high-dose chemotherapy with autologous bone marrow transplant for breast cancer. Health Aff (Millwood). 2001;20(5):101-17. PubMed | Publisher free article

Methodology Committee of the Patient-Centered Outcomes Research Institute. Methodological standards and patient-centeredness in comparative effectiveness research. The PCORI perspective. JAMA. 2012;307(15):1636-40. PubMed

Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Publication No. 10(14)-EHC063-EF. Rockville, MD: Agency for Healthcare Research and Quality. January 2014. Accessed Feb. 1, 2014 at: //

Miller D, Rudick RA, Hutchinson M. Patient-centered outcomes: translating clinical efficacy into benefits on health-related quality of life. Neurology. 2010 Apr 27;74 Suppl 3:S24-35. PubMed

Miller FG, Pearson SD. Coverage with evidence development: ethical issues and policy implications. Med Care 2008;46(7):746-51. PubMed

Nakamura C, Bromberg M, Bhargava S, Wicks P, Zeng-Treitler Q. Mining online social network data for biomedical research: a comparison of clinicians' and patients' perceptions about amyotrophic lateral sclerosis treatments. J Med Internet Res. 2012;14(3):e90. PubMed | PMC free article

National Academies. Policy on Committee Composition and Balance and Conflicts of Interest for Committees Used in the Development of Reports. May 12, 2003. Accessed December 13, 2013 at:

National Institute for Health and Care Excellence (NICE). Citizens Council. Accessed Jan. 2, 2014 at:

National Institute for Health and Care Excellence (NICE). Guide to the Methods of Technology Appraisal. Accessed Jul. 12, 2016 at:

Neumann PJ. What we talk about when we talk about health care costs. N Engl J Med. 2012;366(7):585-6. PubMed

Okike K, Kocher MS, Mehlman CT, Bhandari M. Industry-sponsored research.  Injury. 2008;39(6):666-80. PubMed

Oxman AD, Guyatt G. A consumer’s guide to subgroup analyses. Ann Intern Med 1992;116(1):76-84. PubMed

Patient-Centered Outcomes Research Institute. Patient-centered outcomes research. 2013. Accessed December 13, 2013 at:

Pearson SD, Bach PB. How Medicare could use comparative effectiveness research in deciding on new coverage and reimbursement. Health Aff (Millwood). 2010;29(10):1796-804. PubMed | Publisher free article

Perlis RH, Perlis CS, Wu Y, et al. Industry  sponsorship and financial conflict of interest in the reporting of clinical trials in psychiatry. Am J Psychiatry. 2005;162(10):1957-60. PubMed

Phillips WR. Clinical policies: making conflicts of interest explicit. Task force on clinical policies for patient care. American Academy of Family Physicians. JAMA. 1994;272(19):1479. PubMed

Pilnick A, Dingwall R, Starkey K. Disease management: definitions, difficulties and future directions. Bull World Health Organ. 2001;79(8):755-63. PubMed | PMC free article

Polyzos NP, Valachis A, Mauri D, Ioannidis JP. Industry involvement and baseline assumptions of cost-effectiveness analyses: diagnostic accuracy of the Papanicolaou test. CMAJ. 2011;183(6):E337-43. PubMed | PMC free article

Porter ME. What is value in health care? N Engl J Med. 2010;363(26):2477-81. PubMed

Reda AA, Kotz D, Evers SM, van Schayck CP. Healthcare financing systems for increasing the use of tobacco dependence treatment. Cochrane Database Syst Rev. 2012 Jun 13;(6):CD004305. PubMed

Rettig RA. Health Care in Transition: Technology Assessment in the Private Sector. Santa Monica, Ca: RAND; 1997. Publisher free publication

Rothrock NE, Hays RD, Spritzer K, Yount SE, et al. Relative to the general US population, chronic diseases are associated with poorer health-related quality of life as measured by the Patient-Reported Outcomes Measurement Information System (PROMIS). J Clin Epidemiol. 2010;63(11):1195-204. [PubMed] [PMC free article]

Schauffler HH, Barker DC, Orleans CT. Medicaid coverage for tobacco-dependence treatments. Health Aff (Millwood). 2001;20(1):298-303. PubMed | Publisher free article

Schulman K. Cost-effectiveness analyses. N Engl J Med. 1995;332(2):124. PubMed

Sharf BF. Out of the closet and into the legislature: breast cancer stories. Health Aff (Millwood). 2001;20(1):213-8. PubMed | Publisher free article

Sheingold, SH. Technology assessment, coverage decisions, and conflict: the role of guidelines. Am J Manag Care. 1998;4:SP117-25. PubMed | Publisher free article

Smith R. Conflicts of interest: how money clouds objectivity. J R Soc Med. 2006;99(6):292–7. PubMed | PubMed Central free article

Stead LF, Perera R, Bullen C, Mant D, et al. Nicotine replacement therapy for smoking cessation. Cochrane Database Syst Rev. 2012 Nov 14;11:CD000146. PubMed

Steinberg EP. Cost-effectiveness analyses. N Engl J Med. 1995;332(2):123. PubMed | Publisher free article

Steinberg EP, Tunis S, Shapiro D. Insurance coverage for experimental technologies. Health Aff (Millwood). 1995;14(4):143-58. PubMed |Publisher free article

Stewart M, et al. Patient-Centered Medicine: Transforming the Clinical Method. 3rd ed. United Kingdom: Radcliffe Health; 2013.

Street RL Jr, Elwyn G, Epstein RM. Patient preferences and healthcare outcomes: an ecological perspective. Expert Rev Pharmacoecon Outcomes Res. 2012;12(2):167-80. PubMed

Thompson DF). Understanding financial conflicts of interest. N Engl J Med. 1993;329(8):573-6. PubMed

Trueman P, Grainger DL, Downs KE. Coverage with evidence development: applications and issues. Int J Technol Assess Health Care 2010;26(1):79-85. PubMed

UK National Institute for Health and Care Excellence (NICE). Citizens Council. Accessed Jan. 2, 2014 at:

US Food and Drug Administration. Guidance for Industry. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. December 2009. Accessed January 29, 2015 at: //

Veatch RM. The irrelevance of equipoise. J Med Philos. 2007;32(2):167-83. PubMed

von Below GC, Boer A, Conde-Olasagasti JL, Dillon A, et al. Health technology assessment in policy and practice. Working Group 6 Report. Int J Technol Assess Health Care. 2002;18(2):447-55. PubMed

Wakefield DS, Wakefield BJ. Overcoming barriers to implementation of TQM/CQI in hospitals: myths and realities. QRB. Quality Review Bulletin. 1993;19(3):83‑8. PubMed

Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in medicine--reporting of subgroup analyses in clinical trials. N Engl J Med. 2007;357(21):2189-94. PubMed | Publisher free article

Watt A, Cameron A, Sturm L, Lathlean T, et al. Rapid reviews versus full systematic reviews: an inventory of current methods and practice in health technology assessment. Int J Technol Assess Health Care. 2008;24(2):133-9. PubMed

Whitty JA. An international survey of the public engagement practices of health technology assessment organizations. Value Health. 2013;16(1):155-63. PubMed

Wilson JM. It's time for gene therapy to get disruptive! Hum Gene Ther. 2012;23(1):1-3. PubMed

Wood DE, DeCamp MM. The National Emphysema Treatment Trial: a paradigm for future surgical trials. Ann Thorac Surg. 2001;72(2):327-9. PubMed

Woolacott NF, Jones L, Forbes CA, et al. The clinical effectiveness and cost-effectiveness of bupropion and nicotine replacement therapy for smoking cessation: a systematic review and economic evaluation. Health Technol Assess. 2002;6(16):1-245. PubMed | Publisher free article

Wong MK, Mohamed AF, Hauber AB, Yang JC, et al. Patients rank toxicity against progression free survival in second-line treatment of advanced renal cell carcinoma. J Med Econ. 2012;15(6):1139-48. PubMed

Zandbelt LC, Smets EM, Oort FJ, et al. Medical specialists' patient-centered communication and patient-reported outcomes. Med Care. 2007;45(4):330-9. PubMed

Page of

Previous Section Next Section

Last Reviewed: July 8, 2020