Skip Navigation Bar

National Information Center on Health Services Research and Health Care Technology (NICHSR)



One of the great challenges in HTA is to assemble the evidence--the data, literature and other information--that is relevant to a particular assessment. For very new technologies, this information may be sparse and difficult to find; for many technologies, it can be profuse, scattered and of widely varying quality. Literature searching and related evidence retrieval are integral to successful HTA, and the time and resources required for these activities should be carefully considered in planning any HTA (Auston 1994; Goodman 1993).

Types of Sources

Available information sources cover different, though often overlapping, sectors of health care information. Although some are devoted to health care topics, others cover the sciences more broadly. Multiple sources should be searched to increase the likelihood of retrieving relevant reports. The variety of types of sources that may be useful for HTA include:

  • Computer databases of published literature
  • Computer databases of clinical and administrative data
  • Printed indexes and directories
  • Government reports and monographs
  • Policy and research institute reports
  • Professional association reports and guidelines
  • Market research reports
  • Company reports and press releases
  • Reference lists in available studies and reviews
  • Special inventories/registers of reports
  • Health newsletters and newspapers
  • Colleagues and investigators

Of course, the Internet is an extraordinarily broad and readily accessible medium that provides access to many of these information sources.  

There are hundreds of publicly available computer databases for health care and biomedical literature. Among these are various general types. For example, bibliographic databases have indexed citations for journal articles and other publications. Factual databases provide information in the form of guidelines for diagnosis and treatment, patient indications, and contraindications, and other authoritative information. Referral databases provide information about organizations, services and other information sources.

The National Information Center on Health Services Research & Health Care Technology (NICHSR) [] of the US National Library of Medicine (NLM) provides an extensive, organized set of the many, evolving databases, publications, outreach and training, and other information resources for HTA. One online source, Etext on Health Technology Assessment (HTA) Information Resources [], is a comprehensive textbook on sources of HTA information and searching approaches compiled by information specialists and researchers from around the world (National Library of Medicine 2003). Various other useful compendia of HTA information resources have been prepared (Busse 2002; Glanville 2003; Chan 2003). Some of the main bibliographic and factual databases useful in HTA are listed in Box 39.

The most widely used of these resources for HTA are the large bibliographic databases, particularly MEDLINE, produced by NLM, and EMBASE, produced by Elsevier. MEDLINE can be accessed at the NLM website using PubMed, which also includes new in-process citations (with basic citation information and abstracts before being indexed with MeSH terms and added to MEDLINE), citations from various life science journals, and certain other entries. In addition, there are many specialized or more focused databases in such areas as AIDS, bioethics, cancer treatment, pharmaceutical research and development, ongoing clinical trials (e.g., of NLM), and practice guidelines (e.g., National Guideline Clearinghouse of AHRQ).

The Cochrane Collaboration [] is an international organization that prepares, maintains and disseminates systematic reviews of RCTs (and other evidence when appropriate) of treatments for many clinical conditions. More than 1,500 systematic reviews have been produced by nearly 50 Cochrane review groups in such areas as acute respiratory infections, breast cancer, diabetes, hypertension, infectious diseases, and pregnancy and childbirth. The Cochrane Collaboration produces the Cochrane Library, which includes databases and registers produced by the Cochrane Collaboration as well as some produced by other organizations. The Database of Abstracts of Reviews and Dissemination (DARE) [] and the NHS Economic Evaluation Database are produced by the NHS Centre for Reviews and Dissemination (NHSCRD).  The HTA Database is produced by the International Network of Agencies for Health Technology Assessment (INAHTA) [], in collaboration with the NHSCRD.

The selection of sources for literature searches should depend on the purpose of the HTA inquiry and pertinent time and resource constraints. Most searches are likely to involve MEDLINE or another large database of biomedical literature (Suarez-Almazor 2000; Topfer 1999). However, the selection of other databases may differ by purpose, e.g., horizon scanning, ascertaining regulatory or payment status of technologies, comprehensive systematic review, or identifying literature in particular clinical areas.

Gray Literature

Much valuable information is available beyond the traditional published sources. This "gray" or "fugitive" literature is found in industry and government monographs, regulatory documents, professional association reports and guidelines, market research reports, policy and research institute studies, spot publications of special panels and commissions, conference proceedings, and other sources. Many of these can be found via the Internet. Although the gray literature can be timely and cover aspects of technologies that are not addressed in mainstream sources, it is usually not subject to peer review, and must be scrutinized accordingly.  

Box 39
Selected Bibliographic and Factual Databases for HTA

Some Core Sources

  • MEDLINE: citations for biomedical journal articles
  • EMBASE: citations for biomedical journal articles (Elsevier)
  • Cochrane Database of Systematic Reviews: systematic reviews of controlled trials on hundreds of clinical topics
  • Cochrane Controlled Trials Register: bibliography of controlled trials including sources outside peerreviewed journal literature
  • Database of Abstracts of Reviews of Effectiveness (DARE): structured abstracts of systematic reviews from around the world, critically appraised by NHS Centre for Reviews and Dissemination
  • NHS Economic Evaluation Database: abstracts and other information about published economic evaluations of health care interventions
  • Health Technology Assessment Database: records of ongoing projects of members of INAHTA and completed HTAs by INAHTA members and other organizations
  • National Guideline Clearinghouse: evidence-based clinical practice guidelines (AHRQ)

Additional Sources

  • Other NLM/NIH sources:
    • current information about current clinical research studies in health services research and behavioral and social sciences
    • DIRLINE: directory of organizations
    • HSRProj: ongoing health services research projects
    • HSRR (Health Services/Sciences Research Resources): research datasets and instruments/indices.
    • HSTAT: full text of US clinical practice guidelines, consensus development reports, technology assessment reports, etc.
    • PDQ: cancer treatment, supportive care, screening, prevention, clinical trials
    • Other specialized databases such as AIDSLINE, Bioethics, and HealthSTAR have been incorporated into MEDLINE, accessed, e.g., via PubMed
  • ACP Journal Club: selected studies and systematic reviews for immediate attention of clinicians, with "value added" abstracts and commentary
  • AltHealthWatch: information resources on alternative medicine
  • Bandolier: journal of evidence summaries
  • Best Evidence (ACP Journal Club plus Evidence Based Medicine)
  • BIOSIS Previews: citations of life sciences literature (BIOSIS)
  • CEA Registry: database of standardized cost-utility analyses (Harvard School of Public Health)
  • CINAHL: citations for nursing and allied health literature (Cinahl Information Systems)
  • CDC Wonder: gateway to reports and data of the US Centers for Disease Control and Prevention (CDC)
  • Cochrane Methodology Register: bibliography of articles and books on the science of research synthesis
  • Cochrane Database of Methodology Reviews: full text of systematic reviews of empirical methodological studies
  • HDA Evidence Base: summaries of systematic reviews of effectiveness, literature reviews, meta-analyses, expert group reports, and other review-level information (NHS Health Development Agency, UK)
  • MANTIS: bibliographic database on manual, alternative, and natural therapies
  •  Netting the Evidence: (ScHARR, University of Sheffield, UK)
  •  PsycINFO: citations of psychological literature (American Psychological Association)
  •  SciSearch: citations for scientific journal articles (Institute for Scientific Information)

Publication Bias

Various forms of bias can affect the validity of HTA. One reason for careful planning and conduct of search strategies for HTA is minimize, or at least recognize, the effects of publication bias. Studies of the composition of the biomedical research literature have found imbalances in the publication of legitimate studies (Chalmers 1990). For instance, positive studies (that find statistically significant treatment effects) are more likely than negative studies (that find no treatment effects) to be published in peer-reviewed journals (Dickersin 1993; Dickersin 1997). A study sponsored by a health product company or other group with an interest in the results may be less likely to be submitted for publication if the findings are not favorable to the interests of that group. RCTs conducted for market approval (e.g., by the US FDA) often are not published (MacLean 2003). Some research indicates that, among published studies of health technologies, smaller studies tend to report positive results more frequently (Agema 2002). Positive studies are more likely to be published in English-language journals, be reported in multiple publications, and be cited in other articles (Easterbrook 1991, Götzsche 1989). These multiple appearances and citations increase the likelihood of being identified in literature searches and included in meta-analyses and other systematic reviews, which may introduce bias into the results of these syntheses as well (Sterne 2001). The prevalence of unpublished studies may vary by specialty; for example, oncology appears to have a high prevalence of unpublished studies.  

One detailed analysis of the characteristics of clinical trials used in systematic reviews indicated that, compared to other clinical areas, trials in the fields of psychiatry, rheumatology, and orthopedics tend more often to be published in non-English languages and appear in sources not indexed in MEDLINE (Egger 2003). Time lag bias occurs when the time from completion of a clinical trial to its publication is affected by the direction (positive vs. negative findings) and strength (statistical significance) of the trial results (Ioannidis 1998).

Certainly, bias in selection of studies used in HTA may arise to the extent that the literature search does not include studies that appear in languages other than English (language bias), are not indexed in MEDLINE or other major bibliographic databases, are unpublished, or are of lesser methodological quality. While the validity of an HTA is likely linked to the effort to include an unbiased sample of relevant studies, the size and direction of this relationship varies. There is a growing literature on the extent to which more or less restrictive inclusion criteria for meta-analyses affect their results. For example, some research indicates that systematic reviews limited to the English language literature that is accessible via the major bibliographic databases produces similar or same results to those based on less restricted reviews (Egger 2003). Lowering the standard of methodological quality for inclusion of published studies in an HTA may bias the findings if these studies tend to report positive findings more often that higher-quality studies.  

In planning a literature search, assessors should weigh the anticipated quality of a search with time and resource constraints. Efforts to recognize and minimize bias may be further subject to such factors as the availability of studies by language and for particular clinical areas, and their accessibility via bibliographic databases.


Help for Searchers

Given the great number of databases and the variety in their scope, means of access, controlled vocabularies and search commands, it is advisable to consult health information specialists. These experts can be especially helpful when planning which databases to search, inclusion and exclusion criteria, and other aspects of literature searches. An expanding network of HTA information specialists who work with HTA agencies and other evidence-based medicine organizations around the world have formed the HTAi Information Resources Group, which is extending the capabilities, expertise, and collaboration in the field. Improved indexing, text word searching, user-friendly interfaces, more powerful personal computers and other advances in medical informatics are helping non-expert searchers to retrieve valuable information more effectively and efficiently. Indeed, the enhanced ability of all types of assessors to probe these databases provides a more immediate, hands-on understanding of the scope and quality of literature on any given topic.

During the last decade, the NLM has undertaken to improve its MeSH (Medical Subject Headings) controlled vocabulary (used to index and search literature in MEDLINE and other NLM databases) in the related fields of HTA and health services research. In cooperation with the Cochrane Collaboration and others, NLM has improved the indexing of citations in MEDLINE and other databases to improve identification of RCTs (Dickersin 1994). Most bibliographic and factual databases have user-friendly tutorials, search engines, and other searching tools that are increasingly standard and familiar to expert and non-expert searchers alike. There is a growing number of resources for supporting searching strategies for HTA (Goodman 1993, Sackett 1997). A new resource from the NLM NICHSR, Etext on Health Technology Assessment (HTA) Information Resources [], provides extensive guidance and resources for searching in HTA (National Library of Medicine 2003). Particularly instructive and useful for clinicians is the series of articles published in the Journal of the American Medical Association: Users' Guides to the Medical Literature, from the Evidence-Based Medicine Working Group (Hunt 2000).

The search for pertinent existing evidence is normally one of the first major tasks of an assessment, and should be planned accordingly. Costs associated with evidence searches can be significant, coming in the form of staff time and acquisition of literature, data tapes, and other documentation. Although access to MEDLINE (e.g., via PubMed) and other public-source databases is generally free of inexpensive, using some specialized scientific and business databases can be more costly. Database vendors offer a variety of packages of databases and pricing algorithms for these. HTA programs of such organizations as ECRI, the Blue Cross and Blue Shield Association, and Hayes sell their reports on a subscription basis. Some market research monographs and other reports oriented for health product companies, investors and other business interests are priced in the thousands of dollars.

Previous Section Next Section Table of Contents NICHSR Home Page