National Information Center on Health Services Research and Health Care Technology (NICHSR)
HTA 101: VII. RETRIEVE EVIDENCE
- A. Types of Sources
- B. Grey Literature
- C. Publication Bias
- D. Help for Searchers
- References for Chapter VII
One of the great challenges in HTA is to assemble the evidence—the data, literature and other information—that is relevant to a particular assessment, and to do so efficiently and in a timely manner. For a new or emerging topic, this information may be sparse and difficult to find. For many topics, the evidence is readily available, yet profuse and of widely varying quality. Literature searching and related evidence retrieval are integral to successful HTA, and the time and resources required for these activities should be carefully considered in planning any HTA.
A. Types of Sources
Available information sources cover different, though often overlapping, sectors of health care information. Although some are devoted to health care topics, others cover the biomedical, social, and other sciences more broadly. Multiple sources should be searched to increase the likelihood of retrieving relevant reports. The types of sources that may be useful for HTA include:
- Journal and other published literature databases
- Clinical and administrative databases
- Practice guideline databases
- Government reports and monographs
- Policy and research institute reports
- Professional association reports and guidelines
- Market research reports
- Company reports and press releases
- Reference lists in available studies and reviews
- Special inventories/registers of reports
- Health newsletters and newspapers
- Colleagues and investigators
Some of these are traditional peer-reviewed, published sources; others are recognized as “grey” literature, as described below.
Across the hundreds of publicly available electronic databases of potential relevance to HTA are various general types. Among the major categories, bibliographic databases have indexed citations (or “records”) for journal articles and other publications. Factual databases provide substantive content in the form of, e.g., guidelines for diagnosis and treatment; compendia of information about pharmacologic properties, indications, contraindications, and other information about drugs; and other authoritative information. Referral databases provide information about organizations, services and other information sources.
The National Information Center on Health Services Research & Health Care Technology (NICHSR) of the US National Library of Medicine (NLM) provides an extensive, organized set of the many, evolving databases, publications, outreach and training, and other information resources for HTA. Various other useful listings of HTA information resources and related tools have been prepared, e.g., Health Technology Assessment on the Net International (Institute of Health Economics 2013), Grey Matters: A Practical Search Tool for Evidence-Based Medicine ] (CADTH 2014), and the HTAi Vortal.
The most widely used of these resources for HTA are the large bibliographic databases, particularly MEDLINE, produced by NLM, and Embase, produced by Elsevier. MEDLINE can be accessed at the NLM website using PubMed, which also includes new in-process citations (with basic citation information and abstracts before being indexed with MeSH terms and added to MEDLINE), citations from various life science journals, and certain other entries. In addition, there are certain specialized or more focused databases in such areas as ongoing clinical trials and their results (e.g., ClinicalTrials.gov of NLM) (Zarin 2011), ongoing health services research (e.g., HSRProj), and practice guidelines (e.g., the National Guideline Clearinghouse of AHRQ and the International Guideline Library of the Guidelines International Network).
The Cochrane Collaboration is an international organization that prepares, maintains and disseminates systematic reviews of RCTs (and other evidence when appropriate) of interventions for many diseases and conditions. Nearly 7,000 systematic reviews have been produced by more than 50 Cochrane review groups in such areas as acute respiratory infections; breast cancer; dementia and cognitive improvement; hypertension; infectious diseases; pregnancy and childbirth; and pain, palliative, and supportive care. These reviews are made available via the Cochrane Library, which also includes certain databases and registers produced by the Cochrane Collaboration as well as some produced by other organizations. The Cochrane Central Register of Controlled Trials includes more than 660,000 records. The Database of Abstracts of Reviews and Dissemination (DARE) and the NHS Economic Evaluation Database (NHS EED) are produced by the Centre for Reviews and Dissemination (CRD) at the University of York. The HTA Database is produced by CRD with information supplied by members of the International Network of Agencies for Health Technology Assessment (INAHTA) and other contributing HTA programs.
The selection of sources for literature searches should depend on the evidence questions and related matters of content of the HTA as well as pertinent time and resource constraints. Most searches are likely to involve one or more of the large bibliographic databases, which have their relative merits and can be complementary in some instances (Wilkins 2005). These sources can be supplemented by specialized bibliographic databases relevant to the topic, such as those devoted to mental health, allied health, or health economics. Further, there are strategies and search filters for identifying particular types of studies in such bibliographic databases, such for records on economic evaluations (Glanville, Lefebvre, Miles, Camosso-Stefinovic 2006). The selection of other databases is likely to differ by purpose of the inquiry, e.g., horizon scanning, ascertaining regulatory or payment status of technologies, comprehensive systematic review, or identifying literature in particular clinical areas. The yield from searching the more specialized sources in addition to the large, generic ones varies (Royle 2003).
Some of the most commonly used bibliographic and factual databases in HTA are listed in Box VII-1. Many additional bibliographic and factual databases that can be used in HTA, including additional generic ones and others that are more specialized, are listed in Box VII-2
Box VII-1. Some Core Sources: Bibliographic and Factual Databases for HTA
Among the most commonly used are the following:
- PubMed (including MEDLINE): citations for life sciences journal articles and online books (NLM)
- Embase: citations for biomedical journal articles (Elsevier)
- Cochrane Database of Systematic Reviews: systematic reviews of controlled trials on hundreds of clinical topics
- Cochrane Central Register of Controlled Trials: bibliography of controlled trials including sources outside peer-reviewed journal literature
- Database of Abstracts of Reviews of Effectiveness (DARE): structured abstracts of systematic reviews from around the world, appraised by NHS Centre for Reviews and Dissemination. Archive available until 2021.
- Health Technology Assessment Database: records of completed and ongoing projects of members of INAHTA and other organizations. Records are not eing added as of 2018.
- NHS Economic Evaluation Database (NHS EED): abstracts and other information about published economic evaluations of health care interventions. Archived available until 2021.
Box VII-2. Additional Sources: Bibliographic and Factual Databases for HTA
The variety of additional generic and more specific sources includes the following:
- Other NLM sources:
- Bookshelf: free full-text access to online biomedical books
- ClinicalTrials.gov: registry of ongoing, and summary results of, clinical trials and observational studies funded by U.S. government and private sources conducted in the U.S. and around the world
- HSRProj: ongoing health services research and public health projects
- HSRR (Health Services and Sciences Research Resources): information about research datasets, instruments/indices, software
- ACP Journal Club: selected studies and systematic reviews on clinical and economic topics for attention of clinicians, with “value added" abstracts and commentary
- AMED (Allied and Complementary Medicine): citations to journals in physiotherapy, occupational therapy, rehabilitation, speech therapy, podiatry, complementary medicine, palliative care (British Library)
- Bandolier: journal of summaries of evidence-based health care (University of Oxford, UK)
- BIOSIS Previews: citations from life sciences journals and reports, reviews, meetings (Thomson Reuters)
- Campbell Collaboration Library of Systematic Reviews: systematic reviews of the effects of social interventions, including education, crime and justice, and social welfare (Campbell Collaboration)
- CEA Registry: database of standardized cost-utility analyses (Tufts Medical Center)
- CINAHL: citations for nursing and allied health literature (Cinahl Information Systems)
- CDC Wonder: gateway to epidemiological and public health reports, other information, and data (CDC, US)
- Cochrane Methodology Register: bibliography of articles and books on research synthesis (Cochrane Library)
- ERIC (Educational Resources Information Center): citations of education information, including in health, for journals, books, research syntheses, technical reports, conferences, policy papers, etc. (US Dept. Education)
- Evidence Base: evidence briefing papers, reviews on state of existing evidence, research gaps; research recommendations (NICE, UK)
- Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre): databases of health promotion: Trials Register of Promoting Health Interventions (TRoPHI), Bibliomap, Database of Promoting Health Effectiveness Reviews (DoPHER) (Social Science Research Unit, UK)
- EVIPNet Virtual Health Library: for evidence-informed policymaking, particularly in low- and middle-income countries (WHO)
- Google Scholar: scholarly literature, with citation information
- Health Economic Evaluations Database (HEED): records and bibliographies of economic evaluations of health care interventions, reviewed by health economists (Wiley InterScience)
- Health Systems Evidence: syntheses of research evidence about governance, financial and delivery arrangements within health systems (McMaster University, Canada)
- International Guideline Library: guidelines, evidence reports developed or endorsed by G-I-N member organizations (Guidelines International Network)
- National Guideline Clearinghouse (NGC): evidence-based clinical practice guidelines (AHRQ, US)
- NHS Evidence: portal to organizations and other sources providing health and social care evidence and best practices; most but not limited to UK (NICE, UK)
- OTSeeker: abstracts of systematic reviews and RCTs (appraised and rated) on occupational therapy
- PAIS International: citations of policy information, including for health and social conditions, demography, from journals, books, government documents, research reports, grey literature, conferences, etc. (ProQuest)
- PEDro(Physiotherapy Evidence Database): citations for RCTs, systematic reviews, and clinical practice guidelines in physiotherapy (University of Sydney, Australia)
- PsycINFO: citations of psychological literature (American Psychological Association)
- PROSPERO:prospective registry of systematic reviews in health and social care (NHS Centre for Reviews and Dissemination)
- REHABDATA:citations for reports, articles, other documents on disability and rehabilitation (US National Rehabilitation Information Center)
- SciSearch: citations for journals in science, technology, biomedicine, and related disciplines, including all records of Science Citation Index (Dialog)
- Scopus: citations for peer-reviewed journals, conference proceedings, books, patents in science, medicine, social sciences, other areas; also patent records (Elsevier)
- SUMSearch 2: integrated search of original studies, systematic review, and practice guidelines from PubMED, DARE, and NGC (University of Kansas, US)
- Systematic Review Data Repository (SRDR): archive of systematic reviews and their data, and data extraction tool (AHRQ, US)
- TRIP Database: integrated search tool of multiple sites to identify clinical evidence for practice (TRIP Database, Ltd.)
- UpToDate®: evidence-based, physician-authored clinical decision support resource (Wolters Kluwer)
- Web of Science: citations for journals and conference proceedings in life sciences, social sciences, other fields (Thomson Reuters)
B. Grey Literature
Beyond the traditional sources of published literature, much valuable information is available from the “grey” literature. (Some or all of this is also known as “fugitive” or “ephemeral” literature.) The concept of grey literature is evolving; some define it as any literature except peer-reviewed journals and books cited in the main bibliographic databases such as MEDLINE or Embase. A contemporary definition (not limited to HTA) is as follows:
Grey literature stands for manifold document types produced on all levels of government, academics, business and industry in print and electronic formats that are protected by intellectual property rights, of sufficient quality to be collected and preserved by library holdings or institutional repositories, but not controlled by commercial publishers, i.e., where publishing is not the primary activity of the producing body (Schöpfel 2010).
For HTA, grey literature is found in such sources as:
- Health and economic statistical reports
- Regulatory documents
- Coverage policies of health authorities and insurance plans
- Drug and biologics compendia
- Health professional association reports and guidelines
- Government monographs
- Industry (e.g., life sciences company) reports
- Market research reports for particular health care sectors/industries
- Policy and research institute studies
- Reports of special expert panels and commissions
- Conference proceedings
- Technical specifications and standards
- Working papers
Much of the non-proprietary grey literature and some proprietary (for a fee) grey literature are accessible via the Internet.
The main concern with using grey literature in HTA is that it is usually not subject to peer review, and therefore may be subject to certain biases that the peer review process is designed to diminish. The benefits of including grey literature in any particular literature search are highly variable (Dundar 2006). In some instances, the available peer-reviewed sources may suffice, with little to be gained by expanding a search to include grey sources. In other instances, such as when the available peer-reviewed sources are few, outdated, or have gaps or suspected biases (e.g., publication bias), certain grey sources may strengthen the body of evidence to be considered in HTA. As such, plans for conducting literature searches should weigh the pros and cons of including grey literature, and the inclusion and exclusion criteria for searches should be designed and implemented accordingly.
C. Publication Bias
As described in chapter III, various forms of bias can affect the validity of evidence used in HTA. One reason for careful planning and conduct of search strategies for HTA is to recognize and minimize the effects of publication bias. Studies of the composition of the biomedical research literature have found imbalances in the publication patterns of methodologically sound studies (Chalmers 1990), including in HTA (Song 2010). For instance, positive studies (i.e., that find statistically significant effects of a technology) are more likely than negative studies (i.e., that find no effects of a technology) to be published in peer-reviewed journals (Dickersin 1993, 1997). A study sponsored by a company or other organization with a financial or other interest in the results may be less likely to be submitted for publication if the findings are not favorable to the interests of that organization. RCTs and other clinical trials conducted for market approval (e.g., by the US FDA) often are not published for proprietary reasons (MacLean 2003). Some research indicates that, among studies of health technologies that are published, smaller studies tend to report positive results more frequently than larger ones (Agema 2002). Publication bias is not the same as reporting bias, which refers to differential (e.g., between treatment and control groups) or incomplete reporting by investigators of findings in individual studies. (See further discussion in chapter III.)
Multiple analyses have found that positive studies are more likely to be published in English-language journals, reported in multiple publications, reported for late-phase drug trials, and cited in other articles (Easterbrook 1991, Gøtzsche 1989; Hall 2007). These multiple appearances and citations increase the likelihood of being identified in literature searches and included in meta-analyses and other systematic reviews, which may introduce bias into the results of these syntheses as well (Sterne 2001). Certain forms of bias may be linked to the clinical area under study. A detailed analysis of the characteristics of clinical trials used in systematic reviews indicated that, compared to other clinical areas, trials in the fields of psychiatry, rheumatology, and orthopedics tend more often to be published in non-English languages and appear in sources not indexed in MEDLINE (Egger 2003).
Bias in selection of studies used in HTA may arise when a literature searching strategy (or protocol) excludes reports by language (i.e., language bias), when it draws only on MEDLINE or other major bibliographic databases, when it includes only reports in the peer-reviewed literature, or when it excludes studies of lesser (or poorly reported) methodological quality (Jüni 2002). The recent global trend toward publication in English-language journals may be reducing the impact of language bias (Galandi 2006; Higgins 2011).
Prospective, systematic literature searching protocols for HTA should seek to identify relevant evidence beyond traditional published sources where possible. This should include circumstances in which potential conflicts of interest might affect submission or publication of clinical trial reports or other forms of evidence.
Time lag bias occurs when the time from completion of a clinical trial to its publication is affected by the direction (positive vs. negative findings) and strength (statistical significance) of the trial results. This can have implications for the timing of conducting HTA and may be of particular importance when the number of relevant studies is small (Hopewell, Clarke 2007; Ioannidis 1998).
While the validity of an HTA is likely affected by the effort to include an unbiased sample of relevant studies, the size and direction of this relationship varies. There is a growing literature on the extent to which more or less restrictive inclusion criteria for meta-analyses affect their results. For example, some research indicates that systematic reviews limited to the English language literature that is accessible via the major bibliographic databases produces similar or same results to those based on less restricted reviews (Egger 2003). Even so, exclusion of non-English-language studies may result in higher risk of bias in some areas of research, e.g., complementary and alternative medicine (Song 2010). Lowering the standard of methodological quality for inclusion of published studies in an HTA may bias the findings if these included studies tend to report positive findings more often that higher-quality studies.
Recognition of publication bias in peer-reviewed literature may increase the acceptance of grey literature, to the extent that the grey literature can contribute to a more representative capture of the available relevant evidence pertaining to a given topic. An analysis of the impact of grey literature in meta-analyses of RCTs found that published trials tended to be larger and report greater treatment effects than RCTs reported in the grey literature (i.e., abstracts and unpublished data). The analysis also reported that there was limited evidence to show whether grey RCTs were of poorer methodological quality than published trials (Hopewell, McDonald 2007).
Publication bias can be diminished by prospective registration of clinical trials, as in ClinicalTrials.gov; adherence to guidelines for reporting clinical trials and other types of studies in journals; efforts to locate relevant unpublished studies and non-English language studies for literature searches; and certain statistical techniques (e.g., “funnel plots”) to detect potential publication bias (Song 2010; Sterne 2011).
The emergence, validation, and ongoing improvement of guidelines and checklists for reporting research is helping to strengthen the quality of the biomedical and health services research literature, including regarding aspects of publication bias, for HTA reports. Over the last decade in particular, such guidelines have been developed for published reports of diverse study types relevant to HTA. As detailed in chapter IV, some of these include CONSORT (Consolidated Standards of Reporting Trials), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), and STROBE (Strengthening the Reporting of OBservational Studies in Epidemiology). In addition to guiding the reporting of completed research, researchers’ knowledge of such guidelines and related standards is improving the design and conduct of research. An extensive list of these is maintained in Research Reporting Guidelines and Initiatives by NLM. See: //www.nlm.nih.gov/services/research_report_guide.html.
In planning a literature search, assessors should weigh the anticipated quality of a search with time and resource constraints. Efforts to recognize and minimize bias may be further subject to such factors as the availability of studies by language and for particular clinical areas, and their accessibility via bibliographic databases.
D. Help for Searchers
Given the great number of bibliographic, factual, and other databases and the variety in their content, means of access, controlled vocabularies and search commands, it is advisable to involve librarians and health information specialists in planning and conducting literature searches. These experts can be especially helpful when planning which databases to search, designing search strategies to address evidence questions, and reviewing search strategies (Institute of Medicine 2011). Many such experts communicate via international networks. One such network of HTA information specialists who work with HTA agencies and related types of organizations around the world comprise the Health Technology Assessment International (HTAi) Interest Sub-Group on Information Resources, which has helped to extend the capacity, processes, methods, and collaboration in HTA. The Cochrane Library Users’ Group provides a forum for exchanging information about using the Cochrane Library, including its usefulness, conducting searches, and format.
More powerful search engines, improved indexing, text word searching, user-friendly interfaces, reference management software, and other advances in health informatics are helping non-expert searchers to retrieve valuable information more effectively and efficiently. As such, they are able to gain a more immediate, hands-on understanding of the scope and quality of literature of interest.
There is an expanding set of resources for supporting search strategies for HTA (Goodman 1993; Institute of Health Economics 2013; Niederstadt 2010; Ormstad 2010; Straus 2011). A recent analysis identified eight methods for determining the optimal retrieval of studies in HTA searches: the capture-recapture technique, obtaining feedback from the commissioner of the HTA report, seeking the disconfirming case, undertaking comparison against a known gold standard, evaluating retrieval of known items, recognizing the law of diminishing returns, specifying a priori stopping rules, and identifying a point of theoretical saturation (Booth 2010). Particularly instructive for clinicians is the series of articles originally published in the Journal of the American Medical Association: Users’ Guides to the Medical Literature, which has been compiled and updated by the Evidence-Based Medicine Working Group (Guyatt 2008).
The NLM continues to improve its MeSH (Medical Subject Headings) controlled vocabulary (used to index and search literature in PubMed and other NLM sources) in HTA and such related fields as health services research, evidence-based medicine, and comparative effectiveness research. Continued work has improved the indexing and searching of RCTs in PubMed and other bibliographic databases (Dickersin 1994; Glanville 2006; Royle 2007; Zhang 2006). Most bibliographic and factual databases have user-friendly tutorials, search engines, and other searching tools that are increasingly standard and familiar to expert and non-expert searchers.
The search for pertinent existing evidence is normally one of the first major tasks of an HTA, and should be planned accordingly. Costs associated with evidence searches can be significant, coming in the form of staff time and acquisition of literature, data sets, and other documentation. Although access to PubMed and other government-source databases is generally free or inexpensive, using some specialized scientific and business databases can be more costly. Database vendors offer a variety of packages of databases and pricing algorithms for these. HTA programs of such organizations as ECRI Institute, the BlueCross BlueShield Association Technology Evaluation Center (BCBSA TEC), and Hayes Inc. sell their reports on a subscription basis. (BCBSA TEC reports are also accessible on its web site to any user for private research or study only.) Some market research monographs and other reports oriented for health product companies, investors and other business interests are priced in the thousands of dollars.
Agema WR, Jukema JW, Zwinderman AH, van der Wall EE. A meta-analysis of the angiotensin-converting enzyme gene polymorphism and restenosis after percutaneous transluminal coronary revascularization: evidence for publication bias. Am Heart J. 2002;144(5):760-8. PubMed
Canadian Agency for Drugs and Technologies in Health (CADTH), Information Services. Grey Matters: A Practical Search Tool for Evidence-Based Medicine. February 2014. Accessed April 10, 2014 at:http://www.cadth.ca/en/resources/finding-evidence-is/grey-matters.
Dickersin K. How important is publication bias? AIDS Educ Prev. 1997;9(suppl 1):15-21.PubMed
Dundar Y, Dodd S, Dickson R, Walley T, et al. Comparison of conference abstracts and presentations with full-text articles in the health technology assessments of rapidly evolving technologies. Health Technol Assess. 2006;10(5):iii-iv, ix-145. PubMed | Publisher free article
Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;7(1):1-76. PubMed
Galandi D, Schwarzer G, Antes G. The demise of the randomised controlled trial: bibliometric study of the German-language health care literature, 1948 to 2004. BMC Med Res Methodol. 2006;6:30. PubMed | PMC free article
Goodman C. Literature Searching and Evidence Interpretation for Assessing Health Care Practices. Stockholm, Sweden: Swedish Council on Technology Assessment in Health Care; 1993.
Guyatt G, Rennie D, Meade MO, Cook DJ. Users' Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice, 2nd ed. New York: McGraw-Hill Education; 2008.
Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Accessed July 22, 2018 at: https://training.cochrane.org/handbook.
Institute of Health Economics, Osteba, AUnETS. Health Technology Assessment on the Net International: 2013. Edmonton AB: Institute of Health Economics. 2013. Accessed February 1, 2014 at: http://www.ihe.ca/documents/HTA%20on%20the%20Net%202013.pdf.
Institute of Medicine. Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Press; 2011.
Royle P, Waugh N. Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technol Assess. 2003;7(34):iii, ix-x,1-51. PubMed
Schöpfel J. Towards a Prague Definition of Grey Literature. GL 12. Twelfth International Conference on Grey Literature. Prague, Czech Republic. December 2010. Accessed November 27, 2013 at: http://www.textrelease.com/images/GL12_Abstract_S1N1.pdf.
Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomized controlled trials. BMJ. 2011;343:d4002. PubMed | PMC free article
Straus SE, Richardson WS, Glasziou P, Haynes RB. Evidence-Based Medicine: How to Practice and Teach It. 4th ed. New York, NY: Churchill Livingstone Elsevier, 2011.
Last Reviewed: March 11, 2019