National Information Center on Health Services Research and Health Care Technology (NICHSR)
Literature Search Methods for the Development of Clinical Practice Guidelines
Ione Auston, MLS, Marjorie A. Cahn, MA, Catherine R. Selden, MLS, National Library of Medicine, Office of Health Services Research Information
Agency for Health Care Policy and Research, Office of the Forum for Quality and Effectiveness in Health Care, Forum Methodology Conference
December 13-16, 1992
In December 1989, Title IX amendments to Public Health Service Act established the Agency for Health Care Policy and Research (AHCPR) and required AHCPR and the National Library of Medicine (NLM) to enter into an agreement to improve information services for health services researchers. As part of this effort, NLM has:
- Provided literature searching and back-up document delivery services for 13 AHCPR- sponsored guideline development panels; and
- Analyzed references actually incorporated into AHCPR-sponsored guidelines to identify any patterns that suggest ways to streamline the literature search and review process for future guideline panels (Cahn, Selden, Auston, 1992).
Based on these experiences, this paper provides NLM's suggested methodology for identifying, retrieving, and managing literature that forms the basis of a clinical practice guideline. To search efficiently for all known scientific evidence on a particular clinical topic, it is important to understand the literature search process and to plan well. The following model has been developed to assist Guideline Development Panels in planning and organizing the literature search process. Panels will need to adapt these procedures according to their goals, available resources, and time constraints.
THE LITERATURE SEARCH PROCESS
A literature search is a systematic and explicit approach to the identification, retrieval, and bibliographic management of independent studies (usually drawn from published sources) for the purpose of locating information on a topic, synthesizing conclusions, identifying areas for future study, and developing guidelines for clinical practice. Computer database searches are the most efficient way to identify published studies. Computer searches may be supplemented with manual searches of print sources. Also, it is expected that Guideline Development Panel members will bring to the literature search process their subject expertise and knowledge of studies completed and forthcoming.
Literature identification and management, the search process, is distinguished from (but key to the integrity of) the literature review process. A literature review involves analyzing, evaluating, and synthesizing scientific evidence derived from studies identified through the search process and is the topic of other papers commissioned by the Forum. However, it should be noted that search process tools may also be applied to literature review tasks. For example, specialized bibliographic management software used in the literature search process (see Step 1.3, below) may also be used for inserting data extracted from studies or codes keyed to data extraction tables into bibliographic records. Such use will assist panels in reviewing large numbers of studies and documenting each by stated relevancy criteria.
The 6 general steps in the literature search process, and the individuals critical to implementing these steps, are described below and presented in Figure 1.
STEP 1: ADVANCE PLANNING
1.1 Define Goals and Tasks
Before the first panel meeting, a planning and coordination meeting is usually held to discuss the guideline development process. Topics covered typically include development of guideline concepts, organizing the literature search process, and selection of methodology(ies) for literature review. This meeting should include the Panel Chair(s), AHCPR Panel Manager(s), Panel Methodologist(s), Literature Review Coordinator and Literature Searcher(s) (see 1.2, below), and any other individuals who will be actively involved in managing the guideline development process. Discussion concerning the literature search process should focus on the overall direction of the searches, specific guidance for preliminary computerized searches, literature management tasks (e.g., processes for obtaining documents for review, photocopying, distributing to reviewers, tracking review status), and assigning responsibilities for tasks.
1.2 Select Literature Review Management, Searching, and Document Delivery Staff
Literature Review Coordinator
Because the guideline development process is literature dependent, each panel needs a Literature Review Coordinator (LRC) responsible for the management of the literature review process. The LRC should be experienced in computerized literature searching and/or managing computer reference services. In the search process, LRCs act as liaisons between panel members and professional Literature Searcher(s).
Literature searching is a dynamic and iterative process; there is no one way to conduct a search and there are many variables involved. Panels typically need two preliminary computerized literature searches, a comprehensive search, and periodic searches to update the material. Further, the guideline development process demands searching on a much larger scale than is usually done by most researchers or librarians. Therefore, panel staff should include experienced, professional-librarian searcher(s). Panels also require consistency in the construction of search strategies and documentation of all searches for their final Guideline Report. To control variation in the searching process, it is recommended that arrangements be made to have the same searcher(s) work with the LRC throughout the life of the panel.
Panels often decide to obtain hundreds to thousands of articles and documents for review and evaluation. Retrieval and copying requires a high level of effort. Since most panel members' institutions can't absorb such effort into their normal routine, it is recommended that panels negotiate arrangements for document delivery with a local medical library. This library should be a member of the National Network of Libraries of Medicine (NN/LM) and located close to the Literature Review Coordinator and/or Panel Chair(s) who must consult with the local library throughout the document delivery cycle.
1.3 Obtain Computer Equipment and Reference Management Software
In order to manage the literature search and review processes, the Panel Chair and/or Literature Review Coordinator should have a personal computer (PC) with sufficient hard drive capacity to easily store and manipulate a very large bibliographic database. This PC should be equipped with reference management software and DOS. Additionally, a printer and a modem are essential.
ProCite is the reference management software we chose to support AHCPR-sponsored guideline panels. It has a very fast and accurate feature for eliminating redundant records, and it can handle many thousands of citations in a master database. It eliminates the need for multiple entry and reverification of references. ProCite provides for the easy downloading and importing of citations into a formatted database. ProCite also allows for editing of records so that codes indicating document status can be inserted during the literature retrieval and review phases. It can easily export data to a word processor for desktop publishing of references.
STEP 2: PRELIMINARY LITERATURE SEARCHING
2.1 Conduct Content and Volume Searches
Following Step 1, panel staff (i.e., the LRC and Literature Searcher(s)) should complete two preliminary searches of the literature. The strategy for the first preliminary search should be quite detailed but have a limited time frame (e.g., six months). Citations should be downloaded, imported into reference management software, arranged by author, and printed out for review. Reviewing this sample literature gives the panel an opportunity to scan for missing citations and to see if the search strategy pulled a high proportion of relevant citations. Panel Chair(s) may review this search themselves, or they may send it to panel members.
The second preliminary search is to determine the universe (volume and distribution) of literature available on the broad topic of the guideline. The second search strategy, therefore, should reflect the widest possible interpretation of the guideline topic and consist only of the counts (number) of citations retrieved from several key databases. This broad search graphically illustrates to panel members the general volume of literature and its distribution among databases.
In our analysis of the first three AHCPR-supported panels (Acute Pain, Urinary Incontinence, and Pressure Ulcers), it was found that MEDLINE was the source for between 77 and 91 percent of the literature used in these guidelines. Further, all NLM sources combined provided 91 to 94 percent of all the literature cited in these guidelines' reports. Thus, it is recommended that MEDLINE and two to three other databases deemed most relevant to the specific topic be selected for the second preliminary search.
2.2 Apply Search Results
As panels are interdisciplinary, members may only be familiar with the subset of the literature related to their specialty. The Panel Chair(s) and members need to obtain a clear picture of overall volume and content in order to determine the level of effort and resources required for reviewing the literature. They will then be able to weigh the available time (e.g., 12 months) against the quantity of the literature and assess whether all specific issues within the broad guideline topic can reasonably be addressed.
STEP 3: COMPREHENSIVE LITERATURE SEARCHING
3.1 Limit the Guideline Topic
As suggested above, the panel should review and assign priorities to possible guideline topics and subtopics. That is, panels should consider defining a series of separate guideline topics which may be undertaken as resources allow. Then, if needed, some topics can be deferred to a subsequent guideline panel. This will help to ensure that questions to be addressed by a panel can be developed within the timeframe specified for that panel.
Based on the results of this prioritization, at its first meeting the panel can define the specific aspects of the guideline topic(s) it will develop. The panel should clarify the topic(s) to be addressed in terms of the condition and types of patients for which the guideline is intended, the clinical interventions that will and will not be considered, and the type of provider and practice setting to which the guideline will apply.
3.2 Define Parameters of the Comprehensive Literature Search
While defining their guideline topic(s), the panel must also define the parameters of the comprehensive literature search by constructing statements or questions to be answered using the literature and by preparing lists of specific topics (e.g., a list of diagnostic tests) to be considered. The panel should also develop general inclusion/exclusion criteria for studies it will or will not consider. Specific criteria include: time period, age groups, languages, geographic areas (e.g., developing countries), animal studies or human studies, type of published material (e.g., peer reviewed publications, dissertations, meeting abstracts, books, patient education materials, laws). Some exclusion criteria can be handled in the search strategies (e.g., languages). Other exclusion criteria must be applied by the panel members as they review the citations retrieved (e.g., case reports involving fewer than 10 subjects).
Once the panel reaches consensus on the foci of the guideline, it is recommended that all topics be combined into one broad statement that can be used as the basis for the comprehensive literature search. Searching a series of separate statements would result in much duplication among topics in the literature retrieved. Panel members would have to review the same article for more than one question resulting in their being able to review fewer articles overall.
3.3 Determine Sources of Information
The foci of the guideline topic and the timeframe and resources available for completing the guideline should determine the sources of information. Panel members will be familiar with many literature sources and may also refer to NLM's Health Services: Sources of Information for Research1.
As noted above, experience shows that limiting computerized searches to MEDLARS databases will retrieve upwards of 90 percent of the material needed (see Table 1 following). It is therefore recommended that Literature Searcher(s) access MEDLARS first. Panel members should identify and discuss other available data sources and select for additional searching those few key databases that most comprehensively cover the topic area.
3.4 Conduct Searches
The Literature Searcher(s) should develop search strategies appropriate for each database to be searched. Terminology or keywords must be selected. Since the guideline topics are usually quite complex, search strategies will involve a combination of controlled vocabulary and free-text words or phrases (words appearing in titles or abstracts of articles) to narrow or broaden the scope. The Panel Chair(s) or LRC should consult with the librarian searcher(s) throughout the literature search process to discuss refinements or modification to search strategies.
Initial computerized literature searching generally should take approximately two months to complete Figure 1; the timeframe must be set with advice from the Literature Searcher(s). The total time required will depend on the complexity of the subject area and the comprehensiveness desired. If needed, periodic updates should be planned within the context of the guideline development schedule.
Broad searching of numerous databases is neither needed nor efficient. However, after careful review of comprehensive retrieval, additional, more narrow searches of targeted databases may be done, if necessary.
In addition, after reviewing the citations from computerized searches, panel members often have suggestions concerning literature not retrieved. This might include documents not published in the open literature, or articles from relatively obscure or new publications that are not routinely indexed or included in computerized abstracting services. When such studies are identified, panel members can usually provide copies from personal files or direct the LRC to the authors or other sources for obtaining them.
STEP 4: LITERATURE MANAGEMENT
It is recommended that the search results be merged into one large master bibliographic database. Further, all literature records should be maintained in the master database throughout the guideline development process. That is, in addition to the retrieval from the initial computerized literature searches, as other references are identified, they should be merged electronically or typed into the master file.
Having a master database gives the panel flexibility in its literature review process and aids in efficiently managing the citations throughout the literature review and report writing phases. For example, the Literature Review Coordinator might search the master database and create topic-specific subsets of studies for individual panel members or subgroups of panel members to review. Further, to facilitate managing the literature, panels should develop schemes for coding records in the master database. For example, panels may want to develop quality scores for study design or some other type of weighting criteria and input these into bibliographic records as literature is reviewed. Administrative data (e.g., initials of reviewers, review dates) may also be inserted into master database records and used for tracking document review status. Additional codes may be inserted into records to correspond to sections of the guideline or scientific evidence tables in which the literature will be cited. Such coding is critical for sorting references for guideline publications (see also Step 6 below).
STEP 5: DOCUMENT RETRIEVAL
As noted above, in addition to developing criteria for determining which studies out of the citations retrieved from the comprehensive search they want to obtain for review, the panel should arrange with a local library for document delivery. Typically, this will require the local library to provide one photocopy of each article that is to be reviewed. For items it does not hold, the local library may utilize the resources of the National Network of Libraries of Medicine (NN/LM) as well as other resource libraries to which it has access. Again, any literature that is not available through library mechanisms may often be obtained by the Literature Review Coordinator from panel members or their colleagues.
STEP 6: FINAL BIBLIOGRAPHY
In addition to being key to managing the literature review, the master database is the best source for the final bibliography. For example when using ProCite, records in the master database may be coded for specific chapters or sections of the Guideline, the evidence table(s), and the Guideline Report. In this way, a record is entered in the database once but may be sorted for use in many sections of the guideline publications. Since ProCite allows for easy exportation of records to a word processor, these sorted references may be incorporated into the guideline text and appendices without re-entry or the need for reverification. Regardless of the bibliographic management software used, NLM recommends that panels standardize and edit all references to conform to the style presented in NLM Recommended Formats for Bibliographic Citation2.
CONSULTATION AVAILABLE FROM NLM
NLM no longer provides extensive literature searching and document retrieval to Guideline Development Panels. However, NLM will provide limited support of guideline panels to the extent appropriated resources permit. This may include:
- Consulting on literature searches;
- Providing tutorials on Grateful Med, ProCite, and DOCLINE; and
- Providing back-up document delivery service through the NN/LM.
Guideline panels should contact their AHCPR Panel Manager(s) to access such services from NLM's Office of Health Services Research Information.
Table 1. Citations in AHCPR-sponsored Guidelines, by Source and Topic
|Acute Pain||Urinary Incontinence||Pressure Ulcers|
|Total n (%)||280 (100.0%)||514 (100.0%)||730 (100.0%)|
|n (%) from NLM sources||256 (91.4%)||482 (93.8%)||671 (91.9%)|
|n (%) from MEDLINE||215 (76.8%)||466 (90.7%)||570 (78.1%)|
|n (%) from CATLINE||7 (2.5%)||4 (0.8%)||22 (3.0%)|
|n (%) from HEALTH||2 (0.7%)||2 (0.4%)||13 (1.8%)|
|n (%) from other NLM||32 (11.4%)||10 (1.9%)||66 (9.0%)|
|n (%) from non-NLM sources||8 (2.9%)||29 (5.6%)||29 (4.0%)|
|n (%) Unverified1||16 (5.7%)||3 (0.6%)||30 (4.1%)|
Source: Cahn, Marjorie A., Selden, Catherine R, Auston, Ione. Summary report on analysis of AHCPR guideline panel literature. Bethesda (MD): National Library of Medicine; 1992 August. 5 p.
Note: 1. Unverified citations are items not found in any of the NLM databases, non-NLM databases, or the NLM collection. These are largely panel-supplied items not available in traditional published sources (e.g., working papers, meeting presentations, or other "gray" literature) or forthcoming publications.
Cahn, Marjorie A.; Selden, Catherine R.; Auston, Ione. Summary report on analysis of AHCPR guideline panel literature. Bethesda (MD): National Library of Medicine; 1992 Aug. 5 p.
National Library of Medicine (US). Health Services: sources of information for research. Bethesda, (MD): NLM; 1992 Aug. Report No.: NLM-PSD-92-01. 201 p. Available from: NTIS, Springfield, VA: PB92-221175.
Patrias, Karen. National Library of Medicine recommended formats for bibliographic citation. Bethesda (MD): National Library of Medicine; 1991. Report No.: NLM-LO-91-01. 248 p. Available from: NTIS, Springfield, VA: PB91-182030.
1.Available from the National Technical Information Service, Springfield, VA, (703) 487-4650; publication number PB92-221175.
2.Available from the National Technical Information Service, Springfield, VA, (703) 487-4650; publication number PB91-182030.