Skip Navigation Bar

National Information Center on Health Services Research and Health Care Technology (NICHSR)

Health Services Research and Health Policy Grey Literature Project: Summary Report

Conducted by AcademyHealth

Funded by the National Library of Medicine

 

February 2006

 

 

Background

Importance of Grey Literature

Users of Grey Literature

Long-Term Value of Grey Literature

Archival Practices of Producing Organizations

Recommendations

 

AcademyHealth is defining the scope of grey literature in health services research/health policy for the National Information Center on Health Services Research and Health Care Technology (NICHSR) at the National Library of Medicine. In considering the scope and status of grey literature, it is important to know who the major users are and what they are using grey literature for. In addition, knowledge of what grey literature is being produced and what the preservation procedures are for major producers is critical. Only by better understanding current practice can we make recommendations about what can be done to archive and preserve grey literature.

Background

Grey literature is "that which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers” according to the Luxembourg convention. This category includes reports, memoranda, conference proceedings, standards, technical documentation, and government documents.” (National Library of Medicine Web site, accessed November 2004). However, others may define grey literature more broadly to include everything except peer-reviewed books and journals accepted by Medline. Examples of grey literature publications include reports, theses, non-commercially published conference papers, government reports, translations, committee reports, working papers, technical specifications and standards, and bibliographies.

To better identify and evaluate the grey literature of health services research and health policy, AcademyHealth convened an Advisory Committee to identify and prioritize relevant disciplines, subject areas, and types of grey literature. To supplement the expertise of the Advisory Committee, a small number of grey literature-producing organizations were contacted to determine their preservation policy

Importance of Grey Literature

The importance of grey literature lies in its ability to communicate complex information in simple terms and to disseminate results more quickly. Much of the grey literature is derivative and designed to make technical material or research findings easily understood by a lay audience. For example, policy briefs, issue briefs, and fact sheets are often produced by summarizing more technical reports and providing context. They are also a way for organizations to get results out to their target audience more quickly. Many organizations, including research institutes like the Leonard Davis Institute, the Sheps Center, and the Urban Institute, produce working papers and issue briefs. Some organizations, like the Alliance for Health Reform, write policy briefs of interest to their audiences that summarize the findings of a number of individual research studies. Many of these research studies are included in the published literature.

A number of producers indicated that the material included in grey literature is ultimately incorporated into the published literature, although the content may differ somewhat from the grey literature. For example, AHRQ reported that some of the material produced under contract and grants is ultimately published by the researchers who do the work. Likewise, much of the research funded by entities such as The Robert Wood Johnson Foundation is published in health services journals. Although much of the grey literature is sourced in published literature, Advisory Committee members deemed this characteristic to be irrelevant since much of the published literature is likewise derivative in nature. Rather than compete with the published literature, grey literature has the potential to complement and communicate findings to a wider audience.

Grey literature is also a valuable barometer of public interest, reflective of current priorities and indicative of future ones. Many grey literature producers, including IOM, and GAO, cite congressional agendas and public interests as the driving force behind the determination of subject areas. The Alliance for Health Reform likewise expects management to determine specific topics within broad subject areas determined by the congressional and funding priorities. Accordingly, the focus of subject areas changes as priorities change. At any given time, the most popular items are topical and largely determined by what is currently in the news. Most recently, producers report an emphasis on topics including bioterrorism, disparities, information technology, patient safety, and obesity.

Users of Grey Literature

To identify the most frequent users of grey literature, AcademyHealth, with the assistance of the Advisory Committee, first identified the most prolific producers of grey literature in the fields of health services research and health policy. Since grey literature covers a wide range of subject areas within HSR, AcademyHealth reviewed web sites and meeting agendas to identify subject areas; Advisory Committee members then prioritized subject areas by relevance.

Subject Areas in Health Services Research

Access to Care*

Management & Organization

Behavioral Health

Maternal & Child Health*

Child Health

Medicaid*

Chronic Care and Chronic Care Delivery

Medicare*

Cost Containment*

Medical Records

Coverage

Mental Health*

Disparities in Health & Health Care

Minority Health*

Econometrics

Patient Safety

Emergency Preparedness/Bioterrorism*

Pharmaceuticals & Emerging Technologies*

Evaluating New Technologies

Post Acute Care

Gerontology*

Privacy

Health Care in the Workplaces

Private Insurance Markets*

Health Care Workforce

Program Evaluation/Evaluation Research

Health Insurance Markets*

Public Health*

Health Policy

Quality

Infectious Disease*

Substance Abuse/Addiction

International

Technology, Innovation & Evaluation

Long-Term Care*

Women’s Health

Managed Care*

 

* Voted most important by committee member

Advisory Committee members also identified various types of grey literature and their relative importance. The items used, in order of relevance, with 5 being most frequently used by committee members :

 

5

4

3

2

1

  • Working Papers
  • Testimony
  • Committee Reports
  • Conference Proceedings

 

  • Data
  • Evaluations
  • Foundation Reports
  • Government Reports
  • Grantee Publications
  • Non-commercially Published Conf. Papers
  • Reports
  • Special Reports
  • Speeches
  • Annual Reports
  • Presentations
  • Grantee Reports
  • Webcasts
  • Theses
  • Technical specifications & standards
  • Newsletters
  • Bibliographies
  • Bulletins
  • PowerPoint Presentations
  • Foundation Financial Statements

 

  • Pamphlets
  • Protocols
  • Guidelines
  • Poster Sessions
  • Meeting Agendas
  • Translations

 

There was consensus among committee members, and the producers contacted confirm, that the most common target audiences for grey literature products are policymakers, reporters, federal and state agencies, foundations, researchers and grantees.

  • Policymakers

Overwhelmingly, producers cited policy makers and decision makers as their first priority. Decision makers include members of Congress and congressional staff, federal and state agencies and regulators, and policy analysts. In the case of policymakers, grey lit is used more than non-grey lit, because grey lit is generally more up to date and is more accessible. Cost effective, timely data is often simply not available in published literature. In addition to this, policy documents are not generally published in peer-reviewed journals. Grey literature materials of greatest relevance to policymakers are sourcebooks, chartbooks, evaluations, consensus reports, fact sheets, briefings, transcripts, and issue briefs.

  • Media

Reporters and the media in general were the next group most cited by producers. That decision makers and reporters comprise the main target audience is not surprising when coupled with the fact that most producers described their grey literature products as a means of communicating complex issues to lay persons for the purpose of creating public support and awareness for a particular issue. The materials of greatest relevance include summaries, issue briefs, chart books, source books, fact sheets, transcripts, testimonies, and conference proceedings.

  • Foundations/Funding Agencies

Foundations and funding agencies rank third in terms of priority audiences. Foundations use grey literature to see where research needs are and to determine the course of their research. Technical reports and chartbooks are often produced at the request of funders to summarize the body of work produced under a research contract or grant, and many of these are disseminated widely and are available on their websites. Likewise, most producers cited a high degree of policy relevance as necessary for consideration. Materials of greatest relevance include grants results reports, evaluations, research reports, syntheses, newsletters, and news summaries.

  • Researchers

Researchers rank last in terms of audience importance for producers of grey literature. Researchers use grey literature as a resource to support their research, as well as to identify funding priorities. Likewise, grantees and prospective grantees look to grey literature to determine what is being funded by whom to help determine the course of their research. Materials of greatest relevance include datasets, evaluations, research reports, training materials, working papers, surveys, newsletters, program and issue briefs, and listserves.

Long-Term Value of Grey Literature

Grey literature has long-term value, particularly because it provides policy context and implications that may not be found in the published literature. In fact, advisory committee members believed that the value of grey literature is on par with that of traditional published literature. Relevancy, progress, and how debate changes over time on a particular topic can be assessed from these materials. Another use of grey literature is to establish historical documentation. The progress of a document to its finished form can sometimes be as valuable as the finished product, and the various drafts of a document can fill in gaps in the historical record.

Notably, published literature and grey literature alike cite each other. As described earlier, grey literature is derivative in nature, summarizing and communicating complex issues to a lay audience. What is perhaps not as well understood is the corresponding role of grey literature in developing published literature. For example, journal articles, in particular, make frequent use of grey literature to support claims. Likewise, briefs and summaries often rely on disparate sources of information found in grey literature. Even textbooks incorporate new findings and perspectives derived from grey literature when updating new versions. Grey literature often illustrates the progression of knowledge and supports the development of new published works. To lose certain grey literature products would be to lose many of the very sources of published literature.

Archival Practices of Producing Organizations

The grey literature producers did not always have clear and consistent policies regarding what to preserve and how to preserve materials. Although technology has made it less costly to store information electronically, and much information is archived online, material may be accessible only as long as the organization is in existence. Every producer interviewed acknowledged having hard-copy archives either in an on-site library, off-site storage, or in the care of a storage and fulfillment service such as the National Technical Information Service (NTIS). In general, producers internally archive materials electronically in order to fulfill requests for materials with reprints rather than original copies. Many of the producers indicate that they save materials on-line. All indicated their willingness to share material with researchers who make requests, but acknowledge that it may be difficult for researchers to know exactly what materials are available, especially if they are not available on-line.

Federal agencies follow regulatory guidance regarding preservation; this guidance includes Generally Accepted Accounting Standards and National Archives Standards for the General Accounting Office, NARA Schedule of Retention Regulations, and NTIS. AHRQ and other federal government agencies send reports and other final grant and contract products to NTIS for archiving and preservation. Some of the material in these reports ends up in the published literature in the form of journal articles. More transient materials, such as fact sheets and issue briefs, are not preserved in any organized fashion. Many of these materials, such as fact sheets, are updated periodically and so the content may change over time.

All reports produced by the Institute of Medicine receive Library of Congress numbers, and are preserved according to the Library of Congress standards. In general though, the producers interviewed did not pursue or receive Library of Congress numbers for their grey literature products.

Recommendations

Grey literature is a significant resource in the fields of health services research and health policy. Preserving or archiving it in some way is important, and there may also be a role for an organization to take the lead in preserving this material in a way that will act as a quality seal of approval. This can be accomplished either through directly archiving the material, or creating and providing guidance to producing organizations about appropriate preservation procedures. Transparency is also needed about the creation and review process of grey literature to enable users to better assess the products.

Archiving and Preservation

  • Physical or Electronic Archive : Increasingly, the Internet has enabled organizations to disseminate and archive grey literature materials more easily and at less cost. Likewise, users increasingly turn to the Internet for their information needs. Accordingly, it appears that a central, physical archive of materials may be both impractical and unnecessary. Instead, an electronic archive may be a better option to house appropriate items, as well as make retrieval of these items easier for users. While a clearinghouse with links to other organizations is another option, it does not appear to be as useful given that some organizations internally archive their material, making it inaccessible via links. In addition, some grey literature (such as fact sheets) is continually updated and so may not serve the historical purpose of preservation.
  • Transparency Regarding Preservation Policies : The encouragement of transparency for preservation policies would greatly improve users’ ability to determine the long-term status of individual products. Organizations should be encouraged to provide transparent information on the availability of their products, e.g., whether the organization contains a central library which retains paper copies and whether electronic formats are available. At the item level, users should be informed whether the product is available in hard-copy or electronic format and for how long the product is expected to be available.
  • Development and Dissemination of Best Practices : In the absence of a nation-wide central archive, producers would benefit from the development, production, and distribution of guidance (brochure or pamphlet) on best practices when archiving materials. At the organizational level, organizations should be encouraged to inform users of how long publications are expected to be accessible and how. NLM can share its Web archiving decisions to groups as a guide. For instance, NLM tags Web pages with visitor information with ‘Permanence Not Guaranteed’. Guidelines could be disseminated to Communication and Information Technology Directors at producing organizations, either electronically or in brochure form.

Ensuring Uniform Quality of Grey Literature

  • The quality of the research that appears in grey literature is highly variable, even within grey lit produced by the same organization. The value that a respected organization would bring to a grey literature preservation project is the imprimatur of legitimacy. That organization could establish standards or criteria to promote quality. For example, organizations could be encouraged to include a brief text within each product describing 1) how the product was produced and 2) how the product was reviewed, particularly in regard to conflicts of interest. While the problem of variability of quality is not unique to grey literature, Advisory Committee members agreed that transparency with regard to the production and review of grey literature would significantly increase its long term value.