Outreach impact study: the case of the Greater Midwest Region *

Objectives: The purpose of the study was to assess the impact that funding from the National Network of Libraries of Medicine (NN/LM), Greater Midwest Region (GMR), has on member institutions’ ability to conduct outreach on behalf of NN/LM. Methods: The study employed both content analysis and survey methodologies. The final reports from select GMR-funded outreach projects (n 5 20) were analyzed based on a set of evaluation criteria. Project principal investigators (n 5 13) were then surveyed using the same evaluation criteria. Results: Results indicated that outreach projects supported by GMR funding improved access to biomedical information for professionals and the general public. Barriers to conducting outreach projects included time constraints or commitments, staffing, scheduling and absenteeism, inadequate space, and issues associated with technology (e.g., hardware and software, Internet connectivity and firewall issues, and creation and use of new technologies). Conclusions: The majority of project principal investigators indicated that their attempts to conduct outreach were successful. Moreover, most noted that outreach had a positive impact on professionals as well as the general public. In general, it seems that negative outcomes, as with most barriers to conducting outreach, can be mitigated by more thorough planning.

The mission of the National Network of Libraries of Medicine (NN/LM) is to advance the progress of medicine and improve the public health by providing all US health professionals with equal access to biomedical information and improving the public's access to information to enable them to make informed decisions about their health [1]. The National Library of Medicine's (NLM's) vision statement emphasizes the elimination of health disparities. The National Institutes of Health (NIH) and NLM both attempt to understand and eliminate health disparities between minority and majority populations [2]. One of NLM's goals in trying to achieve elimination of health disparities is to improve access to affordable and easy-to-use health-related information [3]. It is thought that such a dissemination of knowledge will help solve health disparities [4]. Ultimately, NLM believes that improved access to health information will result in higher quality health care. This effort is served through research and community outreach. NLM supports some outreach directly through the NLM Division of Specialized Information Services (SIS), Office of Health Information Program Development (OHIPD), and extramural programs. In addition, NN/LM, which NLM administers, provides and supports outreach as part of NLM's overall mission [5]. NN/LM places a primary emphasis on rural, minority, and other underserved populations [6].
NLM was interested in evaluating NN/LM programs, and one of the key programs of the Greater Midwest Region (GMR) is its funding to support outreach in the region. NLM was also interested in promoting awareness among library school students about NLM and NN/LM and about the value of libraries doing outreach to underserved populations with the community. In September 2009, NN/LM GMR announced the availability of funds for a subcontract to a library school located in the region to assist with evaluating network programs. The GMR operates under a contract from NLM, providing network services to approximately 1,100 libraries * This work was performed under a subcontract with the University of Illinois at Chicago and made possible by grant number N01-LM-6-3503 from the National Library of Medicine, and its contents are solely the responsibility of the authors and do not necessarily represent the official views of the National Library of Medicine.
Supplemental Appendixes A, B, and C and supplemental Table 3 are available with the online version of this journal. N The provision of GMR funding to conduct health information outreach yielded positive outcomes. Implications N Regional offices should make efforts to accurately capture and record the impact that funding has on member institutions' ability to conduct outreach. N Many barriers to conducting outreach can be mitigated by more thorough planning. and information centers in a 10-state region (Illinois, Indiana, Iowa, Kentucky, Michigan, Minnesota, North Dakota, Ohio, South Dakota, and Wisconsin). One of 8 regions in the NN/LM, the GMR conducts much of its outreach by offering subcontracts and awards to network members.
Funding was offered to engage a group of students, under faculty guidance, to plan and implement a study that would determine the impact that GMR funding has on the ability of network members to perform outreach on behalf of NN/LM. The study was intended to be conducted in two phases: a planning phase, in which students were to develop an impact study proposal, and an implementation phase, in which students were to implement the study outlined in the proposal. The desired outcomes of the impact study included: & What did the institutions accomplish with the NN/ LM funding? & What outcomes did the institutions report from their NN/LM-funded outreach projects? & What changes have been made in their institutional programs as a result of the funding they received? & What lessons did the institutions learn through implementation of their NN/LM-funded outreach projects & What changes would the institutions recommend to NN/LM about their funding programs?
The School of Library and Information Science (LIS) at the University of Kentucky received an outreach impact study award from NN/LM GMR in the fall of 2009 for its project, ''Learning By Doing: Engaging LIS Students in an Outreach Impact Study.'' Project funds were used to provide scholarships to support student participation.

METHODOLOGY
The project was divided into two phases: ''Phase I: Planning Phase'' and ''Phase II: Implementation Phase.'' During phase I, select LIS students, working under faculty supervision, developed an outreach impact study proposal. A notice promoting the project was posted to the school's electronic student discussion list. Interested students were invited to apply to participate, based on their career goals, work experience, and completed course work. At the beginning of spring semester 2010, students met with the faculty supervisor and the GMR outreach and evaluation coordinator to go over project objectives and review relevant research methodologies. Based on this meeting and subsequent online discussion regarding research methodologies, students came to consensus about the research study approach and design. Students then began working in groups to draft sections of the study proposal as well as the internal review board (IRB) application. All work was posted in Blackboard for ongoing peer review and comment. At the end of spring semester 2010, a draft study proposal was submitted to the GMR office to route for peer review. In addition, a completed IRB application was submitted to the University of Kentucky's Office of Research Integrity for approval. Based on reviewers' comments, minor revisions were made to the proposal and IRB application (IRB number 10-0380-P4S), and both were subsequently approved. During phase II, the proposed outreach impact study was executed. Working under faculty guidance, the same five LIS students who participated in phase I of the project completed phase II during fall semester 2010.
A combination of content analysis and survey methodologies was deemed suitable to meet the overall goal of the outreach impact study. Content analysis is a method of analyzing written, verbal, or visual communication [7], whereas survey methodology is a nonexperimental research method that is particularly useful in collecting data on phenomena that cannot be directly observed [8].
For the contract period under study (May 2006-April 30, 2011, the GMR funded 2 levels of projects: subcontracts and awards. Awards are in amounts less than $5,000; funded amounts of $5,000 and over are called subcontracts and require completion of additional support documentation. GMR subcontract recipients receive detailed reporting instructions as part of the funding notification. Reporting requirements include submission of quarterly reports and a final report. For project final reports, all funding recipients are instructed to provide a report that includes a description of the project that summarizes major accomplishments, training provided, meetings held, and websites created as well as barriers encountered; perceived impact on the library, institution, or consortium; and recommendations for improvement. Detailed final report instructions are issued for subcontract awards (Appendix A, online only). Recipients are also asked to comment on whether project goals were met, what lessons were learned, and what advice they have for others.
Inductive content analysis was performed on final reports submitted for select GMR-supported outreach projects funded on or before May 1, 2006, andcompleted by December 31, 2009. The content analysis was designed to: 1. identify outcomes reported by institutions from their NN/LM-funded outreach projects 2. determine the impact that funding had on recipient institutions' ability to conduct local outreach 3. report examples of improved access to biomedical information for professionals and examples of improved access to biomedical information for the public 4. report changes made by institutions as a result of the funding 5. identify barriers to outreach reported by participating institutions The GMR office selected twenty-two outreach projects to be included in the outreach impact study analysis. In selecting projects for the study, the GMR office reviewed projects funded to date for 2006 to 2011 contracts. The criteria for selecting projects for the study were based on date of completion and project focus. Projects that had concluded or would be concluded by December 31, 2009, were considered for Huber et al. inclusion. Only awards and subcontracts that were of a true outreach nature were included. Excluded were technology improvement, exhibit, electronic fund transfer system, and professional development and professional instruction awards and the MedlinePlus Go Local subcontracts.
Two of the twenty-two projects did not submit adequate documentation and were not included in the content analysis. The remaining twenty final reports were divided into two groups based on the monetary amount of the award and the type of subcontract awarded (i.e., consumer health subcontracts, health disparities subcontracts, public health subcontracts, and outreach express awards). Smaller awards and certain types of subcontracts typically result in shorter and less detailed final reports. The distribution enabled reviewers to evaluate similar numbers of each report type, resulting in an even workload among reviewers. Reviewers were instructed to examine final reports in their entirety for evidence of the five objectives. Evidence was defined as a stated concept that reflected the content of the objective, rather than a specific word or phrase. Reviewers were instructed to record examples of the evidence found for each objective and note where that evidence was located in a final report. Appendix B (online only) shows a sample coding sheet. Two student reviewers were assigned to each group of ten final reports, and the faculty supervisor read all twenty final reports, totaling three individual reviewers for each final report. A minimum of two reviewers came to consensus over data contained in each final report to ensure inter-rater reliability prior to posting results for further peer review by the entire research team. Once consensus was reached by all members of the research team, analyses were reviewed for common themes.
In addition to the content analysis, the principal investigators (PIs) of the twenty-two outreach projects were asked to take part in a survey. The survey was structured around the same five objectives used to conduct the content analysis of final reports. Survey questions were a mixture of open-ended and closedended questions (Appendix C, online only). The survey sought to elicit details that may not have been reflected in the project final reports. The survey was mounted in SurveyMonkey ,http://www.surveymonkey.com., an online, third-party commercial questionnaire system. Project PIs received an email invitation to participate in the survey. The email invitation included a brief description of the project as well as a link to the survey. The GMR office also contacted project PIs and encouraged them to participate in the survey.

Content analysis results
Several themes emerged from the content analysis. Table 1 shows a list of occurrence of themes by objective, and Table 2 shows sample quotations that provide examples of themes within each objective. Outcomes reported by institutions from their NN/ LM-funded outreach projects included developing new partnerships, gaining enhanced access to information through technology, increasing awareness about resources and population needs, increasing proficiency in resource use, and providing training or education for professionals and the public. While no direct examples were found of funding having had an impact on recipient institutions' ability to conduct local outreach, reviewers found indirect statements to support this outcome. Having collaborations among library professionals and health professionals, ad- Outreach impact study dressing health literacy issues, and providing computer hardware were all common themes related to this objective. Many examples were found for objective 3, improved access to biomedical information for professionals and the public. It was noted in several reports that investigators helped members of the public or health professionals to use a variety of resources. Examples of resources included Medline-Plus, PubMed, ClinicalTrials.gov, Kidshealth.org, and Go Local. Investigators reported some changes made as a result of funding (objective 4), with some of these being indirect. One example was the continuation of relationships between the organizations that were awarded the subcontracts and the groups with whom they were conducting outreach. Another common theme was a continued commitment to using the technologies (e.g., podcasts, websites, portals, etc.) that were introduced during the subcontract time period, including the intention to hold additional training sessions. One minor theme related to objective 4 was to seek additional funding to continue the projects indefinitely. Several examples of barriers to outreach reported by participating institutions were readily identified. These included technological barriers (e.g., hardware and software, Internet connectivity and firewall issues, creation and use of new technologies), scheduling and absenteeism, staffing, time commitments, and inadequate planning.

Survey results
Of the 19 PIs for the 22 outreach projects, 13 submitted responses to the survey, yielding an overall response rate of 68%. However, not all PIs responded to each question. Table 3 (online only) provides a summary of demographic information that PIs reported. In response to the survey, a majority of the PIs indicated that their projects were very successful and the funding had a substantial impact on their outreach projects. Networking and community were the two areas most frequently mentioned as having been most affected by funding. Survey results are summarized in Table 4.
When asked to provide specific examples of how GMR funding improved access to biomedical infor- ''Outcome measures of the workshops and the online tutorials reported a significant increase in proficiency among participants which impacted their ability to access and retrieve the evidence.'' ''The computer equipment, accessories and supplies, and software that they received helped them to access online resources like MedlinePlus … and to share accurate, understandable health information with individuals and groups.'' ''provided health literacy training to over 2,100 healthcare professionals, 498 healthcare students, 311 public and academic librarians, and over 1,000 consumers.'' ''Instruct[ed] teens on how to find reliable health information on the Internet.'' ''498 students have received health literacy training, complete with hands-on learning experiences … 46 faculty also received training, and have incorporated the principles of health literacy into their curriculums.'' Objective 3: Report examples of improved access to biomedical information for professionals and the public ''This project has had an impact by raising awareness of MedlinePlus, some other reliable consumer health resources, PubMed, and the reference service and resources available to health providers and the public from the … Libraries.'' ''74% [of participants] indicated that they could identify web-based resources for evidence based public health practice.'' ''This project … helped to improve access to accurate health information to community members and to health care providers who serve them in a free health clinic.'' Objective 4: Report changes made by institutions as a result of the funding ''The project staff members have become more active in public health professional organizations and have been invited to speak to numerous public health organizations both regionally and nationally.'' ''Sessions are continuing to be developed for different audiences … The upcoming talks will be made to public health students, public health researchers, academic and public librarians and historical audiences.'' ''These partnerships continue to this day and have been strengthened as a result of this project.'' ''This project gave us the confidence to pursue funding to continue this project and publish the Hmong Family Health Guide, a goal established in 2004.'' ''Most impressively, the intranet that was created became the home page on all staff members' computers and the director of information technology spearheaded the project internally after the subcontract ended.'' Objective 5: Identify barriers to outreach reported by participating institutions ''Technical difficulties with the hardware and software caused a lot of delays.'' ''The lack of computer knowledge of some of the nurses, and limited Internet access in the central city … made it a more involved project than we expected for a subcontract award of this size.'' ''Hospital firewalls were a challenge in all the hospitals.'' ''Scheduling the training sessions with the medical staff in community clinics proved to be difficult due to the tight schedules of the staff.'' ''No matter how much we overbooked our 'full' workshops, we still had absenteeism.'' ''The use of volunteers requires much coordination of schedules and works best with those closely connected to the project.'' ''We were not realistic in determining how much time the community partners would need to spend on this project.'' ''We gravely underestimated some of our costs.'' ''A key lesson as well was that two of our objectives were perhaps too grandiose.'' * Note: No direct examples of objective 2 (Determine the impact funding had on recipient institutions' ability to conduct local outreach) were found in final reports. Huber et al. mation for professionals, the 3 main reasons given by PIs (n510) were providing equipment, offering training, and facilitating networking opportunities. They were then asked for specific examples of how GMR funding improved access to biomedical information for the public. The areas PIs commonly listed (n510) were providing equipment, promoting NLM resources, and offering educational opportunities. When asked if changes were made by the institutions as a result of the GMR funding, 10 out of 13 responded. Of those 10 respondents, 6 believed that changes were made by their institutions and 4 believed changes were not made. Next, PIs were asked to describe the changes that were made as a result of the GMR funding. The changes that the 6 respondents most identified were facilitation of networking opportunities and staffing. The PIs were asked to describe barriers that were encountered to successfully planning and implementing the GMRfunded outreach project. Barriers that PIs identified most frequently (n58) were time constraints, staffing issues, and inadequate space.

Limitations
The National Network Office (NNO) issues minimum reporting requirements for subcontracts of $15,001 or greater. Each NN/LM region must follow these guidelines. However, regions may add or tailor requirements specific to their needs. Awards of

DISCUSSION
Both content analysis and survey results indicated positive outcomes resulting from GMR-funded outreach projects. While the survey showed that PIs were confident that providing GMR funding made a positive impact on the ability to conduct outreach, no specific examples of this were given in the final reports. Although some reports mentioned changes made by their institutions as a result of funding, like ongoing relationships and a continued commitment to outreach, there were few specific statements about the direct link between funding and its impact on conducting local outreach. Given that final report instructions asked subcontract recipients to describe work that was accomplished with GMR funds and how project continuation plans would be funded, one might expect statements reflecting the relationship between receiving funds and the impact those funds had on the ability to conduct outreach initiatives. On the other hand, survey results revealed that funding gave rise to networking and community building, and comments from nine of the survey participants indicated that their institutions would not have been able to complete the outreach project without the funding provided by the GMR. PIs indicated, both in final reports and survey responses, that there was improved access to biomedical information for professionals and the public. Outreach in general was believed to be beneficial to the parent institution as well as to the target audience. In several reports, investigators noted that their institutions helped members of the public and health professionals to use the library and various resources. They noted in the survey that training, education, and networking occurred. Several reviewers noted in most reports, that changes made by the institution as a result of having received GMR funding were again only apparent through inference. Sixty percent of survey respondents replied that changes were made as a result of the GMR funding, and comments indicated that among these changes were a higher priority for conducting outreach and a strengthening of relationships between the institutions and their communities. Two reviewers suggested that a recommendation be made to the GMR office to add final report instructions specifically addressing changes made by institutions as a result of funding.
Although conducting outreach initiatives is one way to improve or enhance access to biomedical information, it is not without problems. Barriers to conducting outreach projects were identified in both the content analysis of project final reports and the survey administered to project PIs. Many of the barriers encountered might have been prevented with more thorough planning prior to beginning an outreach project. For example, negotiating roles and responsibilities with partnering agencies in advance might have alleviated some issues associated with staffing and technology. Or developing a more stringent work plan might have mitigated some time constraint or commitment issues by allowing individuals to plan accordingly.

Recommendations made to Greater Midwest Region office
The following recommendations were made to the GMR office as a result of the study. The recommendations were made in an effort to more accurately capture and record the impact that GMR funding has on member institutions' ability to conduct outreach on behalf of the NN/LM. 1. It was recommended that the GMR office add a section to the final report instructions that specifically asks award recipients to indicate how funds had an impact on the institution's ability to conduct outreach. The question could be asked, ''Would the project have occurred without the funding; and, if so, how much more was accomplished with the funds than would have been accomplished without them?'' 2. It was recommended that the GMR office add final report instructions that specifically ask award recipients to indicate what, if any, changes were made as a result of receiving funds to support outreach efforts so that this information is more obvious and less inferred.

CONCLUSION
According to project PIs, GMR funding overall had a positive impact on an institution's ability to conduct outreach. By virtue of survey responses, the majority of PIs indicated that their attempts at outreach were successful. This does not appear to depend on years of professional experience or level of position held. Moreover, most noted that outreach had a positive impact on professionals as well as on the general public. In addition, outcomes associated with conducting outreach projects were generally positive. It seems that negative outcomes, as with most barriers to conducting outreach, can be mitigated by more thorough planning.