Friends and colleagues of Donald A.B. Lindberg M.D. came together to give tribute to his extraordinary contributions during his tenure (1984–2015) as Director of the U.S. National Library of Medicine (NLM). Dr. Lindberg died in 2019. The book, Transforming biomedical informatics and health information access: Don Lindberg and the U.S. National Library of Medicine. includes four sections. The ten edited chapters in section three (the Outreach section) are briefly summarized in this overview. As Associate Director for Health Information Programs Development, Elliot R. Siegel Ph.D. coordinated NLM’s outreach programming under Dr. Lindberg’s leadership from its inception in 1989 to his own retirement in 2010. Dr. Lindberg’s legacy at NLM is one of new possibilities imagined, significant changes made in the mission and ethos of a venerable institution, and numerous successes achieved in a variety of settings and contexts. Like so much else Dr. Lindberg accomplished, these Outreach programs that profoundly changed the character of NLM would likely not have occurred without him. He made a difference.

The U.S. National Library of Medicine (NLM) exhibition known as Native Voices reflected Donald A.B. Lindberg M.D.'s keen and long-held desire to help improve public understanding of Native American health challenges and honor the culture, tradition, and healing ways of Native Peoples. A centerpiece of the exhibition was a large set of video interviews that Dr. Lindberg conducted with Native health and community leaders. Dr. Lindberg and his team engaged Native advisors in the exhibition development; sought Native input through Listening Circles, Tribal Consultations; and site visits, and made the video interviews accessible via interactive kiosks and iPads. For its time, this was state-of-the-art exhibition technology. The exhibition also included Native artifacts and art works to complement the videos, including a scale model of the iconic Hokule'a Native Hawaiian voyaging canoe, and a full-size Lummi Indian Healing Totem Pole. The totem journeyed across the U.S. prior to its installation next to the NLM herbal garden in Bethesda, MD. A traveling version of the exhibition visited more than 130 venues in 40 States across the U.S. The interview clips and other content are accessible on the exhibition website, and the full-length interviews are retained in the NLM permanent video collection.

Donald A.B. Lindberg, M.D., brought with him when he joined NLM an inquisitive mind, tech savvy, and new ideas. He was an early advocate of both outreach and evaluation innovation at NLM. Dr. Lindberg initiated and supported multiple pilot test and implementation projects to strengthen NLM's health information outreach to healthcare providers, research scientists, health science and hospital librarians, and the general public, including minority and underserved populations. He helped steer NLM's transition to the Internet, and NLM's development of a robust framework for evaluating Internet and Web-based health information dissemination and outreach to its many audiences. Dr. Lindberg's leadership led to numerous landmark accomplishments, including the capacity-building "Measuring the Difference" outreach evaluation Guide, and a multidimensional approach to Internet and website evaluation that placed NLM at the forefront of federal agencies using these new and emerging technologies to support their missions.

The National Library of Medicine's AIDS Community Information Outreach Program (ACIOP) supports and enables access to health information on the Internet by community-based organizations. A technical assistance (TA) model was developed to enhance the capacity of ACIOP awardees to plan, evaluate, and report the results of their funded projects. This consisted of individual Consultation offered by an experienced evaluator to advise on the suitability of proposed project plans and objectives, improve measurement analytics, assist in problem resolution and outcomes reporting, and identify other improvement possibilities. Group webinars and a moderated blog for the exchange of project-specific information were also offered. Structured data collections in the form of reports, online surveys, and key informant telephone interviews provided qualitative feedback on project progress, satisfaction with the TA, and the perceived impact of the interventions on evaluation capacity building. The Model was implemented in the 2013 funding cycle with seven organizations, and the level of reported satisfaction was uniformly high. One-on-one TA was requested by four awardee organizations, and was determined to have made a meaningful difference with three. Participation in the webinars was mandatory and high overall; and was deemed to be a useful means for delivering evaluation information. In subsequent funding cycles, submission of a Logic Model will be required of awardees as a new model intervention in the expectation that it will produce stronger proposals, and enable the evaluation consultant to identify earlier intervention opportunities leading to project improvements and evaluation capacity enhancements.

Mentoring in Medicine (MIM) addresses an urgent national need for minority health professionals and promotes careers in health care for urban youth.The MIM After-School Program (ASP or The Course) has as its primary objectives to provide academic enrichment in human biology and motivate disadvantaged youth to pursue a career in the health professions.Secondary objectives of The Course, although not evaluated here, are to improve students' health literacy and knowledge of healthy living behaviors.Since 2009, over 1500 middle and high school students have completed the New York City based Course, which is offered once a week over a 10 week semester in an out-of-school venue.This study assesses the success of The Course in achieving its primary objectives with 84 students at five New York City high schools during the fall 2014 semester.The Course curriculum was created especially for MIM, comprises the body's 11 organ systems, and is presented in discrete modules (one each semester), along with complementary educational activities, including field trips and class projects.This study reports on a formal evaluation using quantitative and qualitative methods.The quantitative evaluation found that the students significantly increased their knowledge of the Gastrointestinal System.Students across the academic spectrum appeared to have learned the MIM ASP Course content -high school GPA was not a predictor of knowledge acquisition.The students also reported that The Course significantly increased their self-confidence in their ability to succeed (self-efficacy).The students expressed a significant increase in five health care related attitudes and an additional increase in their ability to overcome personal issues to succeed in their career and significantly improving their feeling toward, and likely pursuit of, a health career.The students stated that The Course significantly increased their interest and intent to seek out more information about health care, participate in health care activities, and take more health care courses in high school.The qualitative evaluation found that the students and their parents were pleased with the MIM ASP Course's composition, presentation and effectiveness.With a large majority of the parents stating that their child got out of The Course what they had hoped for and that The Course made it more likely that they would recommend a health career for their child.The students and instructional staff also identified The Course elements that they felt were most and least effective.Best practices that were used in designing and conducting The Course were identified.The MIM ASP Course appears to have achieved its principal educational objectives of providing academic enrichment in human biology and improving attitudes towards a health career for a self-selected population of disadvantaged, underrepresented minority high school students in an urban setting.

The AIDS Community Information Outreach Program (ACIOP) was created in 1994 to assist the affected community in utilizing electronic HIV/AIDS information resources. Nearly 300 competitive awards have been made to mostly community-based organizations. A formal evaluation was undertaken to determine the performance and impact of the ACIOP. A mixed methods design combined quantitative abstractions and summarization of 47 awardee final reports from 44 organizations, and qualitative telephone interviews with 17 individuals representing 20 projects. Findings revealed that project objectives were mostly met; high-risk populations were reached; low resource organizations were funded; community partnerships were significant; projects built on existing efforts; information resources and training were tailored to local needs; and most projects overcame barriers experienced. Needed modifications centered on: 1) enhancing evaluation capacity at the individual project level and 2) revising project reporting requirements to increase the amount of information available to assess the ACIOP; both have been implemented.

The National Library of Medicine's (NLM) AIDS Community Information Outreach Program (ACIOP) was launched in 1994 to provide the HIV/AIDS affected community with access to vital health information resources increasingly becoming available on the Internet. Three hundred awards have been made mostly to community-based organizations. An evaluation in 2012 found that most program objectives are being met; a principal recommendation going forward was that NLM seek to enhance the capacity of community-based awardees to conduct evaluations of their own projects. This article reports on a workshop whose invitees were drawn from AIDS serving organizations, along with scientists, clinicians, and information technologists, to review the evaluation findings and recommendations. They considered alternatives for improving awardees' evaluation capabilities with the help of expert consultation, identified additional steps that could be taken to make individual project results more transparent and sharable, and looked at external influences ranging from mobile health devices to the latest HIV/AIDS scientific research findings that could be used to align future awards with unmet needs in the community. The paper identifies efforts subsequently made by ACIOP managers to prioritize and operationalize guidance from the evaluation and the workshop, and discusses the benefits of community engagement.

There are many obstacles that urban youth experience in pursuing health careers, but the benefits of diversifying the classroom and workforce are clear. This is especially true today as educators and policymakers seek to enhance underrepresented minority students' access to health careers, and also achieve the health workforce needed to support the Affordable Care Act. The creation of student pipeline programs began more than 40 years ago, but success has been equivocal. In 2008, Mentoring in Medicine (MIM) conducted a research project to identify how students learn about health careers; develop strategies for an integrated, experiential learning program that encourages underrepresented minority students to pursue careers in health; and translate these into best practices for supporting students through their entire preparatory journey. Six focus groups were conducted with educators, students, and their parents. The inclusion of parents was unusual in studies of this kind. The outcome yielded important and surprising differences between student and parent knowledge, attitudes and beliefs. They informed our understanding of the factors that motivate and deter underrepresented minority students to pursue careers in health care. Specific programmatic strategies emerged that found their place in the subsequent development of new MIM programming that falls into the following three categories: community-based, school-based and Internet based. Best practices derived from these MIM programs are summarized and offered for consideration by other health career education program developers targeting underrepresented minority students, particularly those located in urban settings.

The Virtual Science Camp VSC is a unique demonstration of synchronous e-learning developed by Mentoring in Medicine MIM. This paper reports on a pilot offering during the summer of 2012 that taught advanced biological concepts, healthy living and health care career opportunities to medically underserved urban youth. Livestream's interactive video technology was used to engage a diverse audience of mostly high school students at remote sites in a new two week instructional program that provided custom course content free of charge over the internet. We describe the technical and program preparations undertaken, their implementation, the IT environment, a multi-faceted evaluation plan, the results of the experiment, and lessons learned.

The paper presents an overview of the National Library of Medicine's 1988 field test and evaluation project, including the participation of 21 user sites, on the use of MEDLINE products on CD‐ROM, and reports on the subsequent evaluation forum held at the NLM on 23 September 1988.

Racial and ethnic healthcare disparities remain after differences in income, access, and insurance status have been considered, partly because of the healthcare delivery system’s failure to respond to cultural differences. The Institute of Medicine has called for the development and deployment of culturally appropriate healthcare services to mitigate these disparities. This complex problem of determining how to address the cultural components of racial and ethnic healthcare disparities is an example of what Russell Ackoff terms “messes.” Given consumer health information technology (IT)’s increasing role in patients’ self-care and self-management, one potential solution lies in designing consumer health IT that is culturally-informed. Although both the healthcare informatics community and the engineering design community have begun to seriously consider the role of culture in design to enhance usability, much work remains. Unfortunately, creating culturally-informed consumer health IT can seem daunting, limiting designers’ efforts. We propose the Culturally-Informed Design Framework as a guide for designers of consumer health IT. Designers may use this framework to conceptualize four dimensions of a consumer health IT – technology platform, functionality, content, user interface—in which design choices should be informed by a deep understanding of the users’ culture.

As Web technology evolves, information organizations strive to benefit from the latest developments. Many academic and government organizations develop applications in Second Life, an online virtual world that allows users to interact with one another and the virtual environment via graphical personas, to support education and information outreach. This is accompanied by growing interest in evaluation methods for Second Life applications. While no special methods have been developed for Second Life or other virtual worlds, the field of Web evaluation is mature and likely to offer metrics, methods, and tools that might be adaptable to Second Life. The goal of this project was to analyze how existing Web measures of Internet performance, Web usage, usability, and user feedback could be adapted, expanded, and modified for Second Life. The project employed two facilitated expert panel discussions, followed by an empirical pilot-test of the experts' suggestions. The findings suggest that prevailing methods and metrics of four key evaluation dimensions can be adapted to Second Life. Specific recommendations are made for their adaptation. Challenges involve lack of universal Second Life design practices and user expectations, influence of other avatars on user experience, complexity of 3-D topography, high technical requirements for data collection, and the proprietary nature of Linden Lab's data.

In 2001, Biblioteca Las Americas (BLA), a school library in the South Texas Independent School District (STISD), began the Vital Information for a Virtual Age (iVIVA!) Peer Tutor Project, in which high school students promote MedlinePlus in their high schools and communities. MedlinePlus and its companion site, MedlinePlus en espanol, are consumer health information websites maintained by the National Library of Medicine (NLM) [1]. Approximately 80% of the STISD student population is Hispanic [2]. The district serves counties that are medically underserved [3] and have poverty rates that are double the national average [4].{ The iVIVA! Peer Tutor Project started as a partnership between the University of Texas Health Science Center at San Antonio (UTHSCSA) Libraries and BLA, when the former received funding from NLM to conduct health information outreach projects in the Texas Lower Rio Grande Valley [5]. Four high school students from the STISD’s South Texas High School for Health Professions (Med High), with training and guidance from a UTHSCSA medical librarian, taught their peers and faculty members about MedlinePlus in the first year of the project. The project is now administered directly by BLA and has been extended to all four high schools in the district, which consists of two health careers high schools; a science and technology high school; and a business, education, and technology high school. NLM has supported the project for almost ten years, through funding either to STISD or UTHSCSA. The school district’s administrators have strongly endorsed the project, and the BLA librarians have built a network of librarians and faculty who actively participate as mentors and advisors. Details of the project and its evaluation were published in 2005 [6]. Peer tutors promote MedlinePlus to their peers and teachers through class demonstrations, student orientations, school health fairs, and extracurricular activities such as the Health Occupations Students of America meetings. Over the years, they have extended their reach to parents and the community through school open houses, local health fairs, fun runs, and motorcycle rallies. A popular outreach site for peer tutors is the Harlingen Boys and Girls Club, where they promote health information to children and teens. The 2005 article published assessment results from a questionnaire completed by 500 Med High students in 2003, at the end of the project’s first year. Results showed that 66% of respondents reported using MedlinePlus and 52% had referred the resource to others [6]. More recent evaluation data, described in iVIVA! subcontract reports, showed high awareness of MedlinePlus among STISD students. For example, * This project was funded by the National Library of Medicine under contract (N01-LM-6-3505) with the Houston Academy of Medicine-Texas Medical Center Library through a subcontract with the South Texas Independent School District (STISD), which was solely responsible for project planning, implementation, and evaluation.