Community Webs Evaluation Framework

The Project

Educopia undertook a formal evaluation of the Community Webs project (2019-2020) in order to assist the Community Webs team to develop strategic partnership roadmaps, business plans, and community models that build directly on the project’s strengths. Educopia evaluators designed and implemented an evaluation framework that included surveys of participating libraries and librarians, analysis of the Community History Web Archives collections they built, and focus groups and topical calls with the core cohort.

The Outcome

Building on the work of the CRL Board, CRL’s Collections and Services Policy Committee (CSPC), ICCBased on our iterative findings, the Educopia evaluators guided the Community Webs team through a series of sustainability conversations to help them determine how best to scale the program’s activities and to explore how it might connect with digital library services and training for librarians and museums across the United States.

Explore our work in detail by accessing the Community Webs Phase 1 Evaluation Report, detailing work undertaken by Educopia on behalf of the Community Webs program team.

Project Background

The Community Webs project (IMLS RE-85-17-0060-17) was funded in 2017 to create a Community History Web Archives.

The project provided training, continuing education, cohort development, technology services, and infrastructure to librarians from a diverse set of public libraries, empowering them to build collections of historically valuable, web-published materials. 

Participating libraries learned to archive the web-based histories of their local communities, with an emphasis on marginalized or traditionally underrepresented groups. During the project period, the resulting collection preserved over 245 new collections and 36 terabytes of web based community heritage data from 27 public libraries nationwide. The project has continued to expand, with 13 more libraries added in 2020. As of 2024, Community Webs supports over 150 cultural heritage organizations preserving 75 terabytes of web-based community heritage data. More information can be found on their website. 

What We Learned

Educopia’s evaluation of the Community Webs project found that the project was extremely successful and well-designed, but that it lacked the resources to accommodate all of those who were interested in participating. Thus we sought to craft a written evaluation report that the client could use to leverage additional support for sustainable expansion.

We embarked on a series of interventions, including an Evaluation Framework and a set of facilitated activities that strengthened the program’s administrative and community scaffolding over the course of the project period

The Evaluation Framework was designed and implemented to fulfill several goals specified by the Community Webs program in its 2019 supplemental award, including documenting program deliverables, understand what skills/competencies cohort members developed, establishing and documenting the benefits of involvement for a variety of stakeholders, helping determine how best to scale the program’s activities, and exploring how Community Webs’ success can be mapped to other initiatives to further the impact on digital library services and training for librarians and museums across the United States. This evaluation, conducted by Evaluators Katherine Skinner, Jessica Meyerson, and Terra Graziani of Educopia, focused on five core objectives: 

1. Evaluation of Program Deliverables
2. Evaluation of Participant Development
3. Evaluation and Continued Work on Program Expansion
4. Evaluation of Community and Strategic Partnerships
5. Evaluation of, and Continued Work on, Sustainability and Business Planning.

The methods used in the evaluation included:

• Surveys: Evaluators designed and administered a survey for cohort members that focused on analyzing the educational, collection, and progressional outputs of the original two-year Community Webs project. The survey asked questions related to, but not limited to: cohort members’ experience with training modules, curriculum topics, technical support needed, collecting practices, buy-in from their institutions, communications and public engagement, and how Community Webs might sustain itself or expand going forward.
• Focus Groups: The focus groups were designed to inform the development of the Community Webs program expansion framework, including how best to structure its governance, community engagement strategies, and financial model. They also sought to understand what forms of support cohort members needed, including training and guidance for building accessible digital community archives. Each call focused on three sets of questions on local community, strategic direction, and growth models. This yielded nuanced feedback on the program and its relationship to other work that each participant was engaged in.
• Notes from Cohort Topical Calls: Throughout the evaluation period, Educopia evaluators worked with the Internet Archive team members to design, conduct, and document a series of topical open calls with cohort members. These cohort calls were intended to serve a dual purpose: first, providing cohort members with a designated space for peer support, and second, providing the project team with feedback.
• Notes from Project Team Calls: We conducted team meetings between the Internet Archive project team and the Educopia evaluators on a biweekly basis for the length of the subaward period. These calls were a space to reflect on cohort development, move forward program deliverables, build capacity for the expansion of the program, and share findings iteratively from the survey, focus groups, literature and curriculum review, and cohort calls.
• Documentation Review: Evaluators reviewed a wide range of project documentation throughout the project period to inform our work. This included cohort white papers, training materials, notes, and conference presentations, which provided context on cohort experiences and guided our recommendations. We also examined collection development policies from five participating libraries. These materials helped us identify challenges and opportunities as we developed the survey and focus group scripts. They also provided essential context for designing the Sustainability Framework and supporting the Community Webs team in planning for program expansion and scaling.

Our Takeaways

Evaluators conducted three major forms of data analysis for this evaluation: a survey analysis, qualitative coding of notes from focus groups and cohort calls, synthesis of discussions with project team members, and a review of project deliverables to assess accomplishments versus aims. This comprehensive approach enabled us to achieve the following objectives:

1. Evaluation of Program Deliverables: Educopia’s evaluation included a review of the cohort and its activities, course curriculum, training materials, educational tools, cohort publications and presentations, the Community History Web Archives collection, and survey and focus group data.
   
   Based on our reflections (detailed in the report), we recommended establishing a concrete plan to modify and expand the educational, training, and professional development work of the project; updating training materials; and creating a community-led process for regular evaluation and updating of these resources.
   
2. Evaluation of Participant Development: Evaluators assessed participant development using three main data sources: analysis of the collections built by participants, self-reports from surveys and focus groups, and demonstrated knowledge through presentations, white papers, and publications produced during and after the grant period.
   
   Recommendations included organizing peer check-ins and office hours for communication, offering tech support for ArchiveIt, creating resources for cohort leadership and community ownership, expanding participation in the community of practice, and developing materials to support community leadership and professional development.
   
3. Evaluation and Continued Work on Program Expansion: Evaluators focused on survey results related to growth models and strategic directions. We also analyzed the structure implemented by the project team to onboard, train, and support cohort members.
   
   We recommended that the Internet Archive team hire a full-time community coordinator, create workflow resources to involve library staff in web archiving, engage directly with Community Webs library administrators, integrate network building into the expansion plan, develop outreach materials for public libraries, and establish streamlined onboarding for new libraries.
   
4. Evaluation of Community and Strategic Partnerships: Evaluators worked with the project team to identify and pursue strategic partnership opportunities. We also began exploring what community-based governance (formal or informal) might look like and how community members could engage in scaling up the Community Web program nationally.
   
   Based on our findings, we recommended the team consider: launching a public campaign with libraries, creating outreach templates for cohort members, highlighting those pioneering best practices, developing actionable principles for web archiving, organizing speakers on ethical collecting, engaging underrepresented groups, documenting strategic partnerships, and scheduling outreach milestones for each partnership.
   
5. Sustainability and Business Planning: Evaluators analyzed survey results, focus groups, topical calls, and findings from workshopping sessions with the project team. We began identifying funding opportunities within and beyond the Internet Archive to operationalize a scaled-up version of the Community Webs project and drafted initial business and cost scenarios.
   
   We recommended that the project team use the evaluation findings to develop a roadmap for establishing Community Webs’ governance model, clarify its mission and vision, unify existing platforms, and create a cohesive visual identity. Additionally, we suggested developing onboarding documentation and training pathways for new cohort members, implement a communications plan, engage current members in mentorship and leadership training, and ensure consistent messaging across all public materials.