Developing Archival Metrics in College and University Archives and Special Collections
|
Contact Information |
About the Project
Beginnings and Goals of the Project
AX-SNet (Archival eXcellence in Information Seeking Studies Network) is an international collaboration of researchers working: (1) to improve access to primary sources; (2) to explore the ways users seek information in archives; and (3) to develop new ways to teach people about how to do effective archival research. Collaborators from the University of Toronto, the University of North Carolina, and the University of Michigan lead the AX-SNet research team. The current project focuses on the following problem:
Primary sources are an underpinning of scholarly research in the humanities and social sciences. However, the administration of institutions, agencies, and departments that provide access to archives and manuscripts varies from place to place and often operates in the absence of reliable use and user statistics. The development and adoption of standardized metrics to support the management of both analog and digital collections is a critical need in archives and manuscript collections.
Metrics are measurements taken over time that monitor, assess, and communicate vital information about the results of a program or activity. Without reliable use data the managers of archives and special collections departments cannot make optimal management decisions. To help remedy this problem, in December 2003, the Andrew W. Mellon Foundation awarded an initial $73,000 to the University of Michigan to explore the feasibility of establishing standard metrics for use and user services. For archives, the ability to accurately compare and contrast program statistics among sister institutions would help identify "best practices," set service standards and allow managers to plan for improvements to those functions. Therefore, the ultimate result of the entire project will be the creation of information for decision-making and the development of better user services and access systems for the participants.
The primary goals of the feasibility project (Phase One) were to demonstrate that archivists and manuscript curators in a variety of environments could reach consensus on the need for core metrics; and would agree to be partners in this endeavor. These goals were met through three activities: a comprehensive literature review, an analysis of current practice, and a meeting of diverse archivists from the United States and Canada.
Leading Archivists Meet in June, 2004 to reach consensus on metrics agenda
These preliminary activities would culminate in a three-day meeting of academics, archival practitioners, and researchers in the field of evaluation. The meeting was entitled "Users, Metrics, Archives" and took place at the School of Information and Library Science at the University of North Carolina at Chapel Hill June 3-6, 2004. Twenty-five representatives from a variety of different kinds of archives participated as partners. Religious, governmental, corporate, museum, and university archivists attended, as well as representatives from the Mellon Foundation, the Digital Library of California, and the Research Libraries Group consortium.
Presentations were given at this meeting by Gary Marchionini, Barbara Wildemuth, and Paul Conway. Wendy Duff and Sarah Carson also presented a literature review, and Elizabeth Yakel and Elizabeth Goldman presented an analysis of current practice regarding assessment in archives. Helen Tibbo and Brian Dietz's report on the meeting can be found here.
The two basic questions to be answered at the conference were: "What should we measure?" and "How should we conceptualize measurement, both locally and in a broader sense?" Investigators sought to identify the critical subjects of measurement (for example, archival reference desks, descriptive tools, and exhibits); the specific elements that archivists should measure; the best methods of collecting data; and the participant's perceptions of the most important barriers to access to archived records. Underlying this explicit agenda were fundamental questions of whether archivists and curators in a variety of environments could reach consensus on the need for core metrics for collection usage and user service, and would agree to be partners in the work.
Ultimately, the participants agreed that a good starting place for standardization would be registration data. They were interested in demographic and institutional affiliation information from researchers, as well as what materials they used, the path by which they arrived at the archive, and the purpose and proposed outcome of their research. Several participants also acknowledged the changing world of access to archives through online resources and noted a need to understand these users in order to readjust allocations of resources. Participants were also curious about the usefulness of archival description tools; the effectiveness of outreach and instruction (including online instruction); how users come to trust, or view as authentic, archival holdings; and finding the products that result from archival research. Reasons given for wanting to know this information included self-assessment of collections and services and informing administrators involved in resource allocation.
Participants also agreed that to measure the impact of archives on users, one must understand the goals of each particular type of archives. Thus, for college and university archives, and impact measure must be developed in relation to three main missions of these institutions-teaching, learning and research. Therefore, evaluative instruments needed to be geared toward measures that would support these activities. Business archivists stated the importance of focusing on the return on the investment.
The most significant outcome of the meeting was the overwhelming support for the metrics movement within the archival community. The field is ripe for metrics and recognizes a need for more robust and validated measures for both better internal administration as well as the ability to benchmark across peer institutions. Archivists and special collections curators expressed a great deal of frustration over current data collection and the inability to associate these data with real outcomes and impact measures. While no one wanted to collect more data, participants were keenly interested in learning how to collect data to better achieve their programmatic objectives and demonstrate value to administrators in their organizations.
Post-Meeting Activities
In August, 2004, AX-SNet researchers attended the Society of American Archivists (SAA) meeting and met with participants' core constituencies to spread the word to a broader network. They spoke to several institutional sections: Business, College and University, and Religious Archives, and two functional sections: Description and Reference. They also met with the Standards Board, the Task Force on Descriptive Standards, and the Committee on Education and Professional Development. All were very interested and urged them to proceed.
Wendy Duff attended the annual meeting of the Society of Archivists in the United Kingdom and discussed the metrics project. Archivists at this meeting were interested in the project and encouraged them to collaborate with researchers in the UK who have begun to conduct user-based evaluation. Elizabeth Yakel attended the International Council on Archives in Vienna, August 23-28 and presented the results of the June meeting in Chapel Hill to a group interested in standardizing archival metrics for all functions. Helen Tibbo discussed AX-SNet and the "Developing Standardized Metrics" project at the DELOS Network of Excellence Digital Library Summer School held in Pisa, Italy, September 6-10, 2004 where the year's focus was "User-Centered Design of Digital Libraries." In October, Helen Tibbo and Wendy Duff also attended the final Electronic Records Preservation Access Network (ERPANET) meeting in Switzerland and presented the findings of their work and explored international alliances for metrics development.
Elizabeth Yakel was invited to attend an Association of Research Libraries (ARL) Special Collections Task Force meeting in Washington D.C. during the month of October. The invitation to this meeting was significant, and ARL's interest may have bearing on the issue of sustainability of the data and tools resulting from this project.
At the annual conference of the Society of American Archivists, Helen Tibbo presented project findings at a special session entitled "Developing Standardized Metrics for Assessing Use and User Services for Primary Sources" in August, 2005 in New Orleans.
Phase Two: Towards Understanding the Impact of College and University Archives and Special Collections on Scholarship, Teaching and Learning
After successfully identifying criteria for the measurement of use and user services during Phase One, the AX-SNet investigators have received an additional $329,000 grant from the Mellon Foundation to proceed with the actual creation of questionnaires, forms, transaction logs, and statistically validating these questions and measures. One of the most important findings of the group of archivists that met in North Carolina in 2004, was that measures of impact and outcomes must be tailored to the particular goals of the institution.
For Phase Two, college and university archives and special collections have been selected as the target population because they form the largest segment of the archival population. This will enable the team to examine the interplay between the provision of primary sources and related support services such as descriptive tools, user education, and reference interations, and the research, teaching and learning missions of institutions of higher education.
Phase Two of the "Developing Standardized Metrics" project will accomplish the following five objectives:
- Develop a core minimal standardized data set for college and university archives and special collections and the underlying metrics for those data. These data will serve as the basis of a yearly statistical report sent to a data repository to be used by others as benchmarking data.
- Develop a select set of validatated instruments for college and university archives and special collections they can use to evaluate the effectiveness of their access and user education services in support of research, teaching and learning.
- Develop broad and complex measures to ascertain impact of archival services in higher education settings.
- Investigate options for sustainability of the tools and the data developed, collected, and analyzed during this project.
- Create a model for a data repostory that will maintain the evaluation data so institutions can benchmark against peers and understand the landscape of access and service across the profession.
With information gathered from interviews with archivists, students, and professors, the team will draft initial questions to help evaluate archives' role in teaching and learning. After consulting with several of our institutional partners at a meeting in Ann Arbor in October, 2005, we will begin alpha-testing in early 2006. The team will focus on developing validated questions that reliably probe the various dimensions of teaching, learning and scholarship. Beginning in June, 2006 the team will proceed to beta testing and creation of documentation and instructional packets.
Simultanously, a group led by Professor Helen Tibbo will focus on developing a business plan for the instruments--a roadmap to sustainability. This team will focus on determining how to support long-term warehousing of the data and maintainance of the instruments. It will identify and design guidelines, potential platforms, and an architecture for a data repository and investigate various models for the sustainability of the project and suggest the most feasible.
Phase Two Takes Off: Gathering Data and Meeting with University Archivists
In June, 2005, work on Phase Two began in earnest when project manager, Aprille Cooke McKay joined the research team. Other team members during the 2005-06 academic year include Morgan Daniels (UM), Angela McClendon (UNC), Erin Passehl (UM) Greg Peters (UM), Christie Peterson (UM), Seth Shaw (UM), Beth St. Jean (UM), and James Sweeney (UM).
One of our first tasks was to explore different users' definitions of quality archival services. In July of 2005, therefore, the team created protocols for semi-structured interviews and focus groups of archivists, professors and students. Our questions probed how archives support higher educations' goals of teaching, learning and research. In all, we interviewed fourteen archivists, twelve professors, eight graduate students and eight undergraduates. In addition, we conducted two focus groups of archivists in which we asked them about what types of user feedback would be most useful for good management of their institutions. These interviews have been transcribed, and will be coded during the summer of 2006. The data these interviews present is quite good, and will be one of the most interesting by-products of the project.
In October, 2005, representatives from nine archival partner institutions met in Ann Arbor to advise the investigators about their needs and desires for assessment tools for university archives and special collections. Participants broke into smaller working groups, each tasked with developing tools to help archivists and curators measure particular aspects of their services. The first group, including Wendy Duff, Joan Cherry, Mike Moir, Karen Jania and Diane Kaplan, studied how to measure the impact that archival services make on users. The second group, including Helen Tibbo, Nancy McGovern, Robin Chandler and Aprille McKay, studied web analytics. The third group, including Beth Yakel, Bill Wallach, Bill Lefevre, Laura Clark Brown, Tim Pyatt, Mike Smith and Janet Gertz analyzed the possibility of promulgating a registration card which could be adopted by many institutions, thereby insuring that many institutions used the same categories for analysis.
Since October, these groups have continued to pursue their goals. The Impact Working group has planned a study at Yale University involving four questionnaires, administered at the beginning and the end of the school term. Two short questionnaires assessing professors' and students' use of primary sources and online educational tools will be administered at the beginning of the fall 2006 term. The participants attended an archives orientation session designed to acquaint them with archival services and the use of primary sources. Another two questionnaires (students and faculty) will be administered to participants in the end of the term, assessing the value of the orientation session and the use of primary sources for teaching and learning.
The Web Analytics working group determined that comparison of web-use statistics between institutions is not very useful, because small details in the way that use is measured create large differences in the meaning of the numbers. Rather, self-benchmarking should be the rule for measuring use of archival collections presented on the internet. Therefore, the members of this group determined that they should create a guide to help archivists understand how transaction logs and other forms of web use measurement interact with EAD finding aids, ILS databases, authentication procedures, and digital collections. Members of the working group, augmented by James Sweeney (UM), Eva Reffell (LBL), and Felicia Poe (CDL) participated in a two-day meeting in February, 2006 in Oakland, California, hosted by Robin Chandler at the California Digital Library. This paper is currently being drafted. New members of the working group include Angela McClendon (UNC) and Ellie Buckley (Cornell).
The Registration Card Working Group has collected sample registration cards from 33 colleges and universities to try to determine commonalities and differences. In addition they have gathered standard definitions of information use and users from many different standards bodies to study how different use metrics might describe archival use and users.
Project Sustainability Meetings
This project cannot permanently survive on short-term research grants. The archival community must find a way to support a culture of assessment, in which evaluation of our services and usability of our tools is built into our budgets and supported by our managers. In order to determine the range of possible funding and partnership strategies, we convened a meeting of the Archival Metrics Sustainability Group in Chapel Hill, North Carolina in February, 2006. Participants helped outline the fundamental business plan and suggested networking contacts and funding formulas which deserve further exploration, and also indicated which paths were unlikely to succeed. The group reconvened during the Society of American Archivists meeting in Washington, DC in August, 2006, to discuss pieces of the draft business plan.
Project team members presented several papers during the Summer and Fall of 2006. At the Association of Canadian Archivists meeting in Newfoundland, June 28-July 1, in a session entitled "Understanding New Audiences," all of the project's investigators gathered to discuss the importance of user-based evaluation and usability research. Wendy Duff presented an overview of user-based evaluation studies and usability research; Helen Tibbo provided a summary of the Phase One of this project which studied the feasibility of standard user-based evaluation tools. Elizabeth Yakel discussed the development of standards for collecting registration data. Finally, Joan Cherry presented the findings of an iterative usability study of a Portal for Text Analysis Researchers, a project which is intellectually aligned with the current project.
The joint conference of the Society of American Archivists (SAA), the Council of State Archivists (CoSA), and the National Association of Government Archives and Records Administrators (NAGARA) took place in Washington, DC, July 30-August 6, 2006. Aprille McKay presented a paper which analyzed the content of online archival research tutorials and Beth Yakel presented "Archival Metrics in College and University Archives and Special Collections."
At the International Conference on Archives Section on University and Research Archives (ICA-SUV) in Reykjavik, Iceland, 13-20 September 2006, Aprille McKay and Helen Tibbo presented session entitled "Building a Culture of Assessment in Academic Archives: The Archival Metrics Project and Beyond."
Tool Testing Begins
Small pilot user surveys for professors and students were conducted at the Bentley Historical Library at Michigan and the Special Collections Department at UNC in June of 2006. In August of 2006, a pilot study regarding general research was conducted at the Bentley, and researchers were recruited to critique the questionnaires.
At the same time, a comment period has begun on the data elements to be gathered for the benchmarking database. We would welcome your comments. The draft instrument can be found here
Wendy Duff and Helen Tibbo coordinated a session at the 2007 Annual Meeting of the Association of Canadian Archivists (Kingston, ON, June 21-23) entitled "Tools to help others see how our users see us." The session focused on gathering rigorous user feedback to help archivists understand how users view repositories and their services. Tibbo's paper discussed web server logs and web analytics, and the types of information this analysis provides. Geoff Pick discussed the Survey of Visitors to U.K. Archives - its origins, its findings and its lessons. Duff presented data gathered from testing the project's survey tools.
At the 2007 annual meeting of the Society of American Archivists, (Chicago, August 27-September 2) project researchers coordinated a session entitled "Beyond Evaluation: Measuring the Impact of Archives," chaired by Anne Van Camp. The session focused on impact assessment and asked: “What difference does the archives make in the lives of researchers and society?” After an introduction to impact assessment, Wendy Duff discussed her impact study in an archives, Helen Tibbo presented the collaboration's research on using web analytics, and Beth Yakel presented a report on the project's qualitative analisis of interviews with archivists, faculty who teach using primary sources, and researchers who discuss how archives have had an impact on their lives.
