The DLib Test Suite and Metrics Working Group: Harvesting the Experience from the Digital Library Initiative. Ronald L. Larsen, University of Maryland. April 2002.

DARPA's DLib Test Suite project was an early attempt at organizing a rigorous and well-supported testbed to enable comparative evaluation of digital library technologies and capabilities. The test suite included a diverse and heterogeneous set of resources deliberately selected to foster research in interoperability, and sponsored a Metrics Working Group (MWG) to develop quantitative performance measures. The test suite went largely underutilized by the reserarch community, and the MWG, while making significant progress, found the stated objective daunting. Clearly, much remains to be done in both the conception of effective test beds and the instrumentation to assss progress.

The DELOS Workshop on Evaluation of Digital Libraries provides an opportunity to make further progress in this important area, engaging an international community and building on the collective experience accumulated from a larger and more diverse set of digital library projects. In this paper, the DLib Test Suite is briefly reviewed and the progress of the MWG is described.

At the kickoff meeting of the DLib Working Group on Digital Library Metrics (WG), held January 7-8, 1998, at Stanford University, some discussion was held as to what did we mean by the term "digital library". We concluded that it would be valuable for our own deliberations to document a common understanding of the term, but agreed that such an understanding could only be for the purposes of our deliberations, i.e., we could not and would not aim for a general consensus. This document is intended for that purpose, and this draft is intended to start the discussions.

In preparation for the kickoff meeting of the Working Group, held 7-8 January 1998, members were encouraged to submit position papers. These are some of those received:

Replication of Results and the Need for Test Suites
(Position Paper by William Y. Arms)

Technological and Social Change and Implications for Digital Library Metapolicy
(Position paper by Bob Este)

Position Statement - Approaching D-Lib Metrics
(Position paper by Ed Fox)

White Paper
(Paul B. Kantor)

Digital Library Metrics: Uncertainty and Failure
(Initial Draft by Carl Lagoze)

Evaluation of Concept Space and Category Map Semantic Indexes
(Position paper by William M. Pottenger, Bruce R. Schatz, Duncan H. Lawrie, Robert E. McGrath