The last few years has seen an explosion of research and development of digital libraries and digital library technologies. These systems are complex, heterogeneous, distributed systems, with a diverse array of human users. Currently, there are few ways to evaluate the effectiveness of research systems, compare different approaches, or to measure progress towards long-term goals.
The national D-lib Program has formed a working group on metrics for digital library systems. The goal of this activity is to establish appropriate metrics and standard tests to measure digital library systems. The initial emphasis is on information discovery by humans and retrieval in a heterogeneous world, with the objectives of measuring and documenting the impact of particular system concepts or features, in specific settings, for specific user communities with specific purposes. Because of the importance of the human in the loop, we expect to draw metrics from a broad set of relevant fields, including but not limited to those as diverse as psychology, engineering, and human communications.
This workshop will be a practical session for those who are developing measurements for digital libraries or component technologies.
Position papers are solicited on the topics of
- Development activities for metrics and test suites for digital libraries and component technologies;
- Studies of metrics for digital libraries and component technologies;
- Approaches that support the shared development, maintenance, and use of metrics and standard test suites.
In all cases, the focus must be on current and planned developments to create standard measurements for digital libraries and the papers must emphasize the problems and solutions that will benefit from workshop discussion.
Individuals who are known to have active projects in this area are invited to participate. General announcements through appropriate email distribution will be made. The organizers will select full participants on the basis of the fit with the workshop topic and the potential contribution to progress in this area.
The workshop will be a full day. Roughly half the time will be available for presentations by the full participants and the other half will be discussion. Full participation will be limited to between 20 and 30; observers to the workshop will be welcome to the limit of the room and can participate in the discussions at the discretion of the full participants.
Prior to the workshop, position papers from full participants will be made available at this web site. Discussion leaders among the participants will be assigned to read the papers and develop cross-cutting discussion points.