SIGIR 2004 Workshop
New Directions For IR Evaluation: Online Conversations
Sheffield, UK, July 29 2004

Short Introduction

you may also look at a more extended introduction

In some respects, an online conversation is not very different from the conversations people have been having for thousands of years. Topics are introduced, ideas are shared, and sometimes enlightenment is forthcoming. Online conversations lack some of the contextualizing factors (e.g., location) and non-verbal clues that are present in face-to-face conversations (thus encouraging participants to make their intent explicit), they are normally rendered at least partially as electronic text (making them easier to manipulate) and they are sometimes recorded and archived for future use (potentially making them available for later use).

Examples of "online conversations" include personal electronic mail, mailing lists, Instant Messaging (IM), Short Message Service (SMS) notes, chat rooms, Usenet newsgroups, threaded Web-based discussion lists, and massive multi-player role playing games. Online conversational content poses a number of interesting challenges to systems designed to support access, including: (1) exploitation of discourse and dialog structure (e.g., to support thread-based access), (2) leveraging expressions of attitude, point of view, and affective reaction, (3) the prevalence of informal language and emergent sub-languages, and (4) the importance of establishing adequate context to interpret retrieved content (e.g., for IM and SMS).

Goal

Our goal for this workshop is to identify opportunities for important new research on online conversations and how this research can be facilitated through the creation of standard community-wide resources (e.g., development of standard test collections). To do this, we plan to bring together researchers from several communities with expertise in aspects of this question (e.g., information retrieval, online communities, computer-supported cooperative work, recommender system, computational linguistics, and text data mining) to explore the present state of the art and to synthesize multiple perspectives on evaluation of systems for working with online conversations. We will count the workshop as successful if it results in a concrete understanding of what types of evaluation resources are needed and how those resources can practically be created.