SIGIR 2003 Workshop Report: Implicit Measures of User Interests and Preferences

The goal of the workshop on Implicit Measures was to explore how various implicit measures of user interest can be used in information retrieval and filtering applications.  In most information retrieval or filtering applications, it is difficult to get explicit feedback from users about the relevance of the results, the appropriateness of the presentation, and more generally about the quality of their experience. Yet explicit judgments are assumed by researchers for many activities like the tuning and selection of ranking algorithms, information combination, user modeling, information presentation, etc. This workshop explored how implicit measures of user interest (such as dwell time, click through, and user activities like annotation, printing, and purchasing) can be used to develop predictive models for a variety of purposes.

In the context of information retrieval such models could be used to: improve ranking and relevance assessment (e.g., the extent to which implicit measures can be used to evaluate the quality of systems, ranking algorithms and recommendations, or as input to relevance feedback algorithms); personalize search, filtering or presentation; personalize considering both individual and aggregate data; adaptive link generation or web site design; etc. An examination of theoretical issues such as modeling approaches (Bayesian techniques and other predictive models), gold standards for user behavior (e.g., relevance judgments, purchases), combining implicit and explicit preferences, and biases introduced by reliance on implicit measures were also encouraged.

Since this was the first workshop on the topic at SIGIR, we encouraged participation from people with different backgrounds and perspectives including theoretical modeling, experimental analysis, and applications development. Applications from information retrieval, collaborative filtering, e-commerce, user modeling, and human-computer interaction were encouraged. Participation from both academia and industry was encouraged. As an outcome of the workshop we hoped to identify key theoretical modeling issues, systematize engineering principles and best practices, and spark new research directions.

The workshop was held on August 1, 2003 in Toronto, Canada, immediately following the SIGIR 2003 conference. The workshop drew about 35 participants, representing a nice mix of academia and industry, and information retrieval and learning.  To encourage interaction among participants, we limited the number and length of the talks to 15 minutes. This was followed by a 5 minute discussion by one of the organizers and additional open discussion among participants.  This seemed to work well as a way to encourage interaction. We also had poster presentations available during the lunch break, which worked well, although only two posters were presented.