User Simulation for Evaluating Information Access Systems

Authors: Krisztian Balog and ChengXiang Zhai Status: under review for Foundations and Trends in Information Retrieval Draft: https://arxiv.org/abs/2306.08550

Information access systems, such as search engines, recommender systems, and conversational assistants, have become integral to our daily lives as they help us satisfy our information needs. However, evaluating the effectiveness of these systems presents a long-standing and complex scientific challenge. This challenge is rooted in the difficulty of assessing a system’s overall effectiveness in assisting users to complete tasks through interactive support, and further exacerbated by the substantial variation in user behaviour and preferences. To address this challenge, user simulation emerges as a promising solution. This book focuses on providing a thorough understanding of user simulation techniques designed specifically for evaluation purposes. We begin with a background of information access system evaluation and explore the diverse applications of user simulation. Subsequently, we systematically review the major research progress in user simulation, covering both general frameworks for designing user simulators, utilizing user simulation for evaluation, and specific models and algorithms for simulating user interactions with search engines, recommender systems, and conversational assistants. Realizing that user simulation is an interdisciplinary research topic, whenever possible, we attempt to establish connections with related fields, including machine learning, dialogue systems, user modeling, and economics. We end the book with a detailed discussion of important future research directions, many of which extend beyond the evaluation of information access systems and are expected to have broader impact on how to evaluate interactive intelligent systems in general.

Tutorials

The material has been/will be presented in the form of tutorials at various conferences.

• 31st ACM International Conference on Information and Knowledge Management (CIKM ‘23), Birmingham, UK, October 2023.
  • Initial version, with a broad coverage of the evaluation of information access systems using simulation in general.
• 1st International ACM SIGIR Conference on Information Retrieval in the Asia Pacific (SIGIR-AP ‘23), Beijing, China, November 2023
  • The same initial version of the tutorial, mainly for the Asian audience.
• 38th Annual AAAI Conference on Artificial Intelligence (AAAI ‘24), Vancouver, Canada, February 2024
  • A broader perspective of user simulation for evaluating an interactive AI system, focusing more on simulation algorithms and techniques that are well connected with various sub-fields of AI.
• 2024 ACM Web Conference, Singapore, May 2024
  • This edition emphasizes more on applications of user simulation for evaluating Web information access systems.