Toolkits

This page is maintained by Nolwenn Bernard and Krisztian Balog. We welcome suggestions via email.

Search Engines

Toolkit	Paper	Summary
cwl_eval	Azzopardi et al., SIGIR 2019	Implements various metrics within the C/W/L framework regarding the predicted user interactions with the ranked list of search results.
SimIIR	Maxwell and Azzopardi, SIGIR 2016	Framework for simulating users based on the Complex Searcher Model, including the actions of querying, snippet examination, relevance assessment, and stopping.
SimIIR 2.0	Zerhoudi et al., CIKM 2022	Extends the original SimIIR framework by adding a dynamic query generator, introducing user types with distinct search behaviors, and using Markov models for stopping and query generation decisions based on these user types.
SimIIR 3.0	Azzopardi et al., SIGIR-AP 2024	Extends SimIIR 2.0 with a conversational search workflow, Markovian users, cognitive states, and LLM-based components.

Recommendation

Toolkit	Paper	Summary
RecSim	Ie et al., arXiv 2019	Platform to create configurable simulation environments for interactive recommendation problems, including a user model, a document model, and a user choice model.
RecSim NG	Mladenov et al., arXiv 2021	Platform to create and learn configurable simulation environments for collaborative interactive recommendation problems, including composable dynamic Bayesian networks.
Sim4Rec	Volodkevich et al., ECIR 2025	Framework implementing an interactive learning pipeline including components for the generation of synthetic data and user responses; designed for large-scale data.
KuaiSim	Zhao et al., NeurIPS 2023	Comprehensive RL environment with a user model producing multi-level feedback (clicks, likes, follows) across sessions; built on the large-scale, multi-behaviour KuaiRand dataset.
Lusifer	Ebrat et al., arXiv 2024	LLM-driven simulation environment that dynamically updates user preferences to generate realistic feedback, capturing concept drift and cold-start scenarios for RL-based recommender training.

Conversational Information Access

Conversational Information Access might be seen as a specific type of Task-Oriented Dialogue (TOD), where user goals include search, recommendation and exploratory information gathering.

Toolkit	Paper	Summary
ConvLab-3	Zhu et al., EMNLP 2023	A dialog system platform that implements transformer-based (TUS and GenTUS) and LLM-based user simulators for task-oriented dialogue.
CoSearcher	Salle et al., IRJ 2022	Stochastic user simulator for conversational search refinement, modeling cooperativeness and patience of users.
GenIRSim	Kiesel et al., CLEF 2024	Implements an LLM-based simulator for the evaluation of generative information retrieval systems.
iEvaLM	Wang et al., EMNLP 2023	Provides the implementations of a configurable LLM-based user simulator and multiple conversational recommender systems for evaluation purposes.
PyDial	Ultes et al., ACL 2017	Multi-domain statistical spoken dialogue system toolkit with a user simulation component that operates on the semantic level for training and evaluating reinforcement learning-based algorithms.
UserSimCRS	Afzali et al., WSDM 2023	Provides the implementations of agenda-based and LLM-based user simulators, and metrics for the evaluation of conversational recommendation systems.