Toolkits | UserSim

Showing 15 of 15 toolkits

Search Engines

Toolkit	Paper	Summary
cwl_eval SIGIR 2019	Azzopardi et al. View Paper	Implements various metrics within the C/W/L framework regarding the predicted user interactions with the ranked list of search results.
SimIIR SIGIR 2016	Maxwell and Azzopardi View Paper	Framework for simulating users based on the Complex Searcher Model, including the actions of querying, snippet examination, relevance assessment, and stopping.
SimIIR 2.0 CIKM 2022	Zerhoudi et al. View Paper	Extends the original SimIIR framework by adding a dynamic query generator, introducing user types with distinct search behaviors, and using Markov models for stopping and query generation decisions.
SimIIR 3.0 SIGIR-AP 2024	Azzopardi et al. View Paper	Extends SimIIR 2.0 with a conversational search workflow, Markovian users, cognitive states, and LLM-based components.

Recommendation

Toolkit	Paper	Summary
RecSim arXiv 2019	Ie et al. View Paper	Platform to create configurable simulation environments for interactive recommendation problems, including a user model, a document model, and a user choice model.
RecSim NG arXiv 2021	Mladenov et al. View Paper	Platform to create and learn configurable simulation environments for collaborative interactive recommendation problems, including composable dynamic Bayesian networks.
Sim4Rec ECIR 2025	Volodkevich et al. View Paper	Framework implementing an interactive learning pipeline including components for the generation of synthetic data and user responses; designed for large-scale data.
KuaiSim NeurIPS 2023	Zhao et al. View Paper	Comprehensive RL environment with a user model producing multi-level feedback (clicks, likes, follows) across sessions; built on the large-scale, multi-behaviour KuaiRand dataset.
Lusifer arXiv 2024	Ebrat et al. View Paper	LLM-driven simulation environment that dynamically updates user preferences to generate realistic feedback, capturing concept drift and cold-start scenarios for RL-based recommender training.

Conversational

Toolkit	Paper	Summary
ConvLab-3 EMNLP 2023	Zhu et al. View Paper	A dialog system platform that implements transformer-based (TUS and GenTUS) and LLM-based user simulators for task-oriented dialogue.
CoSearcher IRJ 2022	Salle et al. View Paper	Stochastic user simulator for conversational search refinement, modeling cooperativeness and patience of users.
GenIRSim CLEF 2024	Kiesel et al. View Paper	Implements an LLM-based simulator for the evaluation of generative information retrieval systems.
iEvaLM EMNLP 2023	Wang et al. View Paper	Provides the implementations of a configurable LLM-based user simulator and multiple conversational recommender systems for evaluation purposes.
PyDial ACL 2017	Ultes et al. View Paper	Multi-domain statistical spoken dialogue system toolkit with a user simulation component that operates on the semantic level for training and evaluating RL-based algorithms.
UserSimCRS WSDM 2023	Afzali et al. View Paper	Provides the implementations of agenda-based and LLM-based user simulators, and metrics for the evaluation of conversational recommendation systems.

This page is maintained by Nolwenn Bernard and Krisztian Balog.

We welcome suggestions via email.