Toolkits

This page is maintained by Nolwenn Bernard and Krisztian Balog. We welcome suggestions via email.

Search Engines

Toolkit	Paper	Summary
cwl_eval	Azzopardi et al., SIGIR 2019	Implements various metrics within the C/W/L framework regarding the predicted user interactions with the ranked list of search results.
SimIIR	Maxwell and Azzopardi, SIGIR 2016	Framework for simulating users based on the Complex Searcher Model, including the actions of querying, snippet examination, relevance assessment, and stopping.
SimIIR 2.0	Zerhoudi et al., CIKM 2022	Extends the original SimIIR framework by adding a dynamic query generator, introducing user types with distinct search behaviors, and using Markov models for stopping and query generation decisions based on these user types.
SimIIR 3.0	Azzopardi et al., SIGIR-AP 2024	Extends SimIIR 2.0 with a conversational search workflow, Markovian users, cognitive states, and LLM-based components.

Recommendation

To be populated

Conversational Information Access

Conversational Information Access might be seen as a specific type of Task-Oriented Dialogue (TOD), where user goals include search, recommendation and exploratory information gathering.

Toolkit	Paper	Summary
ConvLab-3	Zhu et al., EMNLP 2023	A dialog system platform that implements transformer-based (TUS and GenTUS) and LLM-based user simulators for task-oriented dialogue.
CoSearcher	Salle et al., IRJ 2022	Stochastic user simulator for conversational search refinement, modeling cooperativeness and patience of users.
GenIRSim	Kiesel et al., CLEF 2024	Implements an LLM-based simulator for the evaluation of generative information retrieval systems.
iEvaLM	Wang et al., EMNLP 2023	Provides the implementations of a configurable LLM-based user simulator and multiple conversational recommender systems for evaluation purposes.
PyDial	Ultes et al., ACL 2017	Multi-domain statistical spoken dialogue system toolkit with a user simulation component that operates on the semantic level for training and evaluating reinforcement learning-based algorithms.
UserSimCRS	Afzali et al., WSDM 2023	Provides the implementations of agenda-based and LLM-based user simulators, and metrics for the evaluation of conversational recommendation systems.