Evaluation methods for information retrieval systems come in three types: offline evaluation, using static data sets annotated for relevance by human judges; user studies, usually conducted in a labbased setting; and online evaluation, using implicit signals such as clicks from actual users. For the latter, preferences between rankers are typically inferred from implicit signals via interleaved comparison methods, which combine a pair of rankings and display the result to the user. We propose a new approach to online evaluation called multileaved comparisons that is useful in the prevalent case where designers are interested in the relative performance of more than two rankers. Rather than combining only a pair of rankings, multileaved comparisons combine an arbitrary number of rankings. The resulting user clicks then give feedback about how all these rankings compare to each other. We propose two specific multileaved comparison methods. The first, called team draft multileave, is an extension of team draft interleave. The second, called optimized multileave, is an extension of optimized interleave and is designed to handle cases where a large number of rankers must be multileaved. We present experimental results that demonstrate that both team draft multileave and optimized multileave can accurately determine all pairwise preferences among a set of rankers using far less data than the interleaving methods that they extend.

Evaluation methods for information retrieval systems come in three types: offline evaluation, using static data sets annotated for relevance by human judges; user studies, usually conducted in a lab-based setting; and online evaluation, using implicit signals such as clicks from actual users. For the latter, preferences between rankers are typically inferred from implicit signals via interleaved comparison methods, which combine a pair of rankings and display the result to the user. We propose a new approach to online evaluation called multileaved comparisons that is useful in the prevalent case where designers are interested in the relative performance of more than two rankers. Rather than combining only a pair of rankings, multileaved comparisons combine an arbitrary number of rankings. The resulting user clicks then give feedback about how all these rankings compare to each other. We propose two specific multileaved comparison methods. The first, called team draft multileave, is an extension of team draft interleave. The second, called optimized multileave, is an extension of optimized interleave and is designed to handle cases where a large number of rankers must be multileaved. We present experimental results that demonstrate that both team draft multileave and optimized multileave can accurately determine all pairwise preferences among a set of rankers using far less data than the interleaving methods that they extend.

We study the problem of optimizing an individual base ranker using clicks. Surprisingly, while there has been considerable attention for using clicks to optimize linear combinations of base rankers, the problem of optimizing an individual base ranker using clicks has been ignored. The problem is different from the problem of optimizing linear combinations of base rankers as the scoring function of a base ranker may be highly non-linear. For the sake of concreteness, we focus on the optimization of a specific base ranker, viz. BM25. We start by showing that significant improvements in performance can be obtained when optimizing the parameters of BM25 for individual datasets. We also show that it is possible to optimize these parameters from clicks, i.e., without the use of manually annotated data, reaching or even beating manually tuned parameters.

Modeling user behavior on a search engine result page is important for understanding the users and supporting simulation experiments. As result pages become more complex, click models evolve as well in order to capture additional aspects of user behavior in response to new forms of result presentation. We propose a method for evaluating the intuitiveness of vertical-aware click models, namely the ability of a click model to capture key aspects of aggregated result pages, such as vertical selection, item selection, result presentation and vertical diversity. This method allows us to isolate model components and therefore gives a multi-faceted view on a model's performance. We argue that our method can be used in conjunction with traditional click model evaluation metrics such as log-likelihood or perplexity. In order to demonstrate the power of our method in situations where result pages can contain more than one type of vertical(e.g., Image and News) we extend the previously studied Federated Click Model such that it models user clicks on such pages. Our evaluation method yields non-trivial yet interpretable conclusions about the intuitiveness of click models, highlighting their strengths and weaknesses.

We consider a framework in which a group of agents communicates by means of emails, with the possibility of replies, forwards and blind carbon copies (BCC). We study the epistemic consequences of such email exchanges by introducing an appropriate epistemic language and semantics. This allows us to find out what agents learn from the emails they receive and to determine when a group of agents acquires common knowledge of the fact that an email was sent. We also show that in our framework from the epistemic point of view the BCC feature of emails cannot be simulated using messages without BCC recipients.

In this paper we investigate Kripke models, used to model knowledge or belief in a static situation, and action models, used to model communicative actions that change this knowledge or belief. The appropriate notion for structural equivalence between modal structures such as Kripke models is bisimulation: Kripke models that are bisimilar are modally equivalent. We would like to find a structural relation that can play the same role for the action models that play a prominent role in information updating. Two action models are equivalent if they yield the same results when updating Kripke models. More precisely, two action models are equivalent if it holds for all Kripke models that the result of updating with one action model is bisimilar to the result of updating with the other action model. We propose a new notion of action emulation that characterizes the structural equivalence of the important class of canonical action models. Since every action model has an equivalent canonical action model, this gives a method to decide the equivalence of any pair of action models. We also give a partial result that holds for the class of all action models. Our results extend the work in van Eijck et al. (Synthese 185(1):131–151, 2012 ).

Enkele veelgebruikte vormen van communicatie (bijvoorbeeld e-mail) kennen subtiele onderlinge verschillen die grote gevolgen hebben voor de vorm van gezamenlijke kennis die mensen kunnen verkrijgen, concludeert Floor Sietsma. Met behulp van modellen uit de logica slaagt ze erin om deze verschillen exact te beschrijven en te modelleren, en op deze manier de gevolgen van de verschillende communicatievormen te onderzoeken. Het onderzoek van Sietsma biedt mogelijkheden voor het zo adequaat mogelijk verspreiden van berichten. Een andere toepassing is de bestudering van uitgelekte informatie, waarbij verzonden e-mails en andere berichten geanalyseerd worden om erachter te komen waar het lek zit.

We model lying as a communicative act changing the beliefs of the agents in a multi-agent system. With Augustine, we see lying as an utterance believed to be false by the speaker and uttered with the intent to deceive the addressee. The deceit is successful if the lie is believed after the utterance by the addressee. This is our perspective. Also, as common in dynamic epistemic logics, we model the agents addressed by the lie, but we do not (necessarily) model the speaker as one of those agents. This further simplifies the picture: we do not need to model the intention of the speaker, nor do we need to distinguish between knowledge and belief of the speaker: he is the observer of the system and his beliefs are taken to be the truth by the listeners. We provide a sketch of what goes on logically when a lie is communicated. We present a complete logic of manipulative updating, to analyse the effects of lying in public discourse. Next, we turn to the study of lying in games. First, a game-theoretical analysis is used to explain how the possibility of lying makes games such as Liar's Dice interesting, and how lying is put to use in optimal strategies for playing the game. This is the opposite of the logical manipulative update: instead of always believing the utterance, now, it is never believed. We also give a matching logical analysis for the games perspective, and implement that in the model checker DEMO. Our running example of lying in games is the game of Liar's Dice.

It is far from obvious to find logical counterparts to crytographic protocol primitives. In logic, a common assumption is that agents are perfectly rational and have no computational limitations. This creates a dilemma. If one merely abstracts from computational aspects, protocols become trivial and the difference between tractable and intractable computation, surely an essential feature of protocols, disappears. This way, the protocol gets lost. On the other hand, if one ‘merely′ (scare quotes indeed) models agents with computational limitations (or otherwise bounded rationality), very obvious aspects of reasoning become problematic. That way, the logic gets lost. We present a novel way out of this dilemma.We propose an abstract logical architecture wherein public and private, or symmetric keys, and their roles in crytographic protocols, all have formal counterparts. Instead of having encryption and decryption done by a principal, the agent sending or receiving messages, we introduce additional, virtual, agents to model that, so that one-way-function aspects of computation can be modelled as constraints on the communication between principals and these virtual counterparts. In this modelling it does not affect essential protocol features if agents are computationally unlimited.We have implemented the proposal in a dynamic epistemic model checker called DEMO.

When you send an email message to more than one address using the Cc: field, all recipients see the same message, including the To: and Cc: fields. Sending a message to undisclosed recipients (using Bcc) has a quite different communicative effect: no common knowledge about the recipient list is created. If a communication network is complex and lots of messages pass through it, it becomes a real challenge to trace who has learnt what from the communication. This challenge is taken up in a project at CWI that started recently.

The notion of non-manipulability (or: strategy-proofness) used in the famous Gibbard-Satterthwaite theorem is too strong to make useful distinctions between voting rules.We explore alternative definitions and suggest how these can be used to classify voting rules.

In this paper, we develop an epistemic logic for specifying and reasoning about information flow on the underlying communication channels. By combining ideas from Dynamic Epistemic Logic (DEL) and Interpreted Systems (IS), our semantics offers a natural and neat way of modeling multi-agent communication scenarios with different assumptions about the observational power of agents. We relate our logic to the standard DEL and IS approaches and demonstrate its use by studying a telephone call communication scenario.

We propose and study a new composition operation on (epistemic) multi-agent models with different vocabularies of propositional letters. This operation allows us to compose large models by small components representing agents' partial observational information. Our investigation provides ways to decompose (locally generated) epistemic models such that the truth of certain formulas are preserved. By using the composition operation we also propose and study action model composition and action model updates on models with arbitrary vocabularies.

We propose a framework for message passing that combines the best properties of dynamic epistemic semantics and history-based approaches. We assume that all communication is truthful and reliable. Our framework consists of Kripke models with records of sent messages in their valuations. We introduce an update operation for message sending. With this update we can study the exact epistemic consequences of sending a message. We define a class of models that is generated from initial Kripke models by means of message updates, and we axiomatize a logic for this class of models. Next, we add an update modality and sketch a procedure for defining it by means of equivalence axioms.

We show how to generate multi-agent Kripke models from message exchanges. With these models we can analyze the epistemic consequences of a message exchange. One novelty in this approach is that we include the messages in our logical language. This allows us to model messages that mention other messages and agents that reason about messages. Our framework can be used to model a wide range of different communication scenarios.

In this paper we forge a connection between dynamic epistemic logics of belief revision on one hand and studies of collective judgement and multi-agent preference change on the other. Belief revision in the spirit of dynamic epistemic logic uses updating with relational substitutions to change the beliefs of individual agents. Collective judgement in social choice theory studies the collective outcomes of individual belief changes. We start out from the logic of communication and change (LCC) without constraints, and then study the effects of imposing a single constraint, namely the constraint that the agent’s preference relations are linked. Finally, we show that the resulting framework can be used to model consensus seeking procedures. We focus on the case of plenary Dutch meetings. In Dutch meetings, a belief change (or rather: preference change) is performed for all agents in the meeting if a majority believes (or: is in favour of) the proposition that is under discussion. A special case of these meetings is judgement aggregation, and we apply our framework to the discursive dilemma in this field. Our framework has obvious connections to coalition logic and social choice theory.

We show how to generate multi-agent Kripke models from message exchanges, and conversely, how to interpret message exchanges with respect to initial multi-agent Kripke models. Any multimodal S5 Kripke model is a suitable initial model for updating by means of message exchange.

We present a case study of two black-box testing techniques. We compare constraint logic programming with symbolic transition systems. Both techniques generate automatically test cases out of the specification of an algorithm. After our case study we decided to design an algorithm to improve the technique using symbolic transition systems with a way to generate test cases automatically based on a coverage criterion. Because we do black-box testing and we have no access to the implementation of the algorithm, we apply this coverage criterion to the specification. Our algorithm attempts to cover all states of the symbolic transition system that models the specification of the algorithm. This task is complicated due to nondeterminism and symbolic constraints in the model. We use an external constraint solver to solve these constraints. We try to simplify them as much as possible because constraint solvers only have limited power and often use a great amount of computational resources.