This article contributes to the ongoing discussion in the computational linguistics community regarding instances that are difficult to annotate reliably.Is it worthwhile to identify those?What information can be inferred from them regarding the nature of the task?What should be done with them when building supervised machine learning systems?We address these questions in the context of a subjective semantic task.In this setting, we show that the presence of such instances in training data misleads a machine learner into misclassifying clear-cut cases.We also show that considering machine learning outcomes with and without the difficult cases, it is possible to identify specific weaknesses of the problem representation.

A number of recent articles in computational linguistics venues called for a closer examination of the type of noise present in annotated datasets used for benchmarking (Reidsma and Carletta, 2008; Beigman Klebanov and Beigman, 2009). In particular, Beigman Klebanov and Beigman articulated a type of noise they call annotation noise and showed that in worst case such noise can severely degrade the generalization ability of a linear classifier (Beigman and Beigman Klebanov, 2009). In this paper, we provide quantitative empirical evidence for the existence of this type of noise in a recently benchmarked dataset. The proposed methodology can be used to zero in on unreliable instances, facilitating generation of cleaner gold standards for benchmarking.

We establish the following characteristics of the task of perspective classification: (a) using term frequencies in a document does not improve classification achieved with absence/presence features; (b) for datasets allowing the relevant comparisons, a small number of top features is found to be as effective as the full feature set and indispensable for the best achieved performance, testifying to the existence of perspective-specific keywords. We relate our findings to research on word frequency distributions and to discourse analytic studies of perspective.

The strategic behavior of legislators depends on the information available before and during the legislation process. It is well established in the literature that interested parties such as voters and agenda setters can influence the outcomes of the process through strategic manipulation when they are sufficiently informed. When only partial information on the individual and collective preference is revealed the question of manipulability boils down to how much information must be revealed before a learner is able to use it strategically? This paper applies a model of single agent learning to address this question. Our results show that learning collective preferences in this setting is possible but hard, giving explicit bounds on the amount of information required. The proofs use a Ramsey type theorem for simple games showing that games with many effective voters embed games from at least one of three well-characterized families.

We present a game-theoretic model of bargaining over a metaphor in the context of political communication, find its equilibrium, and use it to rationalize observed linguistic behavior. We argue that game theory is well suited for modeling discourse as a dynamic resulting from a number of conflicting pressures, and suggest applications of interest to computational linguists.

This article discusses the transition from annotated data to a gold standard, that is, a subset that is sufficiently noise-free with high confidence. Unless appropriately reinterpreted, agreement coefficients do not indicate the quality of the data set as a benchmarking resource: High overall agreement is neither sufficient nor necessary to distill some amount of highly reliable data from the annotated material. A mathematical framework is developed that allows estimation of the noise level of the agreed subset of annotated data, which helps promote cautious benchmarking.

In previous work we have studied the use of sequential second price auctions for sharing a wireless resource, such as bandwidth or power. The resource is assumed to be managed by a spectrum broker (auctioneer), who collects bids and allocates discrete units of the resource. It is well known that a second price auction for a single indivisible good has an efficient dominant strategy equilibrium; this is no longer the case when multiple units of a homogeneous good are sold in repeated iterations. Previous work attempted to bound this inefficiency loss for two users with non-increasing marginal valuations and full information. This work was based on studying a setting in which one agent's valuation for each resource unit is strictly larger than any of the other agent's valuations and assuming a certain property of the price paid by such a dominant user in any sub-game. Using this assumption it was shown that the worst-case efficiency loss was no more than e(-1). However, here we show that this assumption is not satisfied for all non-increasing marginals with this dominance property. In spite of this, we show that it is always true for the worst-case marginals for any number of goods and so the worst-case efficiency loss for any non-increasing marginal valuations is still bounded by e(-1).

Using metaphor-annotated material that is sufficiently representative of the topical composition of a similar-length document in a large background corpus, we show that words expressing a discourse-wide topic of discussion are less likely to be metaphorical than other words in a document. Our results suggest that to harvest metaphors more effectively, one is advised to consider words that do not represent a discourse topic.

Abstract This article discusses the transition from annotated data to a gold standard, that is, a subset that is sufficiently noise-free with high confidence. Unless appropriately reinterpreted, agreement coefficients do not indicate the quality of the data set as a benchmarking resource: High overall agreement is neither sufficient nor necessary to distill some amount of highly reliable data from the annotated material. A mathematical framework is developed that allows estimation of the noise level of the agreed subset of annotated data, which helps promote cautious benchmarking.

It is usually assumed that the kind of noise existing in annotated data is random classification noise. Yet there is evidence that differences between annotators are not always random attention slips but could result from different biases towards the classification categories, at least for the harder-to-decide cases. Under an annotation generation model that takes this into account, there is a hazard that some of the training instances are actually hard cases with unreliable annotations. We show that these are relatively unproblematic for an algorithm operating under the 0--1 loss model, whereas for the commonly used voted perceptron algorithm, hard training cases could result in incorrect prediction on the uncontroversial cases at test time.

We address the problem of distinguishing between two sources of disagreement in annotations: genuine subjectivity and slip of attention. The latter is especially likely when the classification task has a default class, as in tasks where annotators need to find instances of the phenomenon of interest, such as in a metaphor detection task discussed here. We apply and extend a data analysis technique proposed by Beigman Klebanov and Shamir (2006) to first distill reliably deliberate (non-chance) annotations and then to estimate the amount of attention slips vs genuine disagreement in the reliably deliberate annotations.

This article presents a novel automatic method of text analysis aimed at discovering patterns of lexical cohesion in political speech. The unit of analysis are groups of words with related meanings; the software is based on the results of a multiperson annotation experiment that captures reliably identified connections between words in a text. We illustrate the advantages of such a representation by juxtaposing results of a detailed hand-made analysis of Margaret Thatcher's rhetoric with analysis based on the automatically detected groups of words. We both corroborate previous findings regarding Thatcher's rhetorical tools and illuminate additional elements thereof. We suggest that lexical cohesion analysis is a promising technique to bridge the gap between quantitative and qualitative analyses of text as political material, by establishing units that are both robust enough to enable comprehensive coverage and coherent enough to support direct interpretation.

We study a sequential second price auction mechanism for sharing wireless resources among competing transmitters. It is well known that a second price auction for a single indivisible good has an efficient dominant strategy equilibrium; this is no longer the case when multiple units of a homogeneous good are sold in repeated iterations. Nevertheless, in the wireless industry there are pragmatic reasons to prefer mechanisms that are, in effect, equivalent to sequential auctions. Our objective is to study the equilibrium outcomes in these settings and assess their performance. To focus our study on the strategic implications, setting aside issues such as beliefs, incomplete information and collusion, we assume bidders have full information on the mutual valuations. Our results show that, regardless of the number of bidders, there always exists a pure strategy equilibrium. For the two users case, we show that this equilibrium is unique and could result in loss of efficiency. We give bounds on the loss and characterize the worst cases under various constraints. We conclude with some numerical results for losses in the average case.

It has been suggested that light regulation in the form of etiquette protocols, device design and bargaining amongst users will suffice to mitigate a tragedy of the commons in unlicensed spectrum. In this paper we propose a game theoretic model to examine this claim. In this game, each user decides whether or not to set up an access point, which operates on a particular (single) band. The effect of regulation is modeled in reduced form through transfers. A user who sets up an access point, provides payments to each neighbor who does not and suffers a disutility depending on the number of interfering access points. A user who does not set up an access point, receives payments from each neighbor that does. For a suitable model of payoffs, the game is a potential game and best response updates converge to a Nash equilibrium of the game. For any interference parameters, there is a suitable transfer resulting in a Nash equilibrium which is efficient. However, all Nash equilibria may not be efficient.

It has been widely recognized that the current under-utilization of spectrum across many bands could be alleviated through the application of spectrum markets. So far, discussions of market mechanisms for spectrum allocations have focused primarily on secondary markets, which are managed by licensees. Here we explore the consequences of lifting current restrictions on allocations and ownership, and allowing more extensive markets for allocating spectrum across locations, times, and diverse sets of applications (e.g., broadcast, cellular, broadband data, emergency, etc). To motivate our discussion we first estimate the achievable rate per user that could be provided by sharing a large portion of the spectrum suitable for cellular and broadcast types of services. Our results suggest that in general the demand for spectrum may exceed supply implying that market mechanisms are needed to avoid a tragedy of the commons (i.e., associated with an alternative commons model). We then discuss a two- tier spectrum market structure for wireless services in which licenses for spectrum assets at particular locations are traded as commodities. Spectrum owners can choose to rent or lease their spectrum assets via spot markets at particular locations. Such an approach may lower barriers to entry into the wireless services market thereby facilitating competition and the introduction of new services.

ABSTRACT This article discusses methods for automatic annotation of political texts for semantic fields—groups of words with related meanings. This type of annotation is useful when studying political communication, such as legislative debate or political speeches. We present three types of automatic annotation: unsupervised clustering, dictionary-based approaches, and a method based on relevant experimental data. All methods are applied to analyzing Margaret Thatcher's political rhetoric. For this data, we find that unsupervised clustering is most useful for tracing topics; dictionary-based methods are most effective in a comparative setting; whereas the last method is the most promising for detecting off-topic, singular uses of semantic domains, which are often rhetorical tools used to achieve a political end. Applicability, strengths, and weaknesses of each method and of their combinations are addressed in detail.

We study an implementation problem for settings where some of the participants have non concave valuations. These valuations are common in the wireless industry in cases where primary users with property rights on a spectrum band would like to lease some of it to low power users. The non concavity void the efficiency results for standard designs of dynamic auctions. Moreover, policy concerns in such settings is often to prevent collusion and fraudulent bidding, therefore static Vickrey mechanisms do not provide the right incentives. We present an alternative mechanism that selects a core outcome that minimizes seller revenue. Such an allocation is efficient in equilibrium, limits the incentives to use shills, maximizes incentives for truthful bidding, and gives a Vickrey outcome whenever the latter is in the core.

We study a sequential auction for sharing a wireless resource (bandwidth or power) among competing transmitters. The resource is assumed to be managed by a spectrum broker (auctioneer), who collects bids and allocates discrete units of the resource via a sequential second-price auction. It is well known that a second price auction for a single indivisible good has an efficient dominant strategy equilibrium; this is no longer the case when multiple units of a homogeneous good are sold in repeated iterations. For two users with full information, we show that such an auction has a unique equilibrium allocation. The worst-case efficiency of this allocation is characterized under the following cases: (i) both bidders have a concave valuation for the spectrum resource, and (ii) one bidder has a concave valuation and the other bidder has a convex valuation (e.g., for the other useriquests power). Although the worst-case efficiency loss can be significant, numerical results are presented, which show that for randomly placed transmitter-receiver pairs with rate utility functions, the sequential second-price auction typically achieves the efficient allocation. For more than two users it is shown that this mechanism always has a pure strategy equilibrium, but in general there may be multiple equilibria. We give a constructive procedure for finding one equilibrium; numerical results show that when all users have concave valuations the efficiency loss decreases with an increase in the number of users.

We study a sequential second-price auction for allocating wireless resources between two non-cooperative users. This mechanism requires relatively little computation and information exchange among agents, but does not always achieve an efficient allocation. This is a continuation of previous work in which the worst-case efficiency is evaluated, assuming each user has full knowledge of the other user's utility function. Here we assume that the users are randomly placed within a region, and evaluate the associated efficiency via simulation. Sequential auctions for bandwidth (with fixed power) and for power (with fixed bandwidth) are considered, where each user utility is the achievable rate, and interference is treated as background noise. Our results show that the sequential auction typically achieves the efficient (utility-maximizing) allocation. We also relate observed improvements in the worst-case efficiency to constraints on the size of the marginal utilities associated with each resource.