In this paper, we develop a new conceptual framework for an important problem in language acquisition, the correspondence problem: the fact that a given utterance has different manifestations in the speech and articulation of different speakers and that the correspondence of these manifestations is difficult to learn. We put forward the Correspondence-by-Segmentation Hypothesis, which states that correspondence is primarily learned by first segmenting speech in an unsupervised manner and then mapping the acoustics of different speakers onto each other. We show that a rudimentary segmentation of speech can be learned in an unsupervised fashion. We then demonstrate that, using the previously learned segmentation, different instances of a word can be mapped onto each other with high accuracy when trained on utterance-label pairs for a small set of words.
We develop an unsupervised algorithm for morphological acquisition to investigate the relationship between linguistic representation, data statistics, and learning algorithms. We model the phenomenon that children acquire the morphological inflections of a language monotonically by introducing an algorithm that uses a bootstrapped, frequency-driven learning procedure to acquire rules monotonically. The algorithm learns a morphological grammar in terms of a Base and Transforms representation, a simple rule-based model of morphology. When tested on corpora of child-directed speech in English from CHILDES (MacWhinney in The CHILDES-Project: Tools for analyzing talk. Erlbaum, Hillsdale, 2000 ), the algorithm learns the most salient rules of English morphology and the order of acquisition is similar to that of children as observed by Brown (A first language: the early stages. Harvard University Press, Cambridge, 1973 ). Investigations of statistical distributions in corpora reveal that the algorithm is able to acquire morphological grammars due to its exploitation of Zipfian distributions in morphology through type-frequency statistics. These investigations suggest that the computation and frequency-driven selection of discrete morphological rules may be important factors in children’s acquisition of basic inflectional morphological systems.
We examine the impact of domain on parse selection accuracy, in the context of precision HPSG parsing using the English Resource Grammar, using two training corpora and four test corpora and evaluating using exact tree matches as well as dependency F-scores. In addition to determining the relative impact of in- vs. cross-domain parse selection training on parser performance, we propose strategies to avoid cross-domain performance penalty when limited in-domain data is available. Our work supports previous research showing that in-domain training data significantly improves parse selection accuracy, and that it provides greater parser accuracy than an out-of-domain training corpus of the same size, but we verify experimentally that this holds for a handcrafted grammar, observing a 10–16% improvement in exact match and 5–6% improvement in dependency F-score by using a domain-matched training corpus. We also find it is possible to considerably improve parse selection accuracy through construction of even small-scale in-domain treebanks, and learning of parse selection models over in-domain and out-of-domain data. Naively adding an 11,000-token in-domain training corpus boosts dependency F-score by 2–3% over using solely out-of-domain data. We investigate more sophisticated strategies for combining data from these sources to train models: weighted linear interpolation between the single-domain models, and training a model from the combined data, optionally duplicating the smaller corpus to give it a higher weighting. The most successful strategy is training a monolithic model after duplicating the smaller corpus, which gives an improvement over a range of weightings, but we also show that the optimal value for these parameters can be estimated on a case-by-case basis using a cross-validation strategy. This domain-tuning strategy provides a further performance improvement of up to 2.3% for exact match and 0.9% for dependency F-score compared to the naive combination strategy using the same data.
Denotations of anaphoric determiners are treated as functions from sets and relations to sets. Constraints on such denotations are studied, and shown to generalize anaphor conditions known from the study of simpler cases of nominal anaphors. In addition a generalisation of the notion of conservativity as applied to anaphoric functions is presented. Two classes of anaphoric determiners are discussed as examples: complex anaphoric determiners in English (e.g. no…except herself) and simple anaphoric determiners in Slavic languages (e.g. Polish swój “his own”)
In recent years, Bayesian models have become increasingly popular as a way of understanding human cognition. Ideal learner Bayesian models assume that cognition can be usefully understood as optimal behavior under uncertainty, a hypothesis that has been supported by a number of modeling studies across various domains (e.g., Griffiths and Tenenbaum, Cognitive Psychology, 51, 354–384, 2005; Xu and Tenenbaum, Psychological Review, 114, 245–272, 2007). The models in these studies aim to explain why humans behave as they do given the task and data they encounter, but typically avoid some questions addressed by more traditional psychological models, such as how the observed behavior is produced given constraints on memory and processing. Here, we use the task of word segmentation as a case study for investigating these questions within a Bayesian framework. We consider some limitations of the infant learner, and develop several online learning algorithms that take these limitations into account. Each algorithm can be viewed as a different method of approximating the same ideal learner. When tested on corpora of English child-directed speech, we find that the constrained learner’s behavior depends non-trivially on how the learner’s limitations are implemented. Interestingly, sometimes biases that are helpful to an ideal learner hinder a constrained learner, and in a few cases, constrained learners perform equivalently or better than the ideal learner. This suggests that the transition from a computational-level solution for acquisition to an algorithmic-level one is not straightforward.
This paper presents the LinGO Grammar Matrix grammar customization system, a web-based service which elicits typological descriptions of languages and outputs customized grammar fragments which are ready for sustained development into broad-coverage grammars. We describe the infrastructure we have developed to support grammar customization as well as the current set of linguistic phenomena addressed, reflect on what we have learned about a methodology for this style of multilingual grammar engineering, and evaluate the typological breadth of the system by using it to create grammars for seven languages from seven different language families.
How children acquire language has stood, for a long time, as one of the most fundamental, beguiling, and surprisingly open questions of modern science. Recent advances in natural language processing, statistical parsing and machine learning, together with the availability of large corpora of child directed speech and other corpora, make a wide range of computationally-oriented approaches to the study of this problem available. Given a model of some aspect of language acquisition, implementing it as a computational system and evaluating it on naturally occurring corpora has a number of compelling advantages. First of all by implementing the system, we can be sure that the algorithm is fully specified, and the acquisition model does not resort to hand-waving at crucial points. Secondly, by evaluating it on real linguistic data, we can see whether naturally occurring distributions of examples in corpora provide sufficient information to support the studied claims across a divergent range of acquisition theories. Thirdly, study of the system can identify the mechanisms that cause changes in the algorithm’s hypotheses during the course of acquisition. Finally, the computational resources required of the model can be concretely assessed and (not so concretely) compared against the resources that might be available to a human language learner.
We consider the task of learning three verb classes: raising (e.g., seem), control (e.g., try) and ambiguous verbs that can be used either way (e.g., begin). These verbs occur in sentences with similar surface forms, but have distinct syntactic and semantic properties. They present a conundrum because it would seem that their meaning must be known to infer their syntax, and that their syntax must be known to infer their meaning. Previous research with human speakers pointed to the usefulness of two cues found in sentences containing these verbs: animacy of the sentence subject and eventivity of the predicate embedded under the main verb. We apply a variety of algorithms to this classification problem to determine whether the primary linguistic data is sufficiently rich in this kind of information to enable children to resolve the conundrum, and whether this information can be extracted in a way that reflects distinctive features of child language acquisition. The input consists of counts of how often various verbs occur with animate subjects and eventive predicates in two corpora of naturalistic speech, one adult-directed and the other child-directed. Proportions of the semantic frames are insufficient. A Bayesian attachment model designed for a related language learning task does not work well at all. A hierarchical Bayesian model (HBM) gives significantly better results. We also develop and test a saturating accumulator that can successfully distinguish the three classes of verbs. Since the HBM and saturating accumulator are successful at the classification task using biologically realistic calculations, we conclude that there is sufficient information given subject animacy and predicate eventivity to bootstrap the process of learning the syntax and semantics of these verbs.
In this paper, we propose a classifier ensemble technique based on genetic algorithm (GA) for named entity recognition (NER). We assume that the classifiers based on different feature representations can be effectively combined together using GA to achieve better performance. The proposed approach is also able to find the appropriate ensemble approach, i.e. either majority voting or weighted voting. Maximum entropy (ME) model is used as a base to generate a number of different classifiers depending upon the various representations of the available features. The proposed approach is evaluated for three leading Indian languages, namely Bengali, Hindi and Telugu. Evaluation results yield the recall, precision and F-measure values of 88.12, 93.99 and 90.96%, respectively for Bengali, 80.26, 92.70 and 86.03%, respectively for Hindi and 74.79, 85.38 and 79.73%, respectively for Telugu. We also evaluate the proposed approach with the CoNLL-2003 benchmark English datasets and it shows the recall, precision and F-measure values of 83.05, 85.52 and 84.27%, respectively. It is observed that the GA based ensemble attains the performance which is superior to all the individual classifiers as well as two conventional baseline ensembles for all the languages.
We describe a framework for performing morphological analysis to account for learner language, focusing on Russian as an example of an inflecting language. Because a set of linguistic analyses is needed to provide feedback on potentially noisy data, there is a large amount of ambiguity for even well-formed words. Using a segmented POS lexicon as a test case, we show how to analyze subparts of words, in order to analyze variations. After describing and implementing this framework for Russian, we focus on removing undesirable analyses to keep the task feasible. This is essentially an investigation of how much overgeneration of analyses is a problem and under what assumptions it can be reduced.
We provide a corpus-based computational approach to analyzing acquisition data on Turkish, a richly inflected language. We describe the process by which transcripts from the CHILDES database for 16 children aged 2;0 to 3;0 were morphologically tagged and parsed. We computed a number of imitation, overlap, and repetition measures on the transcripts using CLAN and CHIP programs. These measures tended to decrease as a function of mean length of utterance, which was broadly consistent with previously published work on English-speaking children. The data also revealed additional usage patterns, where the adult utterances provided children with rich morphosyntax in the input, while at the same time helping them to maintain discourse. Children on the other hand, tended to omit optional constituents and repeat morphemes from the previous utterance. The Turkish data and previously published English data showed cross-linguistic differences in repetition patterns that were congruent with the typological differences between the two languages. More generally, the data were consistent with a usage-based model for the acquisition of Turkish as a first language. The corpora and methods provided here can be extended to future applications.
This paper is concerned with the relation between mild context-sensitivity and the class of languages generated by Linear Context-Free Rewriting Systems (LCFRSs). We show that there are languages that are polynomial and of constant growth but that are not LCFRLs. Starting from this observation, we define an extension of LCFRS that, roughly, allows for a limited amount of copying (or intersection, if considered in a bottom-up perspective). The proposed LCFRS extension, Literal Movement Grammars of constant non-linearity (CNL-LMG) is such that the parts that are copied into different places during a derivation cannot increase when iterating the non-linear parts of a derivation. We show that this condition guarantees that the string languages are still of constant growth. As a consequence, CNL-LMG is mildly context-sensitive while properly extending LCFRS. This result suggests that a limited and controlled amount of copying and intersection can remain tractable since it does not lead outside mild context-sensitivity. Furthermore, we show that there are natural language phenomena (in particular gapping and scrambling) where such a limited possibility of intersection gives the necessary expressive power beyond LCFRS to model these phenomena.
In this article, we investigate the use of semantic information in parse selection. We show that fully disambiguated sense-based semantic features smoothed using ontological information are effective for parse selection. Training and testing was undertaken using definition and example sentences taken from a Japanese dictionary corpus (Hinoki), which is manually annotated with senses. A model employing both syntactic and semantic information provides better parse selection accuracy than a model using only syntactic features.
This paper provides a brief algebraic characterization of constraint violations in Optimality Theory (OT). I show that if violations are taken to be multisets over a fixed basis set Con then the merge operator on multisets and a ‘min’ operation expressed in terms of harmonic inequality provide a semiring over violation profiles. This semiring allows standard optimization algorithms to be used for OT grammars with weighted finite-state constraints in which the weights are violation-multisets. Most usefully, because multisets are unordered, the merge operation is commutative and thus it is possible to give a single graph representation of the entire class of grammars (i.e. rankings) for a given constraint set. This allows a neat factorization of the optimization problem that isolates the main source of complexity into a single constant γ denoting the size of the graph representation of the whole constraint set. I show that the computational cost of optimization is linear in the length of the underlying form with the multiplicative constant γ. This perspective thus makes it straightforward to evaluate the complexity of optimization for different constraint sets.
In recent years a number of methods have been proposed for the automatic acquisition of feature-based conceptual representations from text corpora. Such methods could offer valuable support for theoretical research on conceptual representation. However, existing methods do not target the full range of concept-relation-feature triples occurring in human-generated norms (e.g. flute produce sound ) but rather focus on concept-feature pairs (e.g. flute – sound ) or triples involving specific relations only (e.g. is-a or part-of relations). In this article we investigate the challenges that need to be met in both methodology and evaluation when moving towards the acquisition of more comprehensive conceptual representations from corpora. In particular, we investigate the usefulness of three types of knowledge in guiding the extraction process: encyclopedic, syntactic and semantic. We present first a semantic analysis of existing, human-generated feature production norms, which reveals information about co-occurring concept and feature classes. We introduce then a novel method for large-scale feature extraction which uses the class-based information to guide the acquisition process. The method involves extracting candidate triples consisting of concepts, relations and features (e.g. deer have antlers , flute produce sound ) from corpus data parsed for grammatical dependencies, and re-weighting the triples on the basis of conditional probabilities calculated from our semantic analysis. We apply this method to an automatically parsed Wikipedia corpus which includes encyclopedic information and evaluate its accuracy using a number of different methods: direct evaluation against the McRae norms in terms of feature types and frequencies, human evaluation, and novel evaluation in terms of conceptual structure variables. Our investigation highlights a number of issues which require addressing in both methodology and evaluation when aiming to improve the accuracy of unconstrained feature extraction further.
In Separating Syntax and Combinatorics in Categorial Grammar, Muskens illustrates a categorial architecture called 'Lambda Grammars' comprising a tectogrammatical level of linear λ-terms relating pointwise a phenogrammatical dimension of form and a semantic dimension of meaning.
We show in this article how an approach developed for the task of recognizing textual entailment relations can be extended to identify paraphrase and elaboration relations. Entailment is a unidirectional relation between two sentences in which one sentence logically infers the other. There seems to be a close relation between entailment and two other sentence-to-sentence relations: elaboration and paraphrase. This close relation is discussed to theoretically justify the newly derived approaches. The proposed approaches use lexical, syntactic, and shallow negation handling. The proposed approaches offer significantly better results than several baselines. When compared to other paraphrase and elaboration approaches they produce similar or better results. We report results on several data sets: the Microsoft Research Paraphrase corpus, a benchmark for evaluating approaches to paraphrase identification, and a data set collected from high-school students' interactions with an intelligent tutoring system iSTART, which includes both paraphrase and elaboration utterances.
Syntactic preprocessing is a step that is widely used in NLP applications. Traditionally, rule-based or statistical Part-of-Speech (POS) taggers are employed that either need considerable rule development times or a sufficient amount of manually labeled data. To alleviate this acquisition bottleneck and to enable preprocessing for minority languages and specialized domains, a method is presented that constructs a statistical syntactic tagger model from a large amount of unlabeled text data. The method presented here is called unsupervised POS-tagging, as its application results in corpus annotation in a comparable way to what POS-taggers provide. Nevertheless, its application results in slightly different categories as opposed to what is assumed by a linguistically motivated POS-tagger. These differences hamper evaluation procedures that compare the output of the unsupervised POS-tagger to a tagging with a supervised tagger. To measure the extent to which unsupervised POS-tagging can contribute in application-based settings, the system is evaluated in supervised POS-tagging, word sense disambiguation, named entity recognition and chunking. Unsupervised POS-tagging has been explored since the beginning of the 1990s. Unlike in previous approaches, the kind and number of different tags is here generated by the method itself. Another difference to other methods is that not all words above a certain frequency rank get assigned a tag, but the method is allowed to exclude words from the clustering, if their distribution does not match closely enough with other words. The lexicon size is considerably larger than in previous approaches, resulting in a lower out-of-vocabulary (OOV) rate and in a more consistent tagging. The system presented here is available for download as open-source software along with tagger models for several languages, so the contributions of this work can be easily incorporated into other applications.
We investigate the problem of computing the partition function of a probabilistic context-free grammar, and consider a number of applicable methods. Particular attention is devoted to PCFGs that result from the intersection of another PCFG and a finite automaton. We report experiments involving the Wall Street Journal corpus.
