Most expressivity results for transformers treat them as language recognizers (which accept or reject strings), and not as they are used in practice, as language models (which generate strings autoregressively and probabilistically). We characterize the probability distributions that transformer language models can express. We show that making transformer language recognizers autoregressive can sometimes increase their expressivity, and that making them probabilistic can break equivalences that hold in the non-probabilistic case. Our overall contribution is to tease apart what functions transformers are capable of expressing, in their most common use-case as language models.

The commentary argues the authors employ misdirection and strawmanning to cast others as polarized extremes and themselves as the reasonable centrists. We argue that these patterns of misrepresentation ultimately damage any consensus and middle ground they claim to hope to reach.

Abstract We provide empirical evidence against Wilson & Gallagher’s 2018 claim that statistics is necessary for phonotactic learning. We implement BUFIA, a feature-based nonstatistical learner and, using the same data and case study as Wilson & Gallagher, show that this non-statistical learner is equally successful at learning phonotactics as the Maximum Entropy-based learner they use. This counters their conclusion that non-statistical phonotactic learning is impossible, while supporting their advocacy for feature-based representations in phonotactic learning.

Most expressivity results for transformers treat them as language recognizers (which accept or reject strings), and not as they are used in practice, as language models (which generate strings autoregressively and probabilistically). We characterize the probability distributions that transformer language models can express. We show that making transformer language recognizers autoregressive can sometimes increase their expressivity, and that making them probabilistic can break equivalences that hold in the non-probabilistic case. Our overall contribution is to tease apart what functions transformers are capable of expressing, in their most common use-case as language models.

We study the sequence-to-sequence mapping capacity of transformers by relating them to finite transducers, and find that they can express surprisingly large classes of (total functional) transductions. We do so using variants of RASP, a programming language designed to help people "think like transformers,"as an intermediate representation. We extend the existing Boolean variant B-RASP to sequence-to-sequence transductions and show that it computes exactly the first-order rational transductions (such as string rotation). Then, we introduce two new extensions. B-RASP[pos] enables calculations on positions (such as copying the first half of a string) and contains all first-order regular transductions. S-RASP adds prefix sum, which enables additional arithmetic operations (such as squaring a string) and contains all first-order polyregular transductions. Finally, we show that masked average-hard attention transformers can simulate S-RASP.

The cognitive sciences are facing questions of how to select from competing theories or develop those that suit their current needs. However, traditional accounts of theoretical virtues have not yet proven informative to theory development in these fields. We advance a pragmatic account by which theoretical virtues are heuristics we use to estimate a theory’s contribution to a field’s body of knowledge and the degree to which it increases that knowledge’s ability to solve problems in the field’s domain or problem space. From this perspective, properties that are traditionally considered epistemic virtues, such as a theory’s fit to data or internal coherence, can be couched in terms of problem space coverage, and additional virtues come to light that reflect a theory’s alignment with problem-having agents and the context in a societally embedded scientific system. This approach helps us understand why the needs of different fields result in different kinds of theories and allows us to formulate the challenges facing cognitive science in terms that we hope will facilitate their resolution through further theoretical development.

Abstract This chapter discusses the nature of vowel harmony (VH) from a computational perspective. In particular, it considers the dependencies present in surface forms of phonological representations obeying various types of harmony. It situates this typological landscape of dependencies within a corresponding landscape of computational functions. The precise nature of these functions gives us a principled window into the characteristic similarities and differences between harmony types. It goes on to show that this landscape is restricted, and that VH is tightly constrained typologically, describing how the interactions of these functions elucidate key properties of harmonic systems, and how these properties enable new avenues for typological and computational work.

We explore the generative capacity of morphological theories of reduplication. We computationally classify theories of reduplication using a hierarchy of string-to-string function classes. Reduplication as a process requires only the regular class of functions. We show that various morphological theories necessarily treat it as a more expressive polyregular function, while others maintain regularity. We discuss the significance of this formal result for reduplicative functions and recognition.

Recombining known primitive concepts into larger novel combinations is a quintessentially human cognitive capability. Whether large neural models in NLP acquire this ability while learning from data is an open question. In this paper, we look at this problem from the perspective of formal languages. We use deterministic finite-state transducers to make an unbounded number of datasets with con-trollable properties governing compositionality. By randomly sampling over many transducers, we explore which of their properties (number of states, alphabet size, number of transitions etc.) contribute to learnability of a compositional relation by a neural network. In general, we find that the models either learn the relations completely or not at all. The key is transition coverage, setting a soft learnability limit at 400 examples per transition.

In this chapter, we examine the nature of the long-term memory representation of the pronunciations of words. A fundamental question concerns how abstract these representations are vis à vis the physical manifestation of words, both as gestures and as physical percepts. We consider this question and related issues within the traditions of linguistic cognition and generative phonology. We first explore the general nature of abstraction, and then review the arguments in generative phonology for positing that the units of speech stored in long-term memory (so called ‘underlying forms’) abstract away from many phonetic details. Motivations for concepts such as phonemes and distinctive phonological features are given. We then visit the open question regarding how abstract underlying forms may be allowed to be. We conclude by highlighting the contributions that evidence from neuroscience and sign language linguistics brings to these issues of phonological abstraction in the mental lexicon.

This chapter examines the brief but vibrant history of learnability in phonology. We trace the question of learnability back to the foundational crises in mathematics and computer science, through the synthesis of these fields with linguistics, and onwards to the foundational problems of language, and phonological, learning. We observe this history is mostly one-sided, with many ideas from learning imported to phonology, but rarely the converse. We review some of the most significant interactions between formal learnability and phonology, topics such as the necessity of structured hypothesis spaces, the credit/blame/hidden structure problem, and the subset principle. We finish by discussing several overarching tensions pervading this field: the role of mathematical descriptions versus computational simulations of learning, typological versus learnability concerns in grammar design, and debates on the psychological reality of phonological grammars. As a field, we should not fear rapid change or the many flowering prospects.

Recombining known primitive concepts into larger novel combinations is a quintessentially human cognitive capability. Whether large neural models in NLP can acquire this ability while learning from data is an open question. In this paper, we investigate this problem from the perspective of formal languages. We use deterministic finite-state transducers to make an unbounded number of datasets with controllable properties governing compositionality. By randomly sampling over many transducers, we explore which of their properties contribute to learnability of a compositional relation by a neural network. We find that the models either learn the relations completely or not at all. The key is transition coverage, setting a soft learnability limit at 400 examples per transition.

The complexity of linguistic patterns has been object of extensive debate in research programs focused on probing the inherent structure of human language abilities. But in what sense is a linguistic phenomenon more complex than another, and what can complexity tell us about the connection between linguistic typology and human cognition? This chapter approaches these questions by presenting a broad and informal introduction to the vast literature on formal language theory, computational learning theory, and artificial grammar learning. In doing so, it hopes to provide readers with an understanding of the relevance of mathematically grounded approaches to cognitive investigations into linguistic complexity, and thus further fruitful collaborations between cognitive scientists and mathematically inclined linguist and psychologist.

We derive well-understood and well-studied subregular classes of formal languages purely from the computational perspective of algorithmic learning problems. We parameterise the learning problem along dimensions of representation and inference strategy. Of special interest are those classes of languages whose learning algorithms are necessarily not prohibitively expensive in space and time, since learners are often exposed to adverse conditions and sparse data. Learned natural language patterns are expected to be most like the patterns in these classes, an expectation supported by previous typological and linguistic research in phonology. A second result is that the learning algorithms presented here are completely agnostic to choice of linguistic representation. In the case of the subregular classes, the results fall out from traditional model-theoretic treatments of words and strings. The same learning algorithms, however, can be applied to model-theoretic treatments of other linguistic representations such as syntactic trees or autosegmental graphs, which opens a useful direction for future research.

We demonstrate a computational restriction on iterative prosody in phonology by using logical transductions. We show that the typology is fundamentally local but requires output recursion, formulated via quantifier-free transductions and least-fixed-point operators, respectively. We focus on two case studies from iterative prosody. One is iterative secondary stress. The other is more complex: iterative syllabification and epenthesis in Arabic dialects. The second case study involves formalizing Ito (1989)'s analysis of directional syllabification.

Spoken and signed languages differ because of the affordances of the human body and the limits of each medium. But can commonalities between the two be compared to find abstract language universals?

Morphological processes are generally computable with 1-way finite-state transducers. However, we show that 1-way transducers do not capture the strong generative capacity of certain morphological analyses for more complex processes, including mobile affixation, infixation, and partial reduplication. As diagnostics for strong generative capacity, we use origin semantics and order-preservation. These analyze the input-output correspondences generated by finite-state transducers and their corresponding logical transductions. For some linguistic analyses of these complex processes, their strong generative capacity is matched by more expressive grammars, such as non-order-preserving transductions and their corresponding 2-way finite-state transducers.

This paper explores a computational characterization of iterative phonological processes including stress, epenthesis, and syllabification. Understanding the computational requirements of these iterative mappings leads to restrictive, testable, and learnable theories of phonology (Heinz, 2018). Previous work in this vein has been done on tone (Koser et al., 2019), and we expand the results to other kinds of processes. We show that these iterative prosodic processes are fundamentally local, thus fitting the typology of other computational work (Chandlee & Heinz, 2018). However, they require reference to local information in the output, rather than just the input. We formulate this output-centeredness via logical transductions (Courcelle, 1997), where an output element receives its output label or output property based on some requirement defined over the input. Additionally, the iterative nature of the processes requires a notion of recursion, formalized in a logical transduction via least fixed point logic (LFP; Libkin, 2013) which we require to be quantifier free (QF; Chandlee & Lindell, in prep). The restriction to QF (i.e. no reference to logical quantifiers ∃ or ∀) ensures that the notion of locality in the output is preserved, while still characterizing the relevant iterative processes (Chandlee & Jardine, 2019). Rather than introduce the full LFP formalism here, we employ the implicit definitions of Rogers (1996), which allow for output predicates to reference themselves as part of their definition. This provides a more intuitive characterization of the recursion necessary to define the target phonological processes. Finally, a substantive restriction of our transduction to the use of only the predecessor or successor function – not both – further constrains the computational power of these transductions to better fit the observed typology of iterative phonological patterns. Informally, this means that we these iterative prosodic processes apply in a single direction: right-to-left, or left-to-right, but not bidirectional.