Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.

Recent evidence reveals a precocious link between language and cognition in human infants: listening to their native language supports infants' core cognitive processes, including object categorization, and does so in a way that other acoustic signals (e.g., time-reversed speech; sine-wave tone sequences) do not. Moreover, language is not the only signal that confers this cognitive advantage: listening to vocalizations of non-human primates also supports object categorization in 3- and 4-month-olds. Here, we move beyond primate vocalizations to clarify the breadth of acoustic signals that promote infant cognition. We ask whether listening to birdsong, another naturally produced animal vocalization, also supports object categorization in 3- and 4-month-old infants. We report that listening to zebra finch song failed to confer a cognitive advantage. This outcome brings us closer to identifying a boundary condition on the range of non-linguistic acoustic signals that initially support infant cognition.

The power and precision with which humans link language to cognition is unique to our species. By 3-4 months of age, infants have already established this link: simply listening to human language facilitates infants' success in fundamental cognitive processes. Initially, this link to cognition is also engaged by a broader set of acoustic stimuli, including non-human primate vocalizations (but not other sounds, like backwards speech). But by 6 months, non-human primate vocalizations no longer confer this cognitive advantage that persists for speech. What remains unknown is the mechanism by which these sounds influence infant cognition, and how this initially broader set of privileged sounds narrows to only human speech between 4 and 6 months. Here, we recorded 4- and 6-month-olds' EEG responses to acoustic stimuli whose behavioral effects on infant object categorization have been previously established: infant-directed speech, backwards speech, and non-human primate vocalizations. We document that by 6 months, infants' 4-9 Hz neural activity is modulated in response to infant-directed speech and non-human primate vocalizations (the two stimuli that initially support categorization), but that 4-9 Hz neural activity is not modulated at either age by backward speech (an acoustic stimulus that doesn't support categorization at either age). These results advance the prior behavioral evidence to suggest that by 6 months, speech and non-human primate vocalizations elicit distinct changes in infants' cognitive state, influencing performance on foundational cognitive tasks such as object categorization.

There is ample evidence of racial and gender bias in young children, but thus far this evidence comes almost exclusively from children's responses to a single social category (either race or gender). Yet we are each simultaneously members of many social categories (including our race and gender). Among adults, racial and gender biases intersect: negative racial biases are expressed more strongly against males than females. Here, we consider the developmental origin of bias at the intersection of race and gender. Relying on both implicit and explicit measures, we assessed 4-year-old children's responses to target images of children who varied systematically in both race (Black and White) and gender (male and female). Children revealed a strong and consistent pro-White bias. This racial bias was expressed more strongly for males than females: children's responses to Black boys were less positive than to Black girls, White boys or White girls. This outcome, which constitutes the earliest evidence of bias at the intersection of race and gender, underscores the importance of addressing bias in the first years of life.

The power of human language derives not only from the precision of its signal or the complexity of its grammar, but also from its links to cognition. Infants as young as 3 months have begun to link language and core cognitive capacities. At 3 and 4 months, this link is not exclusive to human language: listening to vocalizations of nonhuman primates also supports infant cognition. By 6 months, infants have tuned this link to human speech alone. Here we provide evidence that infants’ increasing precision in speech perception shapes which signals they will link to cognition. Infants listening to German, a nonnative language that shares key rhythmic and prosodic properties with their own native language (English), successfully formed object categories. In contrast, those listening to Cantonese, a language that differs considerably in these suprasegmental properties, failed. This provides the first evidence that infants’ increasingly precise perceptual tuning to the sounds of their native language sets constraints on the range of human languages they will link to cognition: infants begin to specify which human languages they will link to core cognitive capacities even before they sever the link between nonhuman primate vocalizations and cognition.

Human language, a signature of our species, derives its power from its links to human cognition. For centuries, scholars have been captivated by this link between language and cognition. In this article, we shift this focus. Adopting a developmental lens, we review recent evidence that sheds light on the origin and developmental unfolding of the link between language and cognition in the first year of life. This evidence, which reveals the joint contributions of infants' innate capacities and their sensitivity to experience, highlights how a precocious link between language and cognition advances infants beyond their initial perceptual and conceptual capacities. The evidence also identifies the conceptual advantages this link brings to human infants. By tracing the emergence of a language-cognition link in infancy, this article reveals a dynamic developmental cascade in infants' first year, with each developmental advance providing a foundation for subsequent advances.

At birth, infants not only prefer listening to human vocalizations, but also have begun to link these vocalizations to cognition: For infants as young as three months of age, listening to human language supports object categorization, a core cognitive capacity. This precocious link is initially broad: At 3 and 4 months, vocalizations of both humans and nonhuman primates support categorization. But by 6 months, infants have narrowed the link: Only human vocalizations support object categorization. Here we ask what guides infants as they tune their initially broad link to a more precise one, engaged only by the vocalizations of our species. Across three studies, we use a novel exposure paradigm to examine the effects of experience. We document that merely exposing infants to nonhuman primate vocalizations enables infants to preserve the early-established link between this signal and categorization. In contrast, exposing infants to backward speech - a signal that fails to support categorization at any age - offers no such advantage. Our findings reveal the power of early experience as infants specify which signals, from an initially broad set, they will continue to link to cognition.

Well before they understand their first words, infants have begun to link language and cognition. This link is initially broad: At 3months, listening to both human and nonhuman primate vocalizations supports infants' object categorization, a building block of cognition. But by 6months, the link has narrowed: Only human vocalizations support categorization. What mechanisms underlie this rapid tuning process? Here, we document the crucial role of infants' experience as infants tune this link to cognition. Merely exposing infants to nonhuman primate vocalizations permits them to preserve, rather than sever, the link between these signals and categorization. Exposing infants to backward speech-a signal that fails to support categorization in the first year of life-does not have this advantage. This new evidence illuminates the central role of early experience as infants specify which signals, from an initially broad set, they will continue to link to core cognitive capacities.

The power of human language rests upon its intricate links to human cognition. By 3 months of age, listening to language supports infants' ability to form object categories, a building block of cognition. Moreover, infants display a systematic shift between 3 and 4 months - a shift from familiarity to novelty preferences - in their expression of this link between language and core cognitive processes. Here, we capitalize on this tightly-timed developmental shift in fullterm infants to assess (a) whether it also appears in preterm infants and (b) whether it reflects infants' maturational status or the duration of their postnatal experience. Healthy late preterm infants (N = 22) participated in an object categorization task while listening to language. Their performance, coupled with that of fullterm infants, reveals that this developmental shift is evident in preterm infants and unfolds on the same maturational timetable as in their fullterm counterparts.

Recent evidence from very young human infants' responses to human and nonhuman primate vocalizations offers new insights - and brings new questions - to the forefront for those who seek to integrate primate-general and human-specific mechanisms of acoustic communication with theories of language acquisition.