We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.

The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced from The Stack, a large collection of permissively licensed GitHub repositories with inspection tools and an opt-out process. We fine-tuned StarCoderBase on 35B Python tokens, resulting in the creation of StarCoder. We perform the most comprehensive evaluation of Code LLMs to date and show that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model. Furthermore, StarCoder outperforms every model that is fine-tuned on Python and still retains its performance on other programming languages. We take several important steps towards a safe open-access model release, including an improved PII redaction pipeline and a novel attribution tracing tool, and make the StarCoder models publicly available under a more commercially viable version of the Open Responsible AI Model license.

We introduce Mistral 7B v0.1, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms Llama 2 13B across all evaluated benchmarks, and Llama 1 34B in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences of arbitrary length with a reduced inference cost. We also provide a model fine-tuned to follow instructions, Mistral 7B -- Instruct, that surpasses the Llama 2 13B -- Chat model both on human and automated benchmarks. Our models are released under the Apache 2.0 license.

Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages that appear only in the pretraining corpus. Finetuning on multilingual tasks with English prompts further improves performance on English and non-English tasks leading to various state-of-the-art zero-shot results. We also investigate finetuning on multilingual tasks with prompts that have been machine-translated from English to match the language of each dataset. We find training on these machine-translated prompts leads to better performance on human-written prompts in the respective languages. Surprisingly, we find models are capable of zero-shot generalization to tasks in languages they have never intentionally seen. We conjecture that the models are learning higher-level capabilities that are both task- and language-agnostic. In addition, we introduce xP3, a composite of supervised datasets in 46 languages with English and machine-translated prompts. Our code, datasets and models are freely available at https://github.com/bigscience-workshop/xmtf.

Large language models (LLMs) excel in many tasks in NLP and beyond, but most open models have very limited coverage of smaller languages and LLM work tends to focus on languages where nearly unlimited data is available for pretraining. In this work, we study the challenges of creating LLMs for Finnish, a language spoken by less than 0.1% of the world population. We compile an extensive dataset of Finnish combining web crawls, news, social media and eBooks. We pursue two approaches to pretrain models: 1) we train seven monolingual models from scratch (186M to 13B parameters) dubbed FinGPT, 2) we continue the pretraining of the multilingual BLOOM model on a mix of its original training data and Finnish, resulting in a 176 billion parameter model we call BLUUMI. For model evaluation, we introduce FIN-bench, a version of BIG-bench with Finnish tasks. We also assess other model qualities such as toxicity and bias. Our models and tools are openly available at https://turkunlp.org/gpt3-finnish.

The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced from The Stack, a large collection of permissively licensed GitHub repositories with inspection tools and an opt-out process. We fine-tuned StarCoderBase on 35B Python tokens, resulting in the creation of StarCoder. We perform the most comprehensive evaluation of Code LLMs to date and show that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model. Furthermore, StarCoder outperforms every model that is fine-tuned on Python, can be prompted to achieve 40\% pass@1 on HumanEval, and still retains its performance on other programming languages. We take several important steps towards a safe open-access model release, including an improved PII redaction pipeline and a novel attribution tracing tool, and make the StarCoder models publicly available under a more commercially viable version of the Open Responsible AI Model license.

Large multimodal models trained on natural documents, which interleave images and text, outperform models trained on image-text pairs on various multimodal benchmarks. However, the datasets used to train these models have not been released, and the collection process has not been fully specified. We introduce the OBELICS dataset, an open web-scale filtered dataset of interleaved image-text documents comprising 141 million web pages extracted from Common Crawl, 353 million associated images, and 115 billion text tokens. We describe the dataset creation process, present comprehensive filtering rules, and provide an analysis of the dataset's content. To show the viability of OBELICS, we train vision and language models of 9 and 80 billion parameters named IDEFICS, and obtain competitive performance on different multimodal benchmarks. We release our dataset, models and code.

As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learning in language models' pretraining (Radford et al., 2019). Can zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we develop a system for easily mapping any natural language tasks into a human-readable prompted form. We convert a large set of supervised datasets, each with multiple prompts with diverse wording. These prompted datasets allow for benchmarking the ability of a model to perform completely held-out tasks. We fine-tune a pretrained encoder-decoder model (Raffel et al., 2020; Lester et al., 2021) on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models up to 16x its size. Further, our approach attains strong performance on a subset of tasks from the BIG-bench benchmark, outperforming models up to 6x its size. All trained models are available at https://github.com/bigscience-workshop/t-zero and all prompts are available at https://github.com/bigscience-workshop/promptsource.

The crystallization of modeling methods around the Transformer architecture has been a boon for practitioners. Simple, well-motivated architectural variations can transfer across tasks and scale, increasing the impact of modeling research. However, with the emergence of state-of-the-art 100B+ parameters models, large language models are increasingly expensive to accurately design and train. Notably, it can be difficult to evaluate how modeling decisions may impact emergent capabilities, given that these capabilities arise mainly from sheer scale alone. In the process of building BLOOM--the Big Science Large Open-science Open-access Multilingual language model--our goal is to identify an architecture and training setup that makes the best use of our 1,000,000 A100-GPU-hours budget. Specifically, we perform an ablation study at the billion-parameter scale comparing different modeling practices and their impact on zero-shot generalization. In addition, we study the impact of various popular pre-training corpora on zero-shot generalization. We also study the performance of a multilingual model and how it compares to the English-only one. Finally, we consider the scaling behaviour of Transformers to choose the target model size, shape, and training setup. All our models and code are open-sourced at https://huggingface.co/bigscience .

Large pretrained Transformer language models have been shown to exhibit zero-shot generalization, i.e. they can perform a wide variety of tasks that they were not explicitly trained on. However, the architectures and pretraining objectives used across state-of-the-art models differ significantly, and there has been limited systematic comparison of these factors. In this work, we present a large-scale evaluation of modeling choices and their impact on zero-shot generalization. In particular, we focus on text-to-text models and experiment with three model architectures (causal/non-causal decoder-only and encoder-decoder), trained with two different pretraining objectives (autoregressive and masked language modeling), and evaluated with and without multitask prompted finetuning. We train models with over 5 billion parameters for more than 170 billion tokens, thereby increasing the likelihood that our conclusions will transfer to even larger scales. Our experiments show that causal decoder-only models trained on an autoregressive language modeling objective exhibit the strongest zero-shot generalization after purely unsupervised pretraining. However, models with non-causal visibility on their input trained with a masked language modeling objective followed by multitask finetuning perform the best among our experiments. We therefore consider the adaptation of pretrained models across architectures and objectives. We find that pretrained non-causal decoder models can be adapted into performant generative causal decoder models, using autoregressive language modeling as a downstream task. Furthermore, we find that pretrained causal decoder models can be efficiently adapted into non-causal decoder models, ultimately achieving competitive performance after multitask finetuning. Code and checkpoints are available at https://github.com/bigscience-workshop/architecture-objective.

Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License.

Aircraft accident investigation has played a pivotal role in improving the safety of aviation. Advances in recorder technology, specifically Cockpit Voice Recorders (CVRs) and Flight Data Recorders (FDR) have made a huge contribution to the understanding of occurrences for accident investigators. However, even these recorders have limitations such as the evidence they provide about pilots' situation awareness or behaviours. Supplementing audio and data recordings with video has been discussed for many years and whilst there continues to be debate among regulators, operators, manufacturers and pilot unions, the International Civil Aviation Organization (ICAO) has mandated Airborne Image Recorders (AIR) from 2023. The purpose of installing such systems is to provide evidence of crew operational behaviours in terms of both human-human and human-computer interactions (HCI) on the flight deck. Video alone is unlikely to provide sufficient evidence for investigators. This study examines the additional value that eye-tracking technology may provide through the case study of an accident involving an Airbus A330-300 aircraft which experienced a rejected landing. Currently, the investigation of such events, where crew interaction with automation is critical to their situation awareness, relies heavily on interview data. Such data may be unavailable (in the case of serious injury) or unreliable (based on hindsight bias). By integrating eye tracking technology into an AIR, accident investigators will potentially gain a better understanding of pilots’ visual scan patterns across flight deck instrumentation. This has implications for flight deck and procedural design as well as training and simulation.

Inflammation has a role in prostate tumorigenesis. Recruitment of inflammatorymonocytes to the tumor site ismediated by C-C chemokine ligand 2 (CCL2) through binding to its receptor CCR2.We hypothesized that androgen could modulate CCL2 expression in hormone-responsive prostate cancer cells and thereby promote recruitment of monocytes. Given the inhibitory effect of broccoli-derived compounds indole-3carbinol (I3C) and 3,30-diindolylmethane (DIM) on androgen-dependent pathways, we also reasoned that I3C andDIM couldmodulate the effect of androgen onCCL2-mediated pathways. Dihydrotestosteronewas found to induce a time-dependent (0–72 hours) and concentration-dependent (0–1 nmol/L) increase in CCL2 mRNA levels in androgen-responsive human prostate cancer cells (LNCaP). This increase in CCL2 mRNA corresponded with increased secretion of CCL2 protein. The effect of dihydrotestosterone was mediated through an androgen receptor (AR)-dependent pathway as small inhibitor RNA against AR negated the induction of CCL2. Although dihydrotestosterone also induced TWIST1 mRNA, an epithelial– mesenchymal transition–related factor, and purported inducer of CCL2, blocking its expression with small inhibitor RNAdidnot inhibit dihydrotestosterone inductionofCCL2mRNA.Moreover, conditionedmedia fromandrogen-treated cells promotedhumanmonocyte THP-1 cellmigration and this effectwas blockedby antibody against CCL-2. Both I3C and DIM inhibited promotional effects of dihydrotestosterone on CCL2 and migration. These results show that androgen may regulate CCL2 and promote inflammatory microenvironment in prostate tumors and that this process can be blocked by broccoli-derived compounds. Cancer Prev Res; 6(6); 519–29. 2013 AACR.

Culture is at the root of action; it underlies the manner by which people communicate and develop attitudes towards life. This research examined statistical differences in the 18 categories of Human factors Analysis and Classification System (HFACS, Shappell & Wiegmann, 2003) across 523 aviation accidents in the Republic of China (a collective culture) and 119 aviation accidents in the USA (an individual culture) . The result suggests that the culture of individualism seems to be superior for promoting aviation safety compared to collectivist cultures, however, factors such as the design of the aircraft, the management procedures and the nature of safety regulation all have a strong Western influence from the individualist culture. All of these factors are culturally congruent with the USA. It is essential to identify the potential causal roots for these differences from the underlying factors in these aviation mishaps, and identify what kind of factors drive people to act or react to dynamic situations that either lead to an accident help to develop an effective accident prevention strategy.

This study examined the mechanisms by which the prostate cancer chemopreventive agent genistein modulates gene expression in LNCaP human prostate cancer cells. Expression of androgen‐ and estrogen‐regulated genes was measured in LNCaP cells cultured in the presence or absence of hormonal stimulation and the presence or absence of genistein. Genistein strongly suppressed basal expression of androgen‐responsive gene (ARG) mRNAs, including prostate‐specific antigen (PSA) and Ste20‐related proline‐alanine‐rich kinase (SPAK). However, genistein had little or no effect on basal expression of two other ARGs, β 2 ‐microglobulin (B2M) or selenoprotein P (SEPP1). Culturing LNCaP cells in the presence of the synthetic androgen R1881‐induced increases in PSA, SPAK, B2M, and SEPP1 mRNA levels. The R1881‐induced expression of these genes was uniformly blocked by genistein. For PSA and SPAK, genistein also blocked or downregulated 17β‐estradiol‐induced increases in mRNA expression. These results indicate that genistein selectively alters expression of ARG mRNAs in LNCaP cells through modulation of both androgen‐ and estrogen‐induced signaling pathways. Published 2005 Wiley‐Liss, Inc.