Expressing natural language descriptions of 001 structured facts or relations – data-to-text gen-002 eration – increases the accessibility of a di-003 verse range of structured knowledge reposito-004 ries. End-to-end neural models for this task 005 require a large training corpus of relations and 006 corresponding descriptions. While such re-007 sources are unrealistic for every domain, we do 008 not fully understand how well different data-to-009 text generation models can generalize to new re-010 lations. This work presents an analysis of data-011 to-text models for unseen relations based on 012 two pre-trained language models (PLMs): T5 013 and GPT-2. We consider different strategies, in-014 cluding few-shot learning, prompt-tuning, and 015 incorporating other domain knowledge (natural 016 language description of the unseen relations) to 017 identify effective strategies and remaining chal-018 lenges for improving performance of PLMs on 019 new relations. 020

The corpus of late antique and medieval He-001 brew texts is vast and represents a crucial lin-002 guistic and cultural bridge between Biblical and 003 modern Hebrew. Poetry is prominent in this 004 corpus and one of its main characteristics is the 005 frequent use of metaphors. Distinguishing figu-006 rative and literal language use is a major task 007 for scholars of the Humanities, especially in the 008 fields of literature, linguistics and hermeneu-009 tics. This paper presents a new, challenging 010 dataset of late antique and medieval Hebrew 011 poetry with expert annotations of metaphors, as 012 well as some baseline results, which we hope 013 will facilitate further research in this area. 1 014

Fake news with detrimental societal effects has 001 attracted extensive attention and research. De-002 spite early success, the state-of-the-art meth-003 ods fall short of considering the propagation 004 of news. News propagates at different times 005 through different mediums, including users, 006 comments, and sources, which form the news 007 propagation network. Moreover, the serious 008 problem of data hiding arises, which means 009 that fake news publishers disguise fake news 010 as real to confuse users by deleting comments 011 that refute the rumor or deleting the news itself 012 when it has been spread widely. Existing meth-013 ods do not consider the propagation of news 014 and fail to identify what matters in the process, 015 which leads to fake news hiding in the prop-016 agation network and escaping from detection. 017 Inspired by the propagation of news, we pro-018 pose a novel fake news detection framework 019 named TaHiD, which models the propagation 020 as a heterogeneous dynamic graph and contains 021 the propagation attention module to measure 022 the influence of different propagation. Exper-023 iments demonstrate that TaHiD extracts use-024 ful information from the news propagation net-025 work and outperforms state-of-the-art methods 026 on several benchmark datasets for fake news 027 detection. Additional studies also show that 028 TaHiD is capable of identifying fake news in 029 the case of data hiding. 030

Model efficiency is a critical aspect of developing and deploying machine learning models. Inference time and latency directly affect the user experience, and some applications have hard requirements. In addition to inference costs, model training also have direct financial and environmental impacts. Although there are numerous well-established metrics (cost indicators) for measuring model efficiency, researchers and practitioners often assume that these metrics are correlated with each other and report only few of them. In this paper, we thoroughly discuss common cost indicators, their advantages and disadvantages, and how they can contradict each other. We demonstrate how incomplete reporting of cost indicators can lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models. We further present suggestions to improve reporting of efficiency metrics.

There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come with an associated computational cost which have both financial and/or environmental impact. The goal of this paper is to present scaling insights from pretraining and finetuning Transformers. While Kaplan et al. presents a comprehensive study of the scaling behaviour of Transformer language models, the scope is only on the upstream (pretraining) loss. Therefore, it is still unclear if these set of findings transfer to downstream task within the context of the pretrain-finetune paradigm. The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient. To this end, we present improved scaling protocols whereby our redesigned models achieve similar downstream fine-tuning quality while having 50\% fewer parameters and training 40\% faster compared to the widely adopted T5-base model. We publicly release over 100 pretrained checkpoints of different T5 configurations to facilitate future research and analysis.

Knowledge transfer between neural language 001 models is a widely used technique that has 002 proven to improve performance in a multitude 003 of natural language tasks. In recent years, high 004 cross-lingual transfer has been shown to oc-005 cur in multilingual language models. Hence, 006 it is of great importance to better understand 007 this phenomenon as well as its limits. While 008 most studies focus on training on independent 009 and identically distributed ( i.e. i.i.d. ) samples, 010 in this paper we study cross-lingual transfer 011 in continual slot filling for natural language 012 understanding. We investigate this by train-013 ing multilingual BERT on one language at a 014 time in sequence from the MultiATIS++ cor-015 pus, that contains a total of 9 languages. Our 016 main findings are that forward transfer is re-017 tained although forgetting is still present, and 018 that lost performance can be recovered with as 019 little as a single training epoch. This may be 020 explained by a progressive shift of model pa-021 rameters towards a better multilingual initial-022 ization. We also find that commonly used met-023 rics might be insufficient to describe continual 024 learning performance. 025

Circular consensus sequencing with Pacific Biosciences (PacBio) technology generates long (10–25 kilobases), accurate ‘HiFi’ reads by combining serial observations of a DNA molecule into a consensus sequence. The standard approach to consensus generation, pbccs, uses a hidden Markov model. We introduce DeepConsensus, which uses an alignment-based loss to train a gap-aware transformer–encoder for sequence correction. Compared to pbccs, DeepConsensus reduces read errors by 42%. This increases the yield of PacBio HiFi reads at Q20 by 9%, at Q30 by 27% and at Q40 by 90%. With two SMRT Cells of HG003, reads from DeepConsensus improve hifiasm assembly contiguity (NG50 4.9 megabases (Mb) to 17.2 Mb), increase gene completeness (94% to 97%), reduce the false gene duplication rate (1.1% to 0.5%), improve assembly base accuracy (Q43 to Q45) and reduce variant-calling errors by 24%. DeepConsensus models could be trained to the general problem of analyzing the alignment of other types of sequences, such as unique molecular identifiers or genome assemblies.

We investigate the problem of generating utter-001 ances from pairs of images showing a before 002 and an after state of a change in a visual scene. 003 We present a transformer model with differ-004 ence attention heads that learns to attend to vi-005 sual changes in consecutive images via a differ-006 ence key. We test our approach in instruction 007 generation, change captioning and difference 008 spotting and compare these tasks in terms of 009 their linguistic phenomena and reasoning abil-010 ities. Our model outperforms the state-of-the-011 art for instruction generation on the BLOCKS 012 and difference spotting on the Spot-the-diff 013 dataset and generates accurate referential and 014 compositional spatial expressions. Finally, we 015 identify linguistic phenomena that pose chal-016 lenges for generation in changing scenes. 017

Effectively decoding semantic frames in task-001 oriented dialogue systems remains a challenge, 002 which typically includes intent detection and 003 slot filling. Although RNN-based neural mod-004 els show promising results by jointly learning 005 of these two tasks, dominant RNNs are pri-006 marily focusing on modeling sequential depen-007 dencies. Rich graph structure information hid-008 den in the dialogue context is seldomly ex-009 plored. In this paper, we propose a novel 010 Graph-to-Sequence model to tackle the spoken 011 language understanding problem by modeling 012 both temporal dependencies and structural in-013 formation in a conversation. We introduce a 014 new Graph Convolutional LSTM (GC-LSTM) 015 encoder to learn the semantics contained in 016 the dialogue dependency graph by incorporat-017 ing a powerful graph convolutional operator. 018 Our proposed GC-LSTM can not only capture 019 the spatio-temporal semantic features in a di-020 alogue, but also learn the co-occurrence rela-021 tionship between intent detection and slot fill-022 ing. Furthermore, a LSTM decoder is utilized 023 to perform final decoding of both slot filling 024 and intent detection, which mutually improves 025 both tasks through global optimization. Exper-026 iments on benchmark ATIS and Snips datasets 027 show that our model achieves state-of-the-art 028 performance and outperforms existing models. 029

Pacific BioScience (PacBio) circular consensus sequencing (CCS) generates long (10-25 kb), accurate “HiFi” reads by combining serial observations of a DNA molecule into a consensus sequence. The standard approach to consensus generation uses a hidden Markov model (pbccs). Here, we introduce DeepConsensus, which uses a unique alignment-based loss to train a gap-aware transformer-encoder (GATE) for sequence correction. Compared to pbccs, DeepConsensus reduces read errors in the same dataset by 42%. This increases the yield of PacBio HiFi reads at Q20 by 9%, at Q30 by 27%, and at Q40 by 90%. With two SMRT Cells of HG003, reads from DeepConsensus improve hifiasm assembly contiguity (NG50 4.9Mb to 17.2Mb), increase gene completeness (94% to 97%), reduce false gene duplication rate (1.1% to 0.5%), improve assembly base accuracy (Q43 to Q45), and also reduce variant calling errors by 24%. ### Competing Interest Statement GB, DEC, KS, TY, FLL, QB, MN, HY, AK, WA, JPV, AV, CYM, PCC, and AC are employees of Google LLC and own Alphabet stock as part of the standard compensation package. AMW, AT, and WJR are full-time employees and shareholders of Pacific Biosciences. This study was funded by Google LLC.

Dense retrieval has been shown to be effective for Open Domain Question Answering, surpassing sparse retrieval methods like BM25. One such model, REALM, (Guu et al., 2020) is an end-to-end dense retrieval system that uses MLM based pretraining for improved downstream QA performance. However, the current REALM setup uses limited resources and is not comparable in scale to more recent systems, contributing to its lower performance. Additionally, it relies on noisy supervision for retrieval during fine-tuning. We propose REALM++, where we improve upon the training and inference setups and introduce better supervision signal for improving performance, without any architectural changes. REALM++ achieves ~5.5% absolute accuracy gains over the baseline while being faster to train. It also matches the performance of large models which have 3x more parameters demonstrating the efficiency of our setup.

Self-attention has the promise of improving computer vision systems due to parameter-independent scaling of receptive fields and content-dependent interactions, in contrast to parameter-dependent scaling and content-independent interactions of convolutions. Self-attention models have recently been shown to have encouraging improvements on accuracy-parameter trade-offs compared to baseline convolutional models such as ResNet-50. In this work, we develop self-attention models that can outperform not just the canonical baseline models, but even the high-performing convolutional models. We propose two extensions to self-attention that, in conjunction with a more efficient implementation of self-attention, improve the speed, memory usage, and accuracy of these models. We leverage these improvements to develop a new self-attention model family, HaloNets, which reach state-of-the-art accuracies on the parameter-limited setting of the ImageNet classification benchmark. In preliminary transfer learning experiments, we find that HaloNet models outperform much larger models and have better inference performance. On harder tasks such as object detection and instance segmentation, our simple local self-attention and convolutional hybrids show improvements over very strong baselines. These results mark another step in demonstrating the efficacy of self-attention models on settings traditionally dominated by convolutions. 1

Transition-based approaches based on local classification are attractive for dependency parsing due to their simplicity and speed, despite producing results slightly below the state-of-the-art. In this paper, we propose a new approach for approximate structured inference for transition-based parsing that produces scores suitable for global scoring using local models. This is accomplished with the introduction of error states in local training, which add information about incorrect derivation paths typically left out completely in locally-trained models. Using neural networks for our local classifiers, our approach achieves 93.61% accuracy for transition-based dependency parsing in English.

We present BoTNet, a conceptually simple yet powerful backbone architecture that incorporates self-attention for multiple computer vision tasks including image classification, object detection and instance segmentation. By just replacing the spatial convolutions with global self-attention in the final three bottleneck blocks of a ResNet and no other changes, our approach improves upon the baselines significantly on instance segmentation and object detection while also reducing the parameters, with minimal overhead in latency. Through the design of BoTNet, we also point out how ResNet bottleneck blocks with self-attention can be viewed as Transformer blocks. Without any bells and whistles, BoTNet achieves 44.4% Mask AP and 49.7% Box AP on the COCO Instance Segmentation benchmark using the Mask R-CNN framework; surpassing the previous best published single model and single scale results of ResNeSt [67] evaluated on the COCO validation set. Finally, we present a simple adaptation of the BoTNet design for image classification, resulting in models that achieve a strong performance of 84.7% top-1 accuracy on the ImageNet benchmark while being up to 1.64x faster in "compute" 1 time than the popular EfficientNet models on TPU-v3 hardware. We hope our simple and effective approach will serve as a strong baseline for future research in self-attention models for vision. 2

Triangular machine translation is a special case 001 of low-resource machine translation where the 002 language pair of interest has limited parallel 003 data, but both languages have abundant par-004 allel data with a pivot language. Naturally, 005 the key to triangular machine translation is the 006 successful exploitation of such auxiliary data. 007 In this work, we propose a transfer-learning-008 based approach that utilizes all types of auxil-009 iary data. As we train auxiliary source-pivot 010 and pivot-target translation models, we initial-011 ize some parameters of the pivot side with a 012 pre-trained language model and freeze them 013 to encourage both translation models to work 014 in the same pivot language space, so that they 015 can be smoothly transferred to the source-target 016 translation model. Experiments show that our 017 approach can outperform previous ones

We propose Composition Sampling, a simple 001 but effective method to generate higher qual-002 ity diverse outputs for conditional generation 003 tasks, compared to previous stochastic decod-004 ing strategies. It builds on recently proposed 005 planning-based neural generation models that 006 are trained to first create a composition of the 007 output using an entity chain and then continue 008 to generate conditioned on the entity chain 009 and the input (Narayan et al., 2021). Our ap-010 proach avoids text degeneration by first sam-011 pling a composition in the form of an entity 012 chain and then using beam search to gener-013 ate the best possible text grounded to the en-014 tity chain. Experiments on CNN/DailyMail 015 and XSum using a variety of automatic metrics 016 and human-based evaluation demonstrate that 017 Composition Sampling is currently the best 018 available decoding strategy for generating di-019 verse meaningful summaries. We further out-020 perform state-of-the-art approaches for ques-021 tion generation in terms of BLEU. 022

The invention relates to attention-based image generation neural networks. Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output image. In one aspect, one of the methods includes generating the output image intensity value by intensity value according to a generation order of pixel-color channel pairs from the output image, comprising, for each particular generation order position in the generation order: generating a current output image representation of a current output image, processing the current output image representation using a decoder neural network to generate a probability distribution over possible intensity values for the pixel-color channel pair at the particular generation order position, wherein the decoder neural network includes one or more local masked self-attention sub-layers; and selecting an intensity value for the pixel-color channel pair at the particular generation order position using theprobability distribution.

Self-attention has recently been adopted for a wide range of sequence modeling problems. Despite its effectiveness, self-attention suffers from quadratic computation and memory requirements with respect to sequence length. Successful approaches to reduce this complexity focused on attending to local sliding windows or a small set of locations independent of content. Our work proposes to learn dynamic sparse attention patterns that avoid allocating computation and memory to attend to content unrelated to the query of interest. This work builds upon two lines of research: It combines the modeling flexibility of prior work on content-based sparse attention with the efficiency gains from approaches based on local, temporal sparse attention. Our model, the Routing Transformer, endows self-attention with a sparse routing module based on online k-means while reducing the overall complexity of attention to O( n1.5d) from O( n2d) for sequence length n and hidden dimension d. We show that our model outperforms comparable sparse attention models on language modeling on Wikitext-103 (15.8 vs 18.3 perplexity), as well as on image generation on ImageNet-64 (3.43 vs 3.44 bits/dim) while using fewer self-attention layers. Additionally, we set a new state-of-the-art on the newly released PG-19 data-set, obtaining a test perplexity of 33.2 with a 22 layer Routing Transformer model trained on sequences of length 8192. We open-source the code for Routing Transformer in Tensorflow.1

A system implemented as computer programs on one or more computers in one or more locations that implements a computer vision model is described. The computer vision model includes a positional local self-attention layer that is configured to receive an input feature map and to generate an output feature map. For each input element in the input feature map, the positional local self-attention layer generates a respective output element for the output feature map by generating a memory block including neighboring input elements around the input element, generates a query vector using the input element and a query weight matrix, for each neighboring element in the memory block, performs positional local self-attention operations to generate a temporary output element, and generates the respective output element by summing temporary output elements of the neighboring elements in the memory block.

Visual referring expression recognition is a challenging task that requires natural language understanding in the context of an image. We critically examine RefCOCOg, a standard benchmark for this task, using a human study and show that 83.7% of test instances do not require reasoning on linguistic structure, i.e., words are enough to identify the target object, the word order doesn’t matter. To measure the true progress of existing models, we split the test set into two sets, one which requires reasoning on linguistic structure and the other which doesn’t. Additionally, we create an out-of-distribution dataset Ref-Adv by asking crowdworkers to perturb in-domain examples such that the target object changes. Using these datasets, we empirically show that existing methods fail to exploit linguistic structure and are 12% to 23% lower in performance than the established progress for this task. We also propose two methods, one based on contrastive learning and the other based on multi-task learning, to increase the robustness of ViLBERT, the current state-ofthe-art model for this task. Our datasets are publicly available at https://github.com/ aws/aws-refcocog-adv.