Transformers are state-of-the-art deep learning models that are composed of stacked attention and point-wise, fully connected layers designed for handling sequential data. Transformers are not only ubiquitous throughout Natural Language Processing (NLP), but also have recently inspired a new wave of Computer Vision (CV) applications research. In this work, a Vision Transformer (ViT) is finetuned to predict the state variables of 2-dimensional Ising model simulations. Our experiments show that ViT outperforms state-of-the-art Convolutional Neural Networks (CNN) when using a small number of microstate images from the Ising model corresponding to various boundary conditions and temperatures. This work explores the possible of applications of ViT to other simulations and introduces interesting research directions on how attention maps can learn the underlying physics governing different phenomena.

As important events in textual data are usu-001 ally highly specific in terms of tasks and do-002 mains, a change in data distribution would have 003 a significant impact on detection performance. 004 Recent methods addressing unsupervised do-005 main adaptation for event detection task typ-006 ically extracted domain-invariant representa-007 tions through combining and balancing various 008 objectives to align the feature space between 009 source and target domains. While effective, 010 these methods are impractical as large-scale 011 language models are drastically growing bigger 012 to achieve optimal performance. To this end, 013 we propose Meta Self-Paced Domain Adaption 014 framework (MSP-DA) that effectively and effi-015 ciently alleviates the need for domain-specific 016 hyperparameter tuning. By imitating the train-017 test dataset split based on the difficulties of 018 source domain’s samples, the model is trained 019 through a meta-learning process that learns 020 to weigh the importance of each labeled in-021 stance and to balance every alignment objec-022 tive, simultaneously. Extensive experiments 023 demonstrate our framework substantially im-024 proves performance on target domains, surpass-025 ing state-of-the-art approaches. Furthermore, 026 we present detailed analyses to validate our 027 method and provide insight into how each do-028 main affects the learned hyperparameters. 029

The scale, variety, and quantity of publicly-available NLP datasets has grown rapidly as researchers propose new tasks, larger models, and novel benchmarks. Datasets is a community library for contemporary NLP designed to support this ecosystem. Datasets aims to standardize end-user interfaces, versioning, and documentation, while providing a lightweight front-end that behaves similarly for small datasets as for internet-scale corpora. The design of the library incorporates a distributed, community-driven approach to adding datasets and documenting usage. After a year of development, the library now includes more than 650 unique datasets, has more than 250 contributors, and has helped support a variety of novel cross-dataset research projects and shared tasks. The library is available at https://github.com/huggingface/datasets.

Current methods for few-shot fine-tuning of 001 pretrained masked language model (PLM) require 002 carefully engineered prompts and verbalizers 003 for each new task, to convert examples into a 004 cloze-format that the PLM can score. In this work, 005 we propose P ERFECT , a simple and efficient 006 method for few-shot fine-tuning of PLMs without 007 relying on any such handcrafting , which is highly 008 effective given as few as 32 data points. P ERFECT 009 makes two key design choices: First, we show 010 that manually engineered task prompts can be 011 replaced with task-specific adapters that enable 012 sample-efficient fine-tuning and reduce memory 013 and storage costs by roughly factors of 5 and 100, 014 respectively. Second, instead of using handcrafted 015 verbalizers, we learn a new multi-token label em-016 bedding during fine-tuning which are not tied to 017 the model vocabulary and which allow us to avoid 018 complex auto-regressive decoding. These embed-019 dings are not only learnable from limited data but 020 also enable nearly 100x faster training and infer-021 ence. Experiments on a wide range of few shot 022 NLP tasks demonstrate that P ERFECT , while be-023 ing simple and efficient, also outperforms existing 024 state-of-the-art few-shot learning methods. 1 025

A significant body of recent work has examined variational autoencoders as a powerful approach for tasks which involve modeling the distribution of complex data such as images and text. In this work, we present a framework for modeling multiple data sets which come from differing distributions but which share some common latent structure. By incorporating architectural constraints and using a mutual information regularized form of the variational objective, our method successfully models differing data populations while explicitly encouraging the isolation of the shared and private latent factors. This enables our model to learn useful shared structure across similar tasks and to disentangle cross-population representations in a weakly supervised way. We demonstrate the utility of our method on several applications including image denoising, sub-group discovery, and continual learning.

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. \textit{Transformers} is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered state-of-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. \textit{Transformers} is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at \url{https://github.com/huggingface/transformers}.

Inferring commonsense knowledge is a key challenge in natural language processing, but due to the sparsity of training data, previous work has shown that supervised methods for commonsense knowledge mining underperform when evaluated on novel data. In this work, we develop a method for generating commonsense knowledge using a large, pre-trained bidirectional language model. By transforming relational triples into masked sentences, we can use this model to rank a triple's validity by the estimated pointwise mutual information between the two entities. Since we do not update the weights of the bidirectional model, our approach is not biased by the coverage of any one commonsense knowledge base. Though this method performs worse on a test set than models explicitly trained on a corresponding training set, it outperforms these methods when mining commonsense knowledge from new sources, suggesting that unsupervised techniques may generalize better than current supervised approaches.

Unsupervised learning of latent representations is useful for a variety of tasks including dimensionality reduction, density estimation, and structure or sub-group discovery. Methods for recovering such representations typically rely on the assumption that the observed data is a manifestation of only a limited number of factors of variation [1, 2]. Variational autoencoders (VAE) [3], a combination of a non-linear latent variable model and an amortized inference scheme [4], is a popular method for recovering such latent structure. VAEs and its extensions have received considerable attention in recent years and have been shown useful for modeling text [5], images [6] , and other data exhibiting complex correlations [7]. However, barring a few notable exceptions [8], the vast majority of this line of work assumes that the data being modeled is independent and identically distributed.

In this work we detail a novel open source library, called MMLSpark, that combines the flexible deep learning library Cognitive Toolkit, with the distributed computing framework Apache Spark. To achieve this, we have contributed Java Language bindings to the Cognitive Toolkit, and added several new components to the Spark ecosystem. In addition, we also integrate the popular image processing library OpenCV with Spark, and present a tool for the automated generation of PySpark wrappers from any SparkML estimator and use this tool to expose all work to the PySpark ecosystem. Finally, we provide a large library of tools for working and developing within the Spark ecosystem. We apply this work to the automated classification of Snow Leopards from camera trap images, and provide an end to end solution for the non-profit conservation organization, the Snow Leopard Trust.