ML models often exhibit unexpectedly poor behavior when they are deployed in real-world domains. We identify underspecification as a key reason for these failures. An ML pipeline is underspecified when it can return many predictors with equivalently strong held-out performance in the training domain. Underspecification is common in modern ML pipelines, such as those based on deep learning. Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains. This ambiguity can lead to instability and poor model behavior in practice, and is a distinct failure mode from previously identified issues arising from structural mismatch between training and deployment domains. We show that this problem appears in a wide variety of practical ML pipelines, using examples from computer vision, medical imaging, natural language processing, clinical risk prediction based on electronic health records, and medical genomics. Our results show the need to explicitly account for underspecification in modeling pipelines that are intended for real-world deployment in any domain.

Self-supervised pretraining followed by supervised fine-tuning has seen success in image recognition, especially when labeled examples are scarce, but has received limited attention in medical image analysis. This paper studies the effectiveness of self-supervised learning as a pre-training strategy for medical image classification. We conduct experiments on two distinct tasks: dermatology condition classification from digital camera images and multi-label chest X-ray classification, and demonstrate that self-supervised learning on ImageNet, followed by additional self-supervised learning on unlabeled domain-specific medical images significantly improves the accuracy of medical image classifiers. We introduce a novel Multi-Instance Contrastive Learning (MICLe) method that uses multiple images of the underlying pathology per patient case, when available, to construct more informative positive pairs for self-supervised learning. Combining our contributions, we achieve an improvement of 6.7% in top-1 accuracy and an improvement of 1.1% in mean AUC on dermatology and chest X-ray classification respectively, outperforming strong supervised baselines pretrained on ImageNet. In addition, we show that big self-supervised models are robust to distribution shift and can learn efficiently with a small number of labeled medical images.

We observe that the development cross-001 entropy loss of supervised neural machine 002 translation models scales like a power law with 003 the amount of training data and the number of 004 non-embedding parameters in the model. We 005 discuss some practical implications of these re-006 sults, such as predicting BLEU achieved by 007 large scale models and predicting the ROI of 008 labeling data in low-resource language pairs. 009

Class imbalance is a common problem in medical diagnosis, causing a standard classifier to be biased towards the common classes and perform poorly on the rare classes. This is especially true for dermatology, a specialty with thousands of skin conditions but many of which have low prevalence in the real world. Motivated by recent advances, we explore few-shot learning methods as well as conventional class imbalance techniques for the skin condition recognition problem and propose an evaluation setup to fairly assess the real-world utility of such approaches. We find the performance of few-show learning methods does not reach that of conventional class imbalance techniques, but combining the two approaches using a novel ensemble improves model performance, especially for rare classes. We conclude that ensembling can be useful to address the class imbalance problem, yet progress can further be accelerated by real-world evaluation setups for benchmarking new methods.

A broad spectrum of metagenomic and single cell sequencing techniques have become popular for dissecting environmental microbial diversity, leading to the characterization of thousands of novel microbial lineages. In addition to recovering bacterial and archaeal genomes, metagenomic assembly can also produce genomes of viruses that infect microbial cells. Because of their diversity, lack of marker genes, and small genome size, identifying novel bacteriophage sequences from metagenomic data is often challenging, especially when the objective is to establish phage–host relationships. The present work describes a computational approach that uses supervised learning to classify metagenomic contigs as phage or non‐phage as well as assigning phage taxonomy based on tetranucleotide frequencies. Furthermore, the method assigns phage–host relationships using co‐occurrence statistics derived from a recently developed mini‐metagenomic experimental technique. This work evaluates method performance at identifying viral contigs and predicting taxonomic classification using publicly available references. Then, using two mini‐metagenomic datasets, over 100 novel phage contigs from hot spring samples of Yellowstone National Park are identified and assigned to putative microbial hosts. Results of this work demonstrate the value of combining viral sequence identification with mini‐metagenomic experimental methods to understand the microbial ecosystem.

Metagenomic sequencing approaches have become popular for the purpose of dissecting environmental microbial diversity, leading to the characterization of novel microbial lineages. In addition of bacterial and fungal genomes, metagenomic analysis can also reveal genomes of viruses that infect microbial cells. Because of their small genome size and limited knowledge of phage diversity, discovering novel phage sequences from metagenomic data is often challenging. Here we describe PhaMers (Phage k -Mers). a phage identification tool that uses supervised learning to classify metagenomic contigs as phage or non-phage on the basis of tetranucleotide frequencies. a technique that does not depend on existing gene annotations. PhaMers compares the tetranucleotide frequencies of metagenomic contigs to phage and bacteria references from online databases. resulting in assignments of lower level phage taxonomy based on sequence similarity. Using PhaMers. we identified 103 novel phage sequences from hot spring samples of Yellowstone National Park based on data generated from a microfluidic-based minimetagenomic approach. We analyzed assembled contigs over 5 kbp in length using PhaMers and compared the results with those generated by VirSorter, a publicly available phage identification and annotation package. We analyzed the performance of phage genome prediction and taxonomic classification using PhaMers. and presented putative hosts and taxa for some of the novel phage sequences. Finally. mini-metagenomic occurrence profiles of phage and prokaryotic genomes were used to verify putative hosts.