Generative models have the potential to transform the way we emulate Earth’s changing climate. Previous generative approaches rely on weather-scale autoregression for climate emulation, but this is inherently slow for long climate horizons and has yet to demonstrate stable rollouts under nonstationary forcings. Here, we introduce Spatiotemporal Pyramid Flows (SPF), a new class of flow matching approaches that model data hierarchically across spatial and temporal scales. Inspired by cascaded video models, SPF partitions the generative trajectory into a spatiotemporal pyramid, progressively increasing spatial resolution to reduce computation and coupling each stage with an associated timescale to enable direct sampling at any temporal level in the pyramid. This design, together with conditioning each stage on prescribed physical forcings (e.g., greenhouse gases or aerosols), enables efficient, parallel climate emulation at multiple timescales. On ClimateBench, SPF outperforms strong flow matching baselines and pre-trained models at yearly and monthly timescales while offering fast sampling, especially at coarser temporal levels. To scale SPF, we curate ClimateSuite, the largest collection of Earth system simulations to date, comprising over 33,000 simulation-years across ten climate models and the first dataset to include simulations of climate interventions. We find that the scaled SPF model demonstrates good generalization to held-out scenarios across climate models. Together, SPF and ClimateSuite provide a foundation for accurate, efficient, probabilistic climate emulation across temporal scales and realistic future scenarios. Data and code is publicly available at [anonymized for review].

Large vision and language assistants have enabled new capabilities for interpreting natural images. These approaches have recently been adapted to earth observation data, but they are only able to handle single image inputs, limiting their use for many real-world tasks. In this work, we develop a new vision and language assistant called TEOChat that can engage in conversations about temporal sequences of earth observation data. To train TEOChat, we curate an instruction-following dataset composed of many single image and temporal tasks including building change and damage assessment, semantic change detection, and temporal scene classification. We show that TEOChat can perform a wide variety of spatial and temporal reasoning tasks, substantially outperforming previous vision and language assistants, and even achieving comparable or better performance than several specialist models trained to perform specific tasks. Furthermore, TEOChat achieves impressive zero-shot performance on a change detection and change question answering dataset, outperforms GPT-4o and Gemini 1.5 Pro on multiple temporal tasks, and exhibits stronger single image capabilities than a comparable single image instruction-following model on scene classification, visual question answering, and captioning. We publicly release our data, models, and code at https://github.com/ermongroup/TEOChat .

A comprehensive, open-access database of oil and gas infrastructure locations is necessary for accurately attributing emissions from satellites and managing pollution impacts on surrounding communities. However, open-access datasets are limited for many infrastructure types, including natural gas compressor stations, which account for approximately one-third of U.S. oil and gas sector methane emissions and are associated with harmful pollution. Here, we developed the first automated deep learning approach for detecting natural gas compressor stations in satellite imagery. We experimented with various neural network architectures trained on different image resolutions and footprints, and found that the best model achieved a precision of 0.81 at 0.95 recall. Incorporating whether a proposed facility is close to an oil and gas pipeline further improved model precision by 0.02. Deploying the best model to identify facilities across a critical 200,000 km2 oil and gas-producing region capturing the Marcellus Shale, we detected 1103 compressor stations that were not previously reported in a large bottom-up oil and gas infrastructure database. Incorporating these new locations revealed that population exposure to potential emitted pollutants may be underestimated by as much as 74 % when relying exclusively on reported data. Our work highlights the utility of machine learning to enhance infrastructure mapping for environmental management and pollution assessment.

Methane emissions from the oil and gas sector are a large contributor to climate change. Robust emission quantification and source attribution are needed for mitigating methane emissions, requiring a transparent, comprehensive, and accurate geospatial database of oil and gas infrastructure. Realizing such a database is hindered by data gaps nationally and globally. To fill these gaps, we present a deep learning approach on freely available, high-resolution satellite imagery for automatically mapping well pads and storage tanks. We validate the results in the Permian and Denver-Julesburg basins, two high-producing basins in the United States. Our approach achieves high performance on expert-curated datasets of well pads (Precision = 0.955, Recall = 0.904) and storage tanks (Precision = 0.962, Recall = 0.968). When deployed across the entire basins, the approach captures a majority of well pads in existing datasets (79.5%) and detects a substantial number (>70,000) of well pads not present in those datasets. Furthermore, we detect storage tanks (>169,000) on well pads, which were not mapped in existing datasets. We identify remaining challenges with the approach, which, when solved, should enable a globally scalable and public framework for mapping well pads, storage tanks, and other oil and gas infrastructure.

Large language models are well-known to be effective at few-shot in-context learning (ICL). Recent advancements in multimodal foundation models have enabled unprecedentedly long context windows, presenting an opportunity to explore their capability to perform ICL with many more demonstrating examples. In this work, we evaluate the performance of multimodal foundation models scaling from few-shot to many-shot ICL. We benchmark GPT-4o and Gemini 1.5 Pro across 10 datasets spanning multiple domains (natural imagery, medical imagery, remote sensing, and molecular imagery) and tasks (multi-class, multi-label, and fine-grained classification). We observe that many-shot ICL, including up to almost 2,000 multimodal demonstrating examples, leads to substantial improvements compared to few-shot (<100 examples) ICL across all of the datasets. Further, Gemini 1.5 Pro performance continues to improve log-linearly up to the maximum number of tested examples on many datasets. Given the high inference costs associated with the long prompts required for many-shot ICL, we also explore the impact of batching multiple queries in a single API call. We show that batching up to 50 queries can lead to performance improvements under zero-shot and many-shot ICL, with substantial gains in the zero-shot setting on multiple datasets, while drastically reducing per-query cost and latency. Finally, we measure ICL data efficiency of the models, or the rate at which the models learn from more demonstrating examples. We find that while GPT-4o and Gemini 1.5 Pro achieve similar zero-shot performance across the datasets, Gemini 1.5 Pro exhibits higher ICL data efficiency than GPT-4o on most datasets. Our results suggest that many-shot ICL could enable users to efficiently adapt multimodal foundation models to new applications and domains. Our codebase is publicly available at https://github.com/stanfordmlgroup/ManyICL .

Low-yield repetitive laboratory diagnostics burden patients and inflate cost of care. In this study, we assess whether stability in repeated laboratory diagnostic measurements is predictable with uncertainty estimates using electronic health record data available before the diagnostic is ordered. We use probabilistic regression to predict a distribution of plausible values, allowing use-time customization for various definitions of "stability" given dynamic ranges and clinical scenarios. After converting distributions into "stability" scores, the models achieve a sensitivity of 29% for white blood cells, 60% for hemoglobin, 100% for platelets, 54% for potassium, 99% for albumin and 35% for creatinine for predicting stability at 90% precision, suggesting those fractions of repetitive tests could be reduced with low risk of missing important changes. The findings demonstrate the feasibility of using electronic health record data to identify low-yield repetitive tests and offer personalized guidance for better usage of testing while ensuring high quality care.

Vehicle damage localization and severity estimation is essential to post-accident assessments, with a traditional process taking an average of seven days and requiring substantial work from both customers and dealers. Towards improving this process, we propose an end-to-end system which inputs a set of user-acquired photographs of a vehicle after an accident and outputs a damage assessment report including the set of damaged parts and the type and size of the damage for each part. The system is composed of three deep learning modules: a model to identify whether a vehicle is present in the image, a model to localize the vehicle parts in the image, and a model to localize the damage in the image. We demonstrate the effectiveness of each module by evaluating them on labeled datasets containing images of vehicles after an accident, some collected by the OE (Original Equipment) Insured Fleet and some acquired by users of the OEM (Original Equipment Manufacturer) mobile application. We also describe how the modules fit together with a post-processing step to aggregate outputs between the different modules across multiple user-acquired views of the accident. Our approach demonstrates the potential for an accurate and automated vehicle damage estimation system to support a substantially more efficient vehicle damage assessment process.

Lack of diagnosis coding is a barrier to leveraging veterinary notes for medical and public health research. Previous work is limited to develop specialized rule-based or customized supervised learning models to predict diagnosis coding, which is tedious and not easily transferable. In this work, we show that open-source large language models (LLMs) pretrained on general corpus can achieve reasonable performance in a zero-shot setting. Alpaca-7B can achieve a zero-shot F1 of 0.538 on CSU test data and 0.389 on PP test data, two standard benchmarks for coding from veterinary notes. Furthermore, with appropriate fine-tuning, the performance of LLMs can be substantially boosted, exceeding those of strong state-of-the-art supervised models. VetLLM, which is fine-tuned on Alpaca-7B using just 5000 veterinary notes, can achieve a F1 of 0.747 on CSU test data and 0.637 on PP test data. It is of note that our fine-tuning is data-efficient: using 200 notes can outperform supervised models trained with more than 100,000 notes. The findings demonstrate the great potential of leveraging LLMs for language processing tasks in medicine, and we advocate this new paradigm for processing clinical text.

Deforestation is a leading contributor to greenhouse gas emissions globally. Understanding the direct drivers of forest loss is essential for developing targeted forest conservation and management policies. However, this data is hard to collect at scale due to the complexity of forest loss drivers and expertise required for accurately identifying them. To address this challenge, we developed a deep learning model called ForestNet which uses publicly available satellite imagery to automatically classify the drivers of primary forest loss. We validated ForestNet on a test set of expert-annotated forest loss events and showed that ForestNet achieved high performance across four major driver classes. We used ForestNet to identify these drivers on over 2 million forest loss events in Indonesia between 2012 and 2019, with significant improvement in spatial and temporal resolution over previously available data. We found that plantations and smallholder agriculture were the primary direct drivers of deforestation in Indonesia during this period, accounting for 64 % of total forest loss. Deforestation has decreased steadily since 2012 after increasing steadily from 2001 to 2009 and peaking from 2009 to 2012, trends that we found are primarily due to changes in plantation-driven deforestation. Our approach can serve as a general framework for scalably attributing deforestation to specific drivers and can be extended to other regions of interest, providing a flexible and cost-effective way for countries to regularly monitor, understand, and address their unique and dynamic drivers of deforestation.

Clouds play a significant role in global temperature regulation through their effect on planetary albedo. Anthropogenic emissions of aerosols can alter the albedo of clouds, but the extent of this effect, and its consequent impact on temperature change, remains uncertain. Human-induced clouds caused by ship aerosol emissions, commonly referred to as ship tracks, provide visible manifestations of this effect distinct from adjacent cloud regions and therefore serve as a useful sandbox to study human-induced clouds. However, the lack of large-scale ship track data makes it difficult to deduce their general effects on cloud formation. Towards developing automated approaches to localize ship tracks at scale, we present CloudTracks, a dataset containing 3,560 satellite images labeled with more than 12,000 ship track instance annotations. We train semantic segmentation and instance segmentation model baselines on our dataset and find that our best model substantially outperforms previous state-of-the-art for ship track localization (61.29 vs. 48.65 IoU). We also find that the best instance segmentation model is able to identify the number of ship tracks in each image more accurately than the previous state-of-the-art (1.64 vs. 4.99 MAE). However, we identify cases where the best model struggles to accurately localize and count ship tracks, so we believe CloudTracks will stimulate novel machine learning approaches to better detect elongated and overlapping features in satellite images. We release our dataset openly at zenodo.org/records/10042922.

GEO-Bench: m-forestnet dataset This dataset has been modified to be included in the GEO-Bench dataset. All changes with respect to the original version are documented at https://github.com/ServiceNow/geo-bench. The original version of this dataset is due to Irvin et al. (2020) and is available at: https://stanfordmlgroup.github.io/projects/forestnet/. See the LICENSE file provided alongside this dataset for applicable licensing information.

Large, self-supervised vision models have led to substantial advancements for automatically interpreting natural images. Recent works have begun tailoring these methods to remote sensing data which has rich structure with multi-sensor, multi-spectral, and temporal information providing massive amounts of self-labeled data that can be used for self-supervised pre-training. In this work, we develop a new encoder architecture called USat that can input multi-spectral data from multiple sensors for self-supervised pre-training. USat is a vision transformer with modified patch projection layers and positional encodings to model spectral bands with varying spatial scales from multiple sensors. We integrate USat into a Masked Autoencoder (MAE) self-supervised pre-training procedure and find that a pre-trained USat outperforms state-of-the-art self-supervised MAE models trained on remote sensing data on multiple remote sensing benchmark datasets (up to 8 to improvements in low data regimes (up to 7 are available at https://github.com/stanfordmlgroup/USat .

Major advancements in computer vision can primarily be attributed to the use of labeled datasets. However, acquiring labels for datasets often results in errors which can harm model performance. Recent works have proposed methods to automatically identify mislabeled images, but developing strategies to effectively implement them in real world datasets has been sparsely explored. Towards improved data-centric methods for cleaning real world vision datasets, we first conduct more than 200 experiments carefully benchmarking recently developed automated mislabel detection methods on multiple datasets under a variety of synthetic and real noise settings with varying noise levels. We compare these methods to a Simple and Efficient Mislabel Detector (SEMD) that we craft, and find that SEMD performs similarly to or outperforms prior mislabel detection approaches. We then apply SEMD to multiple real world computer vision datasets and test how dataset size, mislabel removal strategy, and mislabel removal amount further affect model performance after retraining on the cleaned data. With careful design of the approach, we find that mislabel removal leads per-class performance improvements of up to 8% of a retrained classifier in smaller data regimes.

GEO-Bench: m-SA-crop-type dataset This dataset has been modified to be included in the GEO-Bench dataset. All changes with respect to the original version are documented at https://github.com/ServiceNow/geo-bench. The original version of this dataset is due to the Western Cape Department of Agriculture and the Radiant Earth Foundation (2021) and is available at: https://mlhub.earth/data/ref_south_africa_crops_competition_v1. See the LICENSE file provided alongside this dataset for applicable licensing information.

Self-supervised learning (SSL) enables label efficient training for machine learning models. This is essential for domains such as medical imaging, where labels are costly and time-consuming to curate. However, the most effective supervised or SSL strategy for transferring models to different healthcare systems or novel tasks is not well understood. In this work, we systematically experiment with a variety of supervised and self-supervised pretraining strategies using multimodal datasets of medical images (chest X-rays) and text (radiology reports). We then evaluate their performance on data from two external institutions with diverse sets of tasks. In addition, we experiment with different transfer learning strategies to effectively adapt these pretrained models to new tasks and healthcare systems. Our empirical results suggest that multimodal SSL gives substantial gains over unimodal SSL in performance across new healthcare systems and tasks, comparable to models pretrained with full supervision. We demonstrate additional performance gains with models further adapted to the new dataset and task, using multimodal domain-adaptive pretraining (DAPT), linear probing then finetuning (LP-FT), and both methods combined. We offer suggestions for alternative models to use in scenarios where not all of these additions are feasible. Our results provide guidance for improving the generalization of medical image interpretation models to new healthcare systems and novel tasks.

GEO-Bench: m-brick-kiln dataset This dataset has been modified to be included in the GEO-Bench dataset. All changes with respect to the original version are documented at https://github.com/ServiceNow/geo-bench. The original version of this dataset is due to Lee et al. (2021) and is available at: https://sustainlab-group.github.io/sustainbench/docs/datasets/sdg13/brick_kiln.html. See the LICENSE file provided alongside this dataset for applicable licensing information.

Although rice cultivation is one of the most important agricultural sources of methane and contributes ~8 % of total global anthropogenic emissions, large discrepancies remain among estimates of global methane emissions from rice cultivation due to a lack of observational constraints. The spatial distribution of paddy-rice emissions has been assessed at regional-to-global scales by bottom-up inventories and land surface models over coarse spatial resolution (e.g., > 0.5 degrees) or spatial units (e.g., agro-ecological zones). However, high-resolution CH4 flux estimates capable of capturing the effects of local climate and management practices on emissions, as well as replicating in situ data, remain challenging to produce because of the scarcity of high-resolution maps of paddy-rice and insufficient understanding of CH4 predictors. Here, we combined paddy-rice methane-flux data from 23 global eddy covariance sites and MODIS remote sensing data with machine learning, and produced gridded up-scaling estimates of rice methane emissions at 5000-m resolution at 8-day intervals across Monsoon Asia, where ~87% of global rice area is cultivated and ~90% of global rice production occurs.

Recent progress in self-supervision has shown that pre-training large neural networks on vast amounts of unsupervised data can lead to substantial increases in generalization to downstream tasks. Such models, recently coined foundation models, have been transformational to the field of natural language processing. Variants have also been proposed for image data, but their applicability to remote sensing tasks is limited. To stimulate the development of foundation models for Earth monitoring, we propose a benchmark comprised of six classification and six segmentation tasks, which were carefully curated and adapted to be both relevant to the field and well-suited for model evaluation. We accompany this benchmark with a robust methodology for evaluating models and reporting aggregated results to enable a reliable assessment of progress. Finally, we report results for 20 baselines to gain information about the performance of existing models. We believe that this benchmark will be a driver of progress across a variety of Earth monitoring tasks.

GEO-Bench: m-nz-cattle dataset This dataset has been modified to be included in the GEO-Bench dataset. All changes with respect to the original version are documented at https://github.com/ServiceNow/geo-bench. The original version of this dataset is due to Abuaiadah et al. (2022) and is available at: https://zenodo.org/record/5908869. See the LICENSE file provided alongside this dataset for applicable licensing information.