Cloud-native climate services increasingly expose analysis-ready Zarr archives through web APIs and Python/Xarray workflows. These layouts accelerate coordinateto-value subsetting, yet they are less efficient for value-tocoordinate workloads, including threshold-based extreme-event discovery and regional top-$k$ retrieval. In such queries, geometryoverlapping chunks are often read and decompressed before value predicates can be evaluated, causing unnecessary data access. This paper presents an external value-aware metadata layer that preserves the original archive while enabling pruning before data access. The proposed sidecar maintains lightweight chunk synopses, coarse upper bounds, and hot metadata caches to transform geometry-only candidates into value-aware candidates. ERA5 experiments show up to $5 \times$ fewer chunk reads and up to $6.3 \times$ lower median latency than a geometry-only Zarr+Xarray pipeline, demonstrating a practical direction for low-latency extreme-event services without migrating data into a dedicated array database.
Climate data services support many scientific workflows, but climate datasets are large and costly to store and deliver. Lossy compression can reduce data volume, but climate applications also require controlled error. Motivated by this requirement, we propose ChunkZ, a fast chunked climate data compression method using spatio-temporal predictors. First, we propose a chunk-based parallel pipeline to overlap input and output with compression, thereby improving end-to-end efficiency. Second, we introduce spatio-temporal predictors to improve compression ratios on climate variables that exhibit strong spatio-temporal variation patterns, such as wind, temperature, and humidity fields. Third, we propose a chunkwise predictor selection strategy to improve overall compression ratio with low tuning overhead. Last, we conduct comprehensive experiments on widely used reanalysis datasets at two spatial resolutions, and experimental results show that ChunkZ achieves the best compression throughput on all four physical variables with speedups of up to 7.3 times over runner-up baselines and the best compression ratio on all four variables with up to 65.3 percent improvement under the same error bound.
Research on tropical cyclones (TCs) requires the joint analysis of geostationary satellite imagery and meteorological reanalyses. However, these datasets are typically stored in heterogeneous formats across separate archives, making integrated retrieval and computation inefficient. As a result, researchers must manually download, parse, and coregister multiple sources, leading to tedious and error-prone workflows. To address this challenge, we propose TC-Zarr, an analysis-ready storage framework that fuses geostationary imagery and reanalysis fields into a unified multidimensional Zarr data cube indexed by time, latitude, longitude and variable. TC-Zarr employs a Guided, Mass-conserving Resolution Fusion (GMRF) algorithm to align high-resolution satellite imagery with lower-resolution reanalysis data on a common target grid. The aligned datasets are stored in the Zarr cube, enabling efficient retrieval and seamless joint analysis. In a case study of Super Typhoon Doksuri (2023), TC-Zarr integrates Himawari-8 imagery with ERA5 reanalyses and achieves significant improvements in data access efficiency compared with conventional file-based workflows.
Climate archives are expanding at the petabyte scale, while existing climate data formats and floating-point compressors deliver only limited compression efficiency. Recent neural image compression approaches achieve higher ratios but lack strict error guarantees, limiting their applicability in scientific domains. To address this gap, we propose EBCF (Error-Bounded Climate Flow), a discrete normalizing-flow–based compressor that enforces absolute reconstruction error bounds for numerical climate data. EBCF introduces three key innovations: (i) an error-bounded normalization algorithm that ensures quantization errors remain within user-specified tolerances; (ii) a precision-preserving discrete flow model that learns probability distributions from climate latents without multiplication/division error; and (iii) a cache-based entropy coding strategy for efficient end-to-end compression. Experiments on ERA5 reanalysis data demonstrate that EBCF reaches highest average compression ratio (4.27×) compared to both format-level compressors (1.38×) and state-of-the-art error-bounded methods (3.48×), while maintaining fidelity required for downstream climate applications.
Sea surface variable reconstruction from sparse observations is a key ocean-science challenge. Traditional methods, such as the four-dimensional variational (4DVar) approach, rely on numerical models for background information, leading to high computational costs. Deep learning methods are more efficient but often fail to capture eddy dynamics, resulting in limited effective resolution. We propose 4DVarGen, a 4DVar-inspired generative framework for reconstructing sea surface variable fields at eddy-resolving scales from sparse remote-sensing observations. 4DVarGen establishes a mathematical equivalence between 4DVar and an observation-guided denoising process. Its key innovation is injecting the observation-likelihood gradient into denoising iterations, driving the generated trajectories to evolve in the direction of minimizing the 4DVar objective function toward a maximum a posteriori solution. Spatiotemporal priors learned by a diffusion model serve as background information, reducing computational costs and mitigating the adverse effects of Gaussian assumptions. Experiments show that 4DVarGen effectively leverages the temporal evolution patterns of sea surface temperature (SST) and sea surface height (SSH), as well as their dynamical mappings learned by the diffusion model, leading to improved reconstruction accuracy and effective resolution. Our model, pretrained on GLORYS12V1 reanalysis data, generates sea surface variable fields guided by real observations, achieving accuracy and effective resolution improvements of 18% and 58%, respectively, compared to GLORYS12V1. This study offers a novel framework for reconstructing Earth system states from sparse observations.
Urban socio-semantic segmentation leverages digital and satellite imagery to provide critical spatial semantic information for downstream applications such as urban resource allocation. Although existing methods achieve high segmentation accuracy, they still suffer from inaccurate delineation of target boundaries. The underlying issue is that current models primarily rely on passively aggregated global cross-modal cues, lacking active exploration of the environment. To address this limitation, we propose the EVEREST model, which adopts an egocentric exploration strategy that enables the model to actively investigate boundary cues and perform self-correction. In addition, we formulate discrete natural-language prompts as pseudocode to regularize the execution logic. Reinforcement learning is further employed to implement this irreducible process and elicit the model's structured reasoning capability. Our EVEREST achieves optimal performance on all metrics in the real world urban socio-semantic dataset, demonstrating the superiority of our model. Codes are available at https://anonymous.4open.science/r/EVEREST-9D21/.
Similar weather retrieval asks a meteorological archive to return historical multi-day weather processes that are physically analogous to a query process. Here, similar weather is defined as the comprehensive similarity of time-elastically aligned multi-day processes in event-level evolution, multivariate physical-field trends, and numerical intensity; it is not singlevariable, instantaneous, or purely pixel-level statistical resemblance. We study a generative retrieval formulation in which a query process is mapped directly to discrete identifiers of relevant historical processes. The formulation reuses published physicsaligned weather fingerprints as compact daily tokens and organizes archive segments with hierarchical identifiers whose prefixes encode coarse-to-fine physical similarity. Identifier generation enables direct table lookup, while physics-aligned prefix learning and time-elastic supervision preserve meteorological meaning. Experiments on ERA5 weather-process segments demonstrate that, under both strict and time-elastic protocols, the proposed method achieves low latency while maintaining high similarity.
Ocean dynamics drive global climate patterns and extreme weather events, making accurate spatiotemporal forecasting essential for climate monitoring and marine operations. Traditional Global Ocean Forecasting Systems (GOFSs) offer high accuracy predictions, yet remain computationally expensive and fail to fully leverage growing historical data. Recent deep learning models have achieved notable success, but still face three fundamental challenges: (1) they homogenize ocean variables despite strong physical coupling via equation-of-state relationships; (2) they neglect spherical geometry, resulting in severe distortions at high latitudes; and (3) they struggle to model multi-scale temporal dynamics. We introduce PhyOceanCast, a physics-informed diffusion model that overcomes these limitations through two key innovations. First, the Spherical Graph Attention Network for Multi-scale Ocean Coupling (SGAN-MOC) preserves spherical topology while enabling cross-variable interactions via heterogeneous encoding and k-hop-constrained attention. Second, the Physics-Informed Wavelet Temporal Coherence (PWTC) module that decomposes ocean dynamics across multiple scales with advection-diffusion constraints. PhyOceanCast forecasts 145 ocean variables, including temperature, salinity, and velocity fields, across 36 depth levels plus sea surface height. Extensive experiments demonstrate superior performance over diffusion, transformer, and hybrid baselines, promising a new paradigm for global ocean canonical variable forecasting. Code is available at supplementary materials.
The increasing resolution of global ocean circulation models has made data output an important constraint on runtime efficiency and operational timeliness. The current dedicated-process asynchronous I/O scheme in the Mass Conservation Ocean Model (MaCOM) sends output data from compute processes to a group of reserved I/O processes. Although this design separates part of the writing work from the main time-stepping loop, it still introduces centralized data aggregation, additional I/O process management, and high memory pressure on the I/O side at large process counts. This paper presents MaCOM-PAIO, a PAIO-enabled asynchronous I/O optimization for MaCOM. Built on the existing PAIO/PAIOM asynchronous I/O stack, MaCOM-PAIO implements a thread-based asynchronous output path, adapts the PnetCDF execution path used by MaCOM to route selected collective writes to PAIO, and uses PAIOM asynchronous zones to submit history and restart output operations as background tasks. The implementation keeps the numerical solver unchanged and preserves the PnetCDF-style calling path at the application level, while replacing the dedicated I/O process path with I/O-thread-based asynchronous execution on the allocated HPC nodes. Experiments were conducted on a 1/12 degrees global MaCOM configuration. Strong-scaling tests show that, at 1646 compute processes, MaCOM-PAIO reduces the total runtime from 1167.45 s to 276.53 s and lowers the compute-side I/O blocking ratio from 67.2% to 4.9% under the tested configuration. In an independent bandwidth test at 1080 compute processes, the measured write bandwidth increases from approximately 0.10 GiB/s to 0.90 GiB/s for output volumes of about 82 GiB. The maximum memory footprint of the I/O entities is also reduced from approximately 18.2 GiB in the legacy dedicated-I/O scheme to approximately 1.9 GiB in MaCOM-PAIO. These results demonstrate that PAIO-based integration is a practical approach for improving MaCOM I/O performance under the evaluated hardware/software environment and workload.
Cloud-native metocean data cubes are increasingly served from object storage, but heterogeneous OGC API-EDR and OGC API-Tiles requests can trigger many small remote reads and unstable tail latency. We present a Zarr-native metocean service that implements the five core OGC API-EDR query endpoints together with OGC API-Tiles over two fully materialized sharded layouts stored as independent Zarr stores in MinIO. A QoS-aware semantic router maps each request to a storage-aware plan using the request template, estimated chunk/shard footprint, and runtime statistics. We further add a lightweight learning-augmented planner, implemented as an MLP-based AI-assisted plan selector, that chooses among six bounded layout/parallelism actions and is trained on 8,000 recorded prototype requests collected under controlled workload generation on the deployed service. We optimize a QoS-oriented objective centered on tail latency, but report mean, P50, and P95 latency for completeness. On a held-out replay set derived from these recorded prototype requests, the learned planner reduces mean latency from 774.6 ms to $722.7 \text{ms}(6.7 \%)$ relative to the rule-based heuristic, closes 86.8% of the gap to a bounded oracle, and adds negligible model-selection cost relative to endto-end request latency. These results show that standards-aligned semantic routing plus learning-augmented plan selection can improve QoS for metocean data access on sharded object storage.
Similar weather retrieval is a critical task in geoscientific information processing. Existing approaches often fail to meet the physical interpretability, spatiotemporal coupling, and scalability required in practice. We present Sim-Weather, a label-free retrieval framework for meteorological fields. The method learns physically aligned fingerprints whose Euclidean distances are statistically matched to field-space RMSE via a distance-preserving objective. We design physics-consistent dual-view augmentations separately for scalar and vector variables, employ Swin Transformer 3D with Hierarchical Temporal Attention (HTA) to model spatiotemporal dependencies, and stabilize representations through keyframe reconstruction. The resulting 256-d per-variable fingerprints can be concatenated and searched using scale-adaptive ANN indexing (HNSW or IVFPQ), enabling fast, training-free queries. Experiments demonstrate that Sim-Weather consistently outperforms ClimaX, VideoMAE, and Video Swin Transformer in numerical error, correlation, and perceptual quality. Ablation studies further confirm the complementary roles of each component.
In recent years, Artificial Intelligence (AI)-based weather prediction models have emerged as powerful tools in meteorology, capable of learning complex dependencies from extensive weather datasets and generating rapid forecasts after training. These models achieve prediction accuracies comparable to state-of-the-art Numerical Weather Prediction (NWP) systems. However, these models remain not fully operational due to their dependence on computationally intensive Data Assimilation (DA) systems for generating accurate initial fields. Recent advances in AI techniques offer a potential pathway to develop more efficient and accurate DA systems, advancing the operational feasibility of end-to-end AI-based weather forecasting. Despite growing interest, research in AI-based DA remains fragmented. Therefore, a comprehensive review is necessary to clarify the current progress, identify challenges, and guide the future development of next-generation AI-based DA systems. This review categorizes AI-based DA research into two primary domains. The first domain is AI-empowered DA, where AI enhances individual components such as observation operators, tangent linear and adjoint models, and uncertainty quantification. It also includes latent DA, which helps reduce computational costs. The second domain is AI-based end-to-end DA models, which integrate observations and short-range weather predictions within unified AI frameworks to generate accurate initial fields. We further discuss key challenges and opportunities, including dataset standardization, model evaluation protocols, assimilation of extended observation types, enforcement of physical constraints, and addressing operational scalability. Finally, we emphasize the importance of interdisciplinary collaboration across AI and meteorology in developing practical and reliable AI solutions to enhance DA processes and support more accurate weather forecasting. This review offers practical insights to the research community to expedite the development and operationalization of AI-based DA and end-to-end weather forecasting systems.
This comprehensive survey examines how Reconfigurable Intelligent Surfaces (RIS) revolutionize resource allocation in various network frameworks. It begins by establishing a theoretical foundation with an overview of RIS technologies, including passive RIS, active RIS, and Simultaneously Transmitting and Reflecting RIS (STAR-RIS). The core of the survey focuses on RIS's role in optimizing resource allocation within Single-Input Multiple-Output (SIMO), Multiple-Input Single-Output (MISO), and Multiple-Input Multiple-Output (MIMO) systems. It further explores RIS integration in complex network environments, such as Heterogeneous Wireless Networks (HetNets) and Non-Orthogonal Multiple Access (NOMA) frameworks. Additionally, the survey investigates RIS applications in advanced communication domains like Terahertz (THz) networks, Vehicular Communication (VC), and Unmanned Aerial Vehicle (UAV) communications, highlighting the synergy between RIS and Artificial Intelligence (AI) for enhanced network efficiency. Summary tables provide comparative insights into various schemes. The survey concludes with lessons learned, future research directions, and challenges, emphasizing critical open issues.
Web-oriented services such as image generation and weather forecasting increasingly rely on pre-trained conditional diffusion models (CDMs). When deployed in the open Internet, however, CDMs suffer performance degradation in inference phrase due to noisy inputs, revealing limited robustness. Existing robustness-enhancement techniques such as adversarial training and adversarial purification are not suitable to CDMs. Moreover, those methods fail to meet real-time and reliability requirements of web services. To bridge this gap, we introduce a lightweight, plug-and-play method that can be integrated into any CDMs inference pipeline without altering the underlying architecture. Inspired by deep reinforcement learning, the approach adaptively fuses the outputs of two neural networks, calculating an optimal combination factor via a control-theoretic optimization scheme. Experimental results on CIFAR-10, Moving MNIST and SEVIR datasets demonstrate that CDMs augmented with our method produce higher-quality generation and more accurate forecasts across multiple noise levels. The proposed strategy provides a practical pathway toward the robust deployment of generative AI in safety-critical web applications.
With the rapid urbanization in the modern era, smart traffic profiling based on multimodal sources of data has been playing a significant role in ensuring safe travel, reducing traffic congestion and optimizing urban mobility. Most existing methods for traffic profiling on the road level usually utilize single-modality data, i.e., they mainly focus on image processing with deep vision models or auxiliary analysis on the textual data. However, the joint modeling and multimodal fusion of the textual and visual modalities have been rarely studied in road traffic profiling, which largely hinders the accurate prediction or classification of traffic conditions. To address this issue, we propose a novel multimodal learning and fusion framework for road traffic profiling, named TraffiCFUS. Specifically, given the traffic images, our TraffiCFUS framework first introduces Vision Language Models (VLMs) to generate text and then creates tailored prompt instructions for refining this text according to the specific scene requirements of road traffic profiling. Next, we apply the discrete Fourier transform to convert multimodal data from the spatial domain to the frequency domain and perform a cross-modal spectrum transform to filter out irrelevant information for traffic profiling. Furthermore, the processed spatial multimodal data is combined to generate fusion loss and interaction loss with contrastive learning. Finally, extensive experiments on four realworld datasets illustrate superior performance compared with the state-of-the-art approaches.
In data governance, accurate tracking of table data lineage is the key to ensuring data quality and compliance, and traditional methods mainly rely on static metadata and manual maintenance, which is prone to incomplete or erroneous lineage information for complex nested SQL queries. In order to solve the problem of incomplete lineage extraction, we propose a table-level data lineage acquisition method that combines abstract syntax tree parsing and SQL similarity. The method includes two aspects of table-level kinship acquisition, and the explicit table-level kinship is obtained by parsing SQL scripts through syntax trees. For complex nested SQL query statements that cannot correctly extract table-level kinship from syntax tree parsing, the similarity method is adopted, firstly, the SQL clause is split, the AST tree is constructed, then the structural and semantic similarity are calculated by editing distance and TF-IDF, respectively, and finally AHP comprehensive analysis is used to extract the table information from the high-similarity clauses, supplement the implicit lineage, and store all the obtained lineage in the Neo4j graph database. Experimental results show that the proposed method is better than SQLFlow in terms of accuracy, precision, recall and F1 value, and provides efficient support for table-level lineage tracing under complex nested queries.
Centralized storage in web-based services poses privacy risks for diverse user data, whereas personalized federated learning (PFL) arnesses the inherent privacy-preserving benefits of FL through distributed collaborative training, avoiding the centralization of sensitive data. Besides, PFL also customizes models to individual client needs, enabling personalized web services on non-independent and identically distributed data across varied environments. Existing PFL techniques often rely on prior knowledge to identify personalization layers, overlooking variable layer sensitivities to heterogeneous data, thus constraining the model's adaptability to diverse statistical distributions and limiting its service effectiveness. To overcome this issue, we introduce FedLD, a layer-wise decoupling method for PFL that precisely captures layer-specific sensitivities to heterogeneous data, enhancing service-oriented model personalization. Firstly, FedLD employs a hypernetwork to evaluate each layer's contribution to performance across varied data distributions, determining the optimal proportion of personalized channels per layer for each client, thus enabling fine-grained, channel-level parameter adjustments tailored to individual service needs. Secondly, knowledge distillation is introduced to enforce consistency between the representations of personalized and shared weights, promoting collaboration between them. Thirdly, an improved aggregation strategy is proposed to enhance collaboration among clients with similar data distributions. Extensive experimental results show that FedLD achieves more effective and generalized results compared to baseline methods across various types of heterogeneous data settings.
Graph anomaly detection (GAD), with its ability to accurately identify anomalous patterns in graph data, plays a vital role in areas such as network security, social media platforms, and fraud detection. Graph autoencoder-based methods are widely used for GAD due to their efficiency and effectiveness in capturing complex patterns and learning meaningful representations. However, the above methods are constrained by hardware memory, hindering the detection for large-scale graph data. In this paper, we propose a Memory-Efficient framework for large-scale attributed Graph Anomaly Detection (MEGAD). Specifically, MEGAD first generates node embeddings and then refines them through a lightweight joint optimization model, ensuring minimal memory overhead. The optimized embeddings are subsequently fed into a detector to compute anomaly scores. Extensive experiments demonstrate that our framework achieves comparable accuracy to state-of-the-art methods across multiple datasets while significantly reducing memory consumption on large-scale graphs.
Numerical weather prediction (NWP) models are pivotal for weather forecasting and climate research. As high performance computing (HPC) enters the exascale era, these models are advancing towards unprecedented high-resolution, generating explosive data volumes. In this case, Input/Output (I/O) operations have become a major performance bottleneck, hindering scalability and efficiency. In this paper, we focus on optimizing the I/O performance for the Yin-He Global Spectral Model (YHGSM). To alleviate these bottleneck, we implement a solution centered on a decoupled, asynchronous I/O paradigm. By integrating the XIOS framework, our method externalizes I/O tasks to dedicated server processes, which we enhance with a custom data transfer algorithm that strategically buffers data to maximize the overlap between computation and communication. The resulting integrated system is referred to as YHGSM-XIOS. The experimental results demonstrate substantial performance gains, showing that at a scale of 5,120 processes, the YHGSM-XIOS scheme reduces I/O overhead from 43.08 % to a mere 5.27 % and achieves an overall speedup of 2.31 times compared to the baseline system.
Accurate sea temperature forecasts are crucial for economic and other climate change estimates. However, due to bias, the accuracy of historical simulations and the reliability of future predictions are limited. Moreover, along the coast, such bias is particularly pronounced due to environmental complexity. Prior studies employed deterministic modeling approaches to address this problem. Yet, its bias correction efficacy gradually declines as time progresses, and for bias correction scenarios with varying lead times, the model training exhibits pronounced dependence on forecast products and temporal scale coupling, requiring substantial computational resources and costs. We propose a diffusion model-based approach that simulates natural stochastic processes through partial noise injection. Using an erosion operation to compute the coastal mask, which corrects sea surface temperature forecast biases in target regions. This model, trained solely on historical reanalysis data, effectively addresses the shortcomings of previous methods. Furthermore, we propose an adaptive denoising parameterization scheme targeting noise intensity and denoising steps, which achieves dynamic optimization of parameters. Extensive results from real-world dataset show the efficacy and superiority of our approach. Compared to previous models, our method reduces the RMSE of bias-corrected sea temperature forecasts by $\mathbf{1 9. 3 2 \%}$, achieving state-of-the-art results.