Data assimilation (DA) integrates observations with model forecasts to produce optimized atmospheric states, whose physical consistency is critical for stable weather forecasting and reliable climate research. Traditional Bayesian DA methods enforce these nonlinear, flow-dependent physical constraints through empirical and tunable covariance structures, but with limited accuracy and robustness. Here, we introduce latent DA (LDA), a framework that performs Bayesian DA in a latent space learned from multivariate global atmospheric data via an autoencoder. We demonstrate that the autoencoder can largely capture nonlinear physical relationships, enabling LDA to produce balanced analyses without explicitly modeling physical constraints. Assimilation in latent space also improves both analysis quality and forecast skill compared to traditional model-space DA, under both idealized and real observational settings. Furthermore, LDA exhibits strong robustness across latent dimensions and remains effective even when the autoencoder is trained on inaccurate but physically realistic forecasts, highlighting its flexibility for real-world applications.

Graph analysis has achieved success in challenging tasks such as importance measures, fraud detection, and anti-money laundering. For a deep understanding of various complex systems in real life, a whole graph may involve data and connections from multiple sources. Secure multi-party computation is suitable for this scenario, which allows untrusted parties to compute collectively without revealing their individual data. However, a significant challenge is the high communication overhead, especially during intricate computations of large-scale input, and existing secure graph analysis frameworks incur a lower bound of Omega(|V|+|E|) communication per iteration, where V and E denote vertices and edges, respectively. This paper proposes GraphAce, an efficient secure two-party graph analysis framework, which adopts a distinct technical roadmap from existing solutions. We identify and address the security challenges when utilizing local graph data of parties, with the mixed primitives system of homomorphic encryption and secret sharing, and a novel ChaosTable data structure that protects privacy during cross-party computation. Consequently, GraphAce eliminates any network traffic related to the edges. For each iteration, it achieves low complexities of Theta(|V|) communication, breaking the Omega(|V|+|E|) lower bound of previous secure solutions, and Theta(|V|+|E|) computation, which is the same as insecure methods. Evaluations show that GraphAce exceeds previous methods by up to tens of thousands of times in speed and saves up to 99.99% communication, depending on the application and the network. This is the first secure two-party graph analysis framework capable of processing over 1 million vertices and 132 million edges in a reasonable time, which is 128x larger than previous reports, to the best of our knowledge.

Accurate estimation of background error (i.e., forecast error) distribution is critical for effective data assimilation (DA) in numerical weather prediction (NWP). In state-of-the-art operational DA systems, it is common to account for the temporal evolution of background errors by employing hybrid methods, which blend a static climatological covariance with a flow-dependent ensemble-derived component. While effective to some extent, these methods typically assume Gaussian-distributed errors and rely heavily on hand-crafted covariance structures and domain expertise, limiting their ability to capture the complex, non-Gaussian nature of atmospheric dynamics. In this work, we propose LoRA-EnVar, a novel hybrid ensemble variational DA algorithm that integrates low-rank adaptation (LoRA) into a deep generative modeling framework. We first learn a climatological background error distribution using a variational autoencoder (VAE) trained on historical data. To incorporate flow-dependent uncertainty, we introduce LoRA modules that efficiently adapt the learned distribution in response to flow-dependent ensemble perturbations. Our approach supports online finetuning, enabling dynamic updates of the background error distribution without catastrophic forgetting. We validate LoRA-EnVar in high-resolution assimilation settings using the FengWu forecast model and simulated observations from ERA5 reanalysis. Experimental results show that LoRA-EnVar significantly improves assimilation accuracy over models assuming static background error distribution and achieves comparable or better performance than full finetuning while reducing the number of trainable parameters by three orders of magnitude. This demonstrates the potential of parameter-efficient adaptation for scalable, non-Gaussian DA in operational meteorology.

Ego-centric queries, focusing on a target vertex and its direct neighbors, are essential for various applications. Enabling such queries on graphs owned by mutually distrustful data providers without breaching privacy holds promise for more comprehensive results. In this paper, we propose GORAM , a graph-oriented data structure that enables efficient ego-centric queries on federated graphs with strong privacy guarantees. GORAM leverages secure multiparty computation (MPC) and ensures that no information about the graphs or the querying keys is exposed during the process. For practical performance, GORAM partitions the federated graph and constructs an Oblivious RAM (ORAM) -inspired index atop these partitions. This design enables each ego-centric query to process only a single partition, which can be accessed fast and securely. Utilizing GORAM , we develop a prototype querying engine on a real-world MPC framework. We then conduct a comprehensive evaluation using five commonly used queries similar to the LinkBench workload description [ 11 ] on both synthetic and real-world graphs. Our evaluation shows that all five queries can be completed in just 58.1 milliseconds to 35.7 seconds, even on graphs with up to 41.6 million vertices and 1.4 billion edges. To the best of our knowledge, this represents the first instance of processing billion-scale graphs with practical performance on MPC.

Encrypted matrix-vector multiplication is a fundamental component of a variety of applications that involve data privacy concerns. Current algorithms utilizing fully homomorphic encryption (FHE) generally use batching to enhance computational efficiency while neglecting the sparsity of the matrices, a characteristic that exists naturally in many practical situations. Alternatively, porting plaintext algorithms that skip zero elements to address sparsity may fail to utilize batching and introduce additional privacy concerns. We propose Lodia, an efficient outsourced sparse matrix-vector multiplication (SpMV) algorithm for batched FHE schemes without sacrificing privacy. It only requires Theta((n + m) log(n + m)/s) FHE operations, where n is the number of rows/columns, m is the number of non-zero elements of the matrix, and.. is the batch size of the FHE scheme. This is optimal for m = Omega(n) and m = O(n(rho)) for some rho < 2 (i.e., an <= m <= bn(rho) asymptotically), covering most practical cases. To our knowledge, no method has been published with better than Theta(n(2)/s) FHE operations, suitable for any sparse matrix, and without privacy concerns. Lodia utilizes a novel low-diagonal decomposition, which decomposes a sparse matrix into a series of special matrices named low-diagonal matrices. Based on a conventional method encoding the matrix in diagonal order, each low-diagonal matrix can be efficiently multiplied by a vector. This results in an efficient SpMV method suitable for any sparse matrix. Experiments show that Lodia practically achieves a speedup of up to 96x compared to baselines that ignore matrix sparsity, and up to 3.6x compared to implementations even with fewer security guarantees. This is the first SpMV solution on encrypted data that can process a substantial matrix with over 8 million rows/columns and 125 million non-zero elements.

Secure multi-party computation techniques enable in-depth analysis on joint graphs that inherently encompass comprehensive topology information and extended attributes, while preserving data privacy. In the two-party setting, existing approaches suffer from inefficiencies due to redundant secure sorting or costly secure shuffling operations required for secure message passing. Some works improve efficiency by relaxing security assumptions, either through differential privacy or by introducing helper parties. We present PETAL, a high-performance parallel framework for private iterative two-party graph analysis based on garbled circuits, ensuring simulation-based security under the semi-honest adversary assumption. Exploiting the iterative nature of graph analysis, we propose a novel two-party secret permutation protocol to replace secret sorting for data reordering after the first iteration, significantly improving performance. The protocol leverages correlations between permutations across iterations to reuse intermediate results, achieving linear communication after an initial setup. For the initial reordering, we design a two-party secret sorting protocol with low asymptotic complexity and small constant factors. On 2M-sized graphs, evaluations show PETAL achieves on average a 4.6x speedup and 55% less communication across four real-world applications, over the advanced version of the GraphSC framework by Araki et al.

Pair-then-Aggregate (PtA) introduces a programming paradigm and an automated parallel execution engine for large-scale secure multi-party (MPC) computations, drawing inspiration from the widely-used yet not explicitly defined Table-Generation-and-Look-up (TGL) pattern in privacy-preserving algorithm design. PtA offers an easy-to-use API and a versatile execution engine that harnesses various levels of parallelism and adapts to different MPC deployments, algorithms, and input sizes. Evaluations on a real-world MPC platform demonstrate significant enhancements in scalability, adaptability, and ease of programming. PtA can process one billion input elements with 3-23 lines of C++ code in 5-74 seconds. It outperforms state-of-the-art implementations in 91.4% of 35 test cases, achieving up to a 12.4x speedup with much less coding effort.(1)

Data assimilation (DA) provides more accurate, physically consistent analysis fields and is used for estimating initial conditions in numerical weather forecasting. Traditional DA methods derive statistically optimal analyses in model space based on Bayesian theory. However, their effectiveness is limited by the difficulty of accurately estimating the background error covariances matrix B, which represents the intricate interdependencies among atmospheric variables, as well as the standard linearity assumptions required during the assimilation step. To address these limitations, we propose Latent Data Assimilation (LDA) for a multi-variable global atmosphere, performing non-linear Machine-Learning based Bayesian DA on an atmospheric latent representation learned by an autoencoder. The feasibility of LDA is supported by the near-linear relationship between increments in latent space (within the typical magnitude range for DA) and their corresponding impacts in model space, ensuring that the optimal analysis obtained in latent space approximates the optimal analysis in model space. Due to the relationships among the atmospheric variables encoded in the latent space, LDA can physically propagate observation information across unobserved regions and atmospheric variables, even with a fully diagonal B in latent space. We perform idealized experiments with simulated observations and demonstrate the superiority of LDA over traditional DA methods in model space, while the experiments assimilating real observations highlight its potential application for operational reanalysis and weather forecasting systems.

Trusted processors provide a way to perform joint computations while preserving data privacy. To overcome the performance degradation caused by data-oblivious algorithms to prevent information leakage, we explore the benefits of oblivious memory (OM) integrated in processors, to which the accesses are unobservable by adversaries. We focus on graph analytics, an important application vulnerable to access-pattern attacks. With a co-design between storage structure and algorithms, our prototype system is 100x faster than baselines given an OM sized around the per-core cache which can be implemented on existing processors with negligible overhead. This gives insights into equipping trusted processors with OM.

Wind power are being massively integrated in clusters worldwide which also exhibits strong uncertainty and randomness. Ultra-short-term wind power prediction is an important way to address this uncertainty which plays utmost role for wind farms to participate in spot trading and to enhance production profits. Mining the spatial and temporal correlations within clustered wind farm output data is crucial for enhancing ultra-short-term wind power prediction accuracy. However, privacy concerns and business management regulations often hinder direct data sharing among independently owned wind farms. Although cryptographic techniques have been devised to protect privacy during data sharing, encrypting the original data while extracting nonlinear relationships among multiple wind farms remains a challenging problem in machine learning. In this study, secure multi-party computation (SMPC) is applied to wind farm power prediction for the first time to develop a flexible, efficient, and privacy-preserving prediction algorithm named pwXGBoost, which is capable of extracting complex spatial-temporal correlations during both training and prediction phases. Besides, a maximum mean discrepancy (MMD)-based approach is also proposed to identify suitable adjacent candidate wind farms in the cluster for collaborative model training, thereby enhancing accuracy and reducing data acquisition burdens. The proposed approach is evaluated using real-world data from a cluster of wind farms in China, demonstrating its efficiency and performance improvements while maintaining data privacy.

In response to climate change, renewable energy such as wind power are being massively integrated in clusters worldwide. But wind power also exhibits strong uncertainty and randomness. Ultra-short-term power forecasting for wind farm clusters is an important way to address this uncertainty and plays a vital role in the participation of wind farms in the electricity spot market and in enhancing their revenue. Mining the spatial and temporal correlations within clustered wind farm output data is crucial for enhancing ultra-short-term wind power prediction accuracy. However, privacy concerns and business management regulations often hinder direct data sharing among independently owned wind farms. Although cryptographic techniques have been devised to protect privacy during data sharing, encrypting the original data while extracting nonlinear relationships among multiple wind farms remains a challenging problem in machine learning. This paper introduces pwXGBoost, a distribted prediction method based on machine learning tree models and secure multi-party computation (SMPC) that successfully uncovers intricate relationships while safeguarding data privacy. We also propose a maximum mean discrepancy (MMD)-based approach to identify suitable adjacent candidate wind farms in the cluster for collaborative model training, thereby enhancing accuracy and reducing data acquisition burdens. We evaluate our approach using real-world data from a cluster of wind farms in China, demonstrating its efficiency and performance improvements while maintaining data privacy.

Enabling various parties to share data enhances online fraud detection capabilities considering fraudsters tend to reuse resources attacking multiple platforms. Multi-party computation (MPC) techniques, such as secret sharing, offer potential privacy-preserving solutions but face efficiency challenges when handling large-scale data. This paper presents a novel approach, SecureFD (Secure Fraud Detector), aimed at detecting fraud in multi-party graph data, ensuring privacy, accuracy, and scalability. We propose a graph neural network EPR-GNN, which is MPC-friendly, as the base detector. Then we design a framework that allows multiple parties to train EPR-GNN collaboratively on secure sparse graphs in a privacy-preserving manner. The oblivious node embedding sharing protocol in the collaborative training procedure achieves up to a 45x speed-up, supporting over four million users compared to the naive solution. Additionally, we further reduce secure computation by locally pruning a significant number of non-suspicious users and selecting only the most valuable resources for sharing. Experiments on real datasets demonstrate that by securely integrating data from different parties, SecureFD achieves superior detection performance compared to state-of-the-art local detectors. And the local pruning greatly improves the scalability without compromising detection accuracies.

Mining the spatial and temporal correlation of wind farm output data is beneficial for enhancing the precision of ultra-short-term wind power prediction. However, if the wind farms are owned by separate entities, they may be reluctant to share their data directly due to privacy concerns as well as business management regulation policies. Although cryptographic approaches have been designed to protect privacy in the process of data sharing, it is still a challenging problem to encrypt the original data while extracting the nonlinear relationship among multiple wind farms in the machine learning process. This paper presents pwXGBoost, a technique based on the machine learning tree model and secure multi-party computation (SMPC) that can successfully extract complicated relationships while preserving data privacy. A maximum mean discrepancy (MMD) based scheme is proposed to effectively choose adjacent candidate wind farms to participate in the collaborative model training, therefore improving the accuracy and reducing the burden of data acquisition. The proposed method was evaluated on real world data collected from a cluster of wind farms in Inner Mongolia, China, demonstrating that it is capable of achieving considerable efficiency and performance improvements while preserving privacy

Due to the absence of a library for non-linear function evaluation, so-called general-purpose secure multi-party computation (MPC) are not as ”general” as MPC programmers expect. Prior arts either naively reuse plaintext methods, resulting in suboptimal performance and even incorrect results, or handcraft ad hoc approximations for specific functions or platforms. We propose a general technique, NFGen, that utilizes pre-computed discrete piecewise polynomials to accurately approximate generic functions using fixed-point numbers. We implement it using a performance-prediction-based code generator to support different platforms. Conducting extensive evaluations of 23 non-linear functions against six MPC protocols on two platforms, we demonstrate significant performance, accuracy, and generality improvements over existing methods.