We survey continuous-time generative modeling methods based on transporting a simple reference distribution to a data distribution via stochastic or deterministic dynamics. We present a unified framework in which diffusion models, score-based generative models, and flow matching are instances of learning a time-dependent vector field that induces a family of marginals (ρ_t)_t ∈ [0,1] governed by continuity and Fokker-Planck equations. Such a unified theory is timely because these methods are converging methodologically, yet fragmented notation and competing derivations continue to obscure their shared structure and the practical tradeoffs governing sampling, stability, and computation. Within this framework, we (i) derive reverse-time sampling for diffusion and score-based models as controlled stochastic dynamics, (ii) show that the probability flow ODE yields identical marginals and connects diffusion to likelihood-based normalizing flows, and (iii) interpret flow matching as direct regression of the velocity field under a chosen interpolation, clarifying when it coincides with or differs from score-based training. We compare objectives, sampling schemes, and discretization errors under unified notation, discuss connections to Schrodinger bridges and entropic optimal transport, and summarize theoretical guarantees and open problems on approximation, stability, and scalability.

Stem Water Potential (SWP) is the standard method for assessing water stress and irrigation scheduling in tree crops. This method is time-consuming and labor-intensive, limiting data collection to only a few trees in the orchard. To find an alternative approach that predicts water stress in every tree in the orchard, we implemented a novel Stochastic Decision Tree (SDT) method, utilizing remote sensing and weather data to predict SWP in almond and pistachio orchards. The input data for our model included various vegetative indices such as NDVI, GNDVI, OSAVI, LCI, and NDRE, as well as local weather parameters, such as temperature ( T_a ), relative humidity (RH), air pressure (P), Vapour Pressure Deficit (VPD), and the Water Stress Index (WSI). Our results indicate that the SDT model achieves a prediction accuracy of nearly 94

Stochastic gradient descent and other first-order variants, such as Adam and AdaGrad, are commonly used in the field of deep learning due to their computational efficiency and low-storage memory requirements. However, these methods do not exploit curvature information. Consequently, iterates can converge to saddle points or poor local minima. On the other hand, Quasi-Newton methods compute Hessian approximations which exploit this information with a comparable computational budget. Quasi-Newton methods re-use previously computed iterates and gradients to compute a low-rank structured update. The most widely used quasi-Newton update is the L-BFGS, which guarantees a positive semi-definite Hessian approximation, making it suitable in a line search setting. However, the loss functions in DNNs are non-convex, where the Hessian is potentially non-positive definite. In this paper, we propose using a limited-memory symmetric rank-one quasi-Newton approach which allows for indefinite Hessian approximations, enabling directions of negative curvature to be exploited. Furthermore, we use a modified adaptive regularized cubics approach, which generates a sequence of cubic subproblems that have closed-form solutions with suitable regularization choices. We investigate the performance of our proposed method on autoencoders and feed-forward neural network models and compare our approach to state-of-the-art first-order adaptive stochastic methods as well as other quasi-Newton methods.x

The Byzantine Agreement (BA) problem is a fundamental challenge in distributed systems, focusing on achieving reaching an agreement among parties, some of which may behave maliciously. With the rise of cryptocurrencies, there has been significant interest in developing atomic broadcast protocols, which facilitate agreement on a subset of parties' requests. However, these protocols often come with high communication complexity (O(ln^2 + λ n^3 log n), where l is the bit length of the input, n is the number of parties, and λ represents the security parameter bit length). This can lead to inefficiency, especially when the requests across parties exhibit little variation, resulting in unnecessary resource consumption. In this paper, we introduce Slim-ABC, a novel atomic broadcast protocol that eliminates the O(ln^2 + λ n^3 log n) term associated with traditional atomic broadcast protocols. While Slim-ABC reduces the number of accepted requests, it significantly mitigates resource wastage, making it more efficient. The protocol leverages the asynchronous common subset and provable-broadcast mechanisms to achieve a communication complexity of O(ln^2 + λ n^2). Despite the trade-off in accepted requests, Slim-ABC maintains robust security by allowing only a fraction (f+1) of parties to broadcast requests. We present an extensive efficiency analysis of Slim-ABC, evaluating its performance across key metrics such as message complexity, communication complexity, and time complexity. Additionally, we provide a rigorous security analysis, demonstrating that Slim-ABC satisfies the agreement, validity, and totality properties of the asynchronous common subset protocol.

Signal recovery often involves separating and realizing multiple superimposed signals at once. Separating multiple images that have been superimposed is a challenging signal recovery problem. This situation arises when a detector, such as a microphone, receives multiple signals simultaneously. In order to recover the original signals, a signal separator needs to be applied. In this paper, we will explore machine learning techniques for separating such signals. In particular, we investigate two approaches: an autoencoder approach and a transformer-based approach, and test their accuracy in recovering two separate images from noisy low-resolution superimposed measurements.

Deep Neural Networks (DNN) have become a promising paradigm when developing Artificial Intelligence (AI) and Machine Learning (ML) applications. However, DNN applications are vulnerable to fake data that are crafted with adversarial attack algorithms. Under adversarial attacks, the prediction accuracy of DNN applications suffers, making them unreliable. In order to defend against adversarial attacks, we introduce a novel noise-reduction procedure, Vector Quantization U-Net (VQUNet), to reduce adversarial noise and reconstruct data with high fidelity. VQUNet features a discrete latent representation learning through a multi-scale hierarchical structure for both noise reduction and data reconstruction. The empirical experiments show that the proposed VQUNet provides better robustness to the target DNN models, and it outperforms other state-of-the-art noise-reduction-based defense methods under various adversarial attacks for both Fashion-MNIST and CIFAR10 datasets. When there is no adversarial attack, the defense method has less than 1% accuracy degradation for both datasets.

Active cyber defense mechanisms are necessary to perform automated, and even autonomous operations using intelligent agents that defend against modern/sophisticated AI-inspired cyber threats (e.g., ransomware, cryptojacking, deep-fakes). These intelligent agents need to rely on deep learning using mature knowledge and should have the ability to apply this knowledge in a situational and timely manner for a given AI-inspired cyber threat. In this paper, we describe a ‘domain-agnostic knowledge graph-as-a-service’ infrastructure that can support the ability to create/store domain-specific knowledge graphs for intelligent agent Apps to deploy active cyber defense solutions defending real-world applications impacted by AI-inspired cyber threats. Specifically, we present a reference architecture, describe graph infrastructure tools, and intuitive user interfaces required to construct and maintain large-scale knowledge graphs for the use in knowledge curation, inference, and interaction, across multiple domains (e.g., healthcare, power grids, manufacturing). Moreover, we present a case study to demonstrate how to configure custom sets of knowledge curation pipelines using custom data importers and semantic extract, transform, and load scripts for active cyber defense in a power grid system. Additionally, we show fast querying methods to reach decisions regarding cyberattack detection to deploy pertinent defense to outsmart adversaries.

LoRa has recently become one of the most promising networking technologies for the Internet of Things applications. Distant end devices have to use a low data rate to reach a LoRa gateway, which can cause long in-the-air transmission time and high energy consumption. Compared with the end devices using high data rate, they will drain the batteries much earlier and the network may be broken. Such an energy unfairness can be mitigated by deploying more gateways, since it allows end devices to reach closer gateways with higher data rates. However, multiple gateways may not solve the energy unfairness problem efficiently due to the collision problem caused by the chirp spread spectrum modulation of LoRa networks. Spreading factors of LoRa links can determine both data rate and multiplexing of different transmissions. With more gateways, more end devices may choose low spreading factors and reach closer gateways, which increase the collision probability. In this paper, we propose a networking solution for LoRa networks named EF-LoRa that can achieve fair energy consumption among end devices by carefully allocating different network resources, including frequency channels, spreading factors and transmission power, to achieve fair energy consumption among end devices in LoRa networks. We develop a LoRa network model to study the energy consumption of all the end devices in a network by considering the unique features of LoRa networks, such as LoRaWAN MAC protocol, spreading factors, interference, and the capacity limitation of a LoRa gateway. We formulate the energy fairness problem as an optimization problem and finally propose a greedy resource allocation algorithm to achieve the max-min fairness of energy efficiency in the LoRa networks. Simulation results show that the proposed solution EF-LoRa can improve the energy fairness of legacy LoRa networks by 177.8%.

Tree construction is one of the popular methods for tackling any supervised task in machine learning. However, there has been little effort in applying trees for unsupervised tasks. The traditional unsupervised trees are based on recursively partitioning the space such that the achieved partitions contain similar samples. Sense of similarity depends on the models and applications. This paper tackles the issue of learning optimal clustering oblique trees for the first time and proposes a linear time algorithm for training it. Optimizing performance of infrastructures and energy consumption in the field of Internet of things can be mentioned as applications of tree and clustering, respectively. The motivation of unsupervised tree models is to preserve the data manifold, while keeping the query, while keeping the query time fast. Popular unsupervised models consist of k-d trees, random projection (RP trees), principal component analysis trees (PCA trees) and clustering trees. However, all existing methods for unsupervised tree are sub-optimal. Additionally, existing clustering trees are limited to axis-aligned trees. Further, some of the mentioned methods suffer from curse of dimensionality such as k-d trees. Despite the mentioned challenges, trees are fast in query time. On the other hand, a non-hierarchical clustering such as k-means has both: It performs well in high-dimensional problems and is locally optimal. Its learning algorithm is efficient. However, k-means clustering is not fast in query time. To address the mentioned issues, this paper proposes a novel k-means tree, a tree that outputs the centroids of clusters. The advantages of such tree are being fast in query time and also learning as good cluster centroids as k-means. As a result, problem of learning such trees is to learn both centroids and the tree parameters optimally and jointly. In this paper, this problem is first cast as a constrained minimization problem and then solved using quadratic penalty method. The method consists of learning clusters from k-means and gradually adapting centroids to the outputs of an optimal oblique tree. The alternating optimization is used, and alternation steps consist of weighted k-means clustering and tree optimization. Additionally, the training complexity of proposed algorithm is efficient. Proposed algorithm is optimal in the sense of learned clusters and tree jointly. Trees used in the k-means tree are oblique, and as per our knowledge, this is the first time that oblique trees are applied to the task of clustering. As a side product of the proposed method, sample reduction is explored and shown its merits. It is shown that computational complexity of training KMT (K-means tree) as a sample reduction method is faster than training K-means as a sample reduction. The training complexity of KMT sample reduction algorithm is logarithmic over the size of reduced train set, while training complexity of K-means is linear over the size of reduced dataset. Finally, proposed method is compared to other tree-based clustering algorithms and its superiority in terms of reconstruction error is shown. Additionally, its query complexity is compared with k-means.

Synthetic Reduced Nearest Neighbor is a nearest neighbor model that is constrained on synthetic samples (i.e., prototypes). The body of work on such models includes proposals for improving the interpretability and optimization of SRNN models using expectation maximization. Motivated by the promise of this paradigm, we propose a novel Expectation Maximization approach for Synthetic Reduced Nearest Neighbors leveraging neural networks. Furthermore, we compare the performance of our proposed technique to classical state-of-the-art machine learning methods such as random forest and ensemble models. The empirical results demonstrate the advantages of using neural networks in lieu of an expectation maximization algorithm.

Stochastic gradient descent and other first-order variants, such as Adam and AdaGrad, are commonly used in the field of deep learning due to their computational efficiency and low-storage memory requirements. However, these methods do not exploit curvature information. Consequently, iterates can converge to saddle points and poor local minima. To avoid these points, directions of negative curvature can be utilized, which requires computing the second-derivative matrix. In Deep Neural Networks (DNNs), the number of variables ($n$) can be of the order of tens of millions, making the Hessian impractical to store ($\mathcal{O}(n^2)$) and to invert ($\mathcal{O}(n^3)$). Alternatively, quasi-Newton methods compute Hessian approximations that do not have the same computational requirements. Quasi-Newton methods re-use previously computed iterates and gradients to compute a low-rank structured update. The most widely used quasi-Newton update is the L-BFGS, which guarantees a positive semi-definite Hessian approximation, making it suitable in a line search setting. However, the loss function in DNNs are non-convex, where the Hessian is potentially non-positive definite. In this paper, we propose using a Limited-Memory Symmetric Rank-1 quasi-Newton approach which allows for indefinite Hessian approximations, enabling directions of negative curvature to be exploited. Furthermore, we use a modified Adaptive Regularized Cubics approach, which generates a sequence of cubic subproblems that have closed-form solutions. We investigate the performance of our proposed method on autoencoders and feed-forward neural network models and compare our approach to state-of-the-art first-order adaptive stochastic methods as well as L-BFGS.

Nearest neighbor models are among the most established and accurate approaches to machine learning. In this paper, we investigate Synthetic Reduced Nearest Neighbor (SRNN) as a novel approach to regression tasks. Existing prototype nearest neighbor models are initialized by training a k-means model over each class. However, such initialization is only applicable to classification tasks. In this work, we propose a novel initialization and expectation maximization approach for enabling the application of SRNN to regression. The proposed initialization approach is based on applying the k-means algorithm on the target responses of samples to create various clusters of targets. This is proceeded by learning several centroids in the input space for each cluster found over the targets. Essentially, the initialization consists of finding target clusters and running k-means in the space of feature vectors for the corresponding target cluster. The optimization procedure consists of applying an expectation maximization approach similar to the k-means algorithm that optimizes the centroids in the input space. This algorithm is comprised of two steps: (1) The assignment step, where assignments of the samples to each centroid is found and the target response (i.e., prediction) of each centroid is determined; and (2) the update/centroid step, where each centroid is updated such that the loss function of the entire model is minimized. We will show that the centroid step operates over all samples via solving a weighted binary classification. However, the centroid step is NP-hard and no surrogate objective function exists for solving this problem. Therefore, a new surrogate is proposed to approximate the solution for the centroid step. Furthermore, we consider the consistency of the model, and show that the model is consistent under mild assumptions. The bias-variance relationship in this model is also discussed. We report the empirical evaluation of the proposed SRNN regression model in comparison to several state-of-the-art techniques.

Recurrent neural networks (RNNs) are traditionally used for machine learning applications for temporal sequences such as natural language processing. Its application to image processing is relatively new. In this paper, we apply RNNs to denoise images corrupted by mixed Poisson and Gaussian noise. The motivation for using an RNN comes from viewing the denoising of the Poisson-Gaussian realization as a temporal process. The network then attempts to trace back the steps that create the noisy realization in order to arrive at the noiseless reconstruction. Numerical experiments demonstrate that our proposed RNN approach outperforms convolutional autoencoder methods for denoising and upsampling low-resolution images from the CIFAR-10 dataset.

In this paper, we propose a novel machine learning technique to detect and correctly classify adversarially modified images. Our approach takes advantage of the adversarial training objective (to influence the prediction with small perturbations to the original image) by searching for large discrepancies in the model’s output relative to the input image. We find clusters in each space, and for an image whose clusters do not agree with each other, we flag the image as potentially adversarial and use the input space cluster’s prediction in place of the attacked model’s output. We have found that this method tends to find adversarial samples which are more likely to be misclassified by the attacked model, and that simply deferring to the input cluster’s prediction on such samples is enough to increase accuracy on adversarial data, even when the attacked model is adversarially trained.

Decision trees are a convenient and established approach for any supervised learning task. Decision trees are trained by greedily splitting a leaf nodes, into two leaf nodes until a specific stopping criterion is reached. Splitting a node consists of finding the best feature and threshold that minimizes a criterion. The criterion minimization problem is solved through a costly exhaustive search algorithm. This paper proposes a novel stochastic approach for criterion minimization. The algorithm is compared with several other related state-of-the-art decision tree learning methods, including the baseline non-stochastic approach. We apply the proposed algorithm to learn a Haar tree over MNIST dataset that consists of over 200, 000 features and 60, 000 samples. The result is comparable to the performance of oblique trees while providing a significant speed-up in both inference and training times.

State-of-the-art machine learning models are vulnerable to data poisoning attacks whose purpose is to undermine the integrity of the model. However, the current literature on data poisoning attacks is mainly focused on ad hoc techniques that are only applicable to specific machine learning models. Additionally, the existing data poisoning attacks in the literature are limited to either binary classifiers or to gradient-based algorithms. To address these limitations, this paper first proposes a novel model-free label-flipping attack based on the multi-modality of the data, in which the adversary targets the clusters of classes while constrained by a label-flipping budget. The complexity of our proposed attack algorithm is linear in time over the size of the dataset. Also, the proposed attack can increase the error up to two times for the same attack budget. Second, a novel defense technique based on the Synthetic Reduced Nearest Neighbor (SRNN) model is proposed. The defense technique can detect and exclude flipped samples on the fly during the training procedure. Through extensive experimental analysis, we demonstrate that (i) the proposed attack technique can deteriorate the accuracy of several models drastically, and (ii) under the proposed attack, the proposed defense technique significantly outperforms other conventional machine learning models in recovering the accuracy of the targeted model.

Deep Neural Networks (DNNs) approaches have been used successfully in various computer vision tasks. However, they are particularly susceptible to adversarial attacks, which can cause incorrect predictions and raise security risk to real-world deep learning applications, e.g., autonomous driving and surveillance systems. Many state-of-the-art adversarial defense models use Variational Autoencoder Decoder (VAE) to reconstruct clean images to counter adversarial attacks. However, images reconstructed from VAEs are often blurry, thus, not only their use for classification is limited, but also such defense methods blur the benign inputs and impair the accuracy when there is no adversarial attack. We propose to use Cycle Consistency GAN (CycleGAN) as the image reconstruction block, which generates higher quality images for better classification performance. Our method achieves 5%-13% higher accuracy than the best VAE-based defense models on CIFAR10 and achieves 90%-98% accuracy for Fashion-MNIST across wide range of adversarial attacks.

The motivation of active learning is that by providing limited labeled training samples, a machine learning algorithm can provide higher accuracy. The provided training samples are selected from a large or streaming dataset. The selection procedure often incorporates some measure of informativeness of samples. This measure is also defined based on the machine learning model itself. The data used in active learning is usually unlabeled; hence, the selected samples have to be labeled by an oracle (e.g., a human or a machine annotator). This is in case that labeling data is time-consuming or expensive in some way.

In this paper, we implement multi-stage deep learning methods to recover signals from noisy observations. We seek an alternative to the traditional iterative optimization-based methods by exploiting the denoising properties of autoencoders. The novelty of our method is in the reformulation of the denoising problem as a multi-stage process and in the use of recurrent neural networks to gradually recover the intended signal. Numerical experiments on a modified version of the CIFAR-10 dataset show that signal recovery in multiple stages produces sharper images than a standard autoencoder approach. We present quantitative evidence of improvement using both the mean squared error and the structural similarity index.