Feature selection is a critical step in data-driven applications, reducing input dimensionality to enhance learning accuracy, computational efficiency, and interpretability. Existing state-of-theart methods often require post-selection retraining and extensive hyperparameter tuning, complicating their adoption. We introduce a novel, non-intrusive feature selection layer that, given a target feature count k, automatically identifies and selects the k most informative features during neural network training. Our method is uniquely simple, requiring no alterations to the loss function, network architecture, or post-selection retraining. The layer is mathematically elegant and can be fully described by: (x) over tilde (i) = a(i)x(i) + (1 - a(i))z(i) where xi is the input feature, (x) over tilde (i) the output, z(i) a G Paussian noise, and ai trainable gain such that Sigma(i) a(i)(2) = k. This formulation induces an automatic clustering effect, driving k of the a(i) gains to 1 (selecting informative features) and the rest to 0 (discarding redundant ones) via weighted noise distortion and gain normalization. Despite its extreme simplicity, our method achieves competitive performance on standard benchmark datasets and a novel real-world dataset, often matching or exceeding existing approaches without requiring hyperparameter search for k or retraining. Theoretical analysis in the context of linear regression further validates its efficacy. Our work demonstrates that simplicity and performance are not mutually exclusive, offering a powerful yet straightforward tool for feature selection in machine learning.

The training phase of deep neural networks requires substantial resources and as such is often performed on cloud servers. However, this raises privacy concerns when the training dataset contains sensitive content, e.g., facial or medical images. In this work, we propose a method to perform the training phase of a deep learning model on both an edge device and a cloud server that prevents sensitive content being transmitted to the cloud while retaining the desired information. The proposed privacy-preserving method uses adversarial early exits to suppress the sensitive content at the edge and transmits the task-relevant information to the cloud. This approach incorporates noise addition during the training phase to provide a differential privacy guarantee. We extensively test our method on different facial and medical datasets with diverse attributes using various deep learning architectures, showcasing its outstanding performance. We also demonstrate the effectiveness of privacy preservation through successful defenses against different white-box, deep and GAN-based reconstruction attacks. This approach is designed for resource-constrained edge devices, ensuring minimal memory usage and computational overhead.

Deep neural networks (DNNs) provide state-of-the-art accuracy for vision tasks, but they require significant resources for training. Thus, they are trained on cloud servers far from the edge devices that acquire the data. This issue increases communication cost, runtime, and privacy concerns. In this study, a novel hierarchical training method for DNNs is proposed that uses early exits in a divided architecture between edge and cloud workers to reduce the communication cost, training runtime, and privacy concerns. The method proposes a brand-new use case for early exits to separate the backward pass of neural networks between the edge and the cloud during the training phase. We address the issues of most available methods that, due to the sequential nature of the training phase, cannot train the levels of hierarchy simultaneously or they do it with the cost of compromising privacy. In contrast, our method can use both edge and cloud workers simultaneously, does not share the raw input data with the cloud, and does not require communication during the backward pass. Several simulations and on-device experiments for different neural network architectures demonstrate the effectiveness of this method. It is shown that the proposed method reduces the training runtime for VGG-16 and ResNet-18 architectures by 29% and 61% in CIFAR-10 classification and by 25% and 81% in Tiny ImageNet classification, respectively, when the communication with the cloud is done over a low bit rate channel. This gain in the runtime is achieved, while the accuracy drop is negligible. This method is advantageous for online learning of high-accuracy DNNs on sensor-holding low-resource devices such as mobile phones or robots as a part of an edge-cloud system, making them more flexible in facing new tasks and classes of data.

Our advanced computer vision system allows for the precise tracking of serial numbers on steel billets in challenging industrial settings. It combines cutting-edge hardware and machine learning, excelling in character recognition (99.8%) and localization while adapting to dynamic ambient lighting conditions (104 ). Moreover, it accurately measures crucial geometric parameters such as side sizes, bulging, and skewness. This multifaceted technology promises to elevate material tracking, quality assessment, and production optimization in the steel industry to unprecedented levels.

Water quality monitoring in sewer networks remains a technical challenge even though water pollution and control are high priorities since decades. Current water quality monitoring usually analyzes samples in laboratories, allowing only sporadic measurements, or uses immersed sensors in the wastewater, leading to clogging and sensor fouling resulting in expense due to intensive maintenance. Both techniques thus have serious limitations. Previous research showed that UV-Vis reflectance spectrometry can be used for non-contact monitoring of turbidity (TUR) and Chemical Oxygen Demand (COD), which are two key water quality indicators. Although spectrometer achieve high spectral resolution their limited spatial field of view is problematic for highly inhomogeneous surfaces as is the case wastewater In this study, we obtain beyond state-of-art measurement accuracies by combining machine learning techniques with increased spatial field-of-view Multi-Spectral Imaging (MSI) whilst substantially reducing the spectral resolution. We designed and built a dedicated setup with a monochromatic camera and an active illumination of thirteen LEDs covering the spectrum range of 200-700 nm. We acquired and calibrated data on 27 samples with different concentrations of TUR and COD. Machine learning regression models were trained and evaluated with the extracted spectra. We tested the Partial Least Square (PLS), Support Vector Machine (SVM) and Random Forest (RF). PLS regression performed best with excellent correlation coefficients (R2 ) of the 0.99 for TUR and 0.93 for COD. We obtained similar results with the SVM algorithm (R2 = 0.99 and 0.92), whilst RF had lower scores (R2 = 0.96 and 0.71).

Preserving privacy is a growing concern in our society where cameras are ubiquitous. In this work, we propose a trainable image acquisition method that removes the sensitive information in the optical domain before it reaches the image sensor. The method benefits from a trainable optical convolution kernel, which transmits the desired information whilst filtering out the sensitive information, making it irretrievable against different privacy attacks in the digital domain. This is in contrast with the current digital privacy-preserving methods that are all vulnerable to direct access attacks. Also, in contrast with most of the previous optical privacy-preserving methods that cannot be trained, our method is data-driven and optimized for the specific application at hand. Moreover, there is no additional computation or power burden on the acquisition system since it works passively in the optical domain and can be even used in conjunction with other privacy-preserving techniques in the digital domain. We demonstrate our new, generic method in several scenarios such as smile or open-mouth detection as the desired attribute while the gender or wearing make-up is filtered out as the sensitive content. Through several experiments, we show that this method is able to reduce around $\mathbf {65}\%$ of sensitive content while causing a negligible reduction in the desired information. Moreover, we tested our method by deep reconstruction attack and confirmed the ineffectiveness of this attack to reconstruct the original sensitive content. This new method has different use cases such as feedback systems for smart TV content or outdoor advertising.

Preserving privacy is a growing concern in our society where sensors and cameras are ubiquitous. In this work, for the first time, we propose a trainable image acquisition method that removes the sensitive identity revealing information in the optical domain before it reaches the image sensor. The method benefits from a trainable optical convolution kernel which transmits the desired information while filters out the sensitive content. As the sensitive content is suppressed before it reaches the image sensor, it does not enter the digital domain therefore is unretrievable by any sort of privacy attack. This is in contrast with the current digital privacy-preserving methods that are all vulnerable to direct access attack. Also, in contrast with the previous optical privacy-preserving methods that cannot be trained, our method is data-driven and optimized for the specific application at hand. Moreover, there is no additional computation, memory, or power burden on the acquisition system since this processing happens passively in the optical domain and can even be used together and on top of the fully digital privacy-preserving systems. The proposed approach is adaptable to different digital neural networks and content. We demonstrate it for several scenarios such as smile detection as the desired attribute while the gender is filtered out as the sensitive content. We trained the optical kernel in conjunction with two adversarial neural networks where the analysis network tries to detect the desired attribute and the adversarial network tries to detect the sensitive content. We show that this method can reduce 65.1% of sensitive content when it is selected to be the gender and it only loses 7.3% of the desired content. Moreover, we reconstruct the original faces using the deep reconstruction method that confirms the ineffectiveness of reconstruction attacks to obtain the sensitive content.

We demonstrate an intelligent multispectral system for various applications. The multispectral camera is handheld ideal for high resolution (5 Mpixel) applications, the system has 16 bands from 365 to 1020 nm that works at 40 fps, and can collect a hyperspectral cube in less the 500 milliseconds. The system uses deep convolution neural networks trained in a supervised learning fashion to characterize up to 7 skin conditions including skin cancer. Given a specific skin image the system characterizes the underline condition along with a confidence score.

Despite the substantial progress made in deep learning in recent years, advanced approaches remain computationally intensive. The trade-off between accuracy and computation time and energy limits their use in real-time applications on low power and other resource-constrained systems. In this paper, we tackle this fundamental challenge by introducing a hybrid optical-digital implementation of a convolutional neural network (CNN) based on engineering of the point spread function (PSF) of an optical imaging system. This is done by coding an imaging aperture such that its PSF replicates a large convolution kernel of the first layer of a pre-trained CNN. As the convolution takes place in the optical domain, it has zero cost in terms of energy consumption and has zero latency independent of the kernel size. Experimental results on two datasets demonstrate that our approach yields more than two orders of magnitude reduction in the computational cost while achieving near-state-of-the-art accuracy, or equivalently, better accuracy at the same computational cost.

Multispectral imagers collect a hypercube of data, where the spatial image is along two-dimensions and the spectral information is in the third. Two main technologies are used for multispectral imaging: sweeping, where the hypercube is built by scanning through different wavelengths or spatial positions and snapshot multispectral spectral imaging, where the 3D cube of images is taken in one shot. Sweeping imaging systems tend to have more lines and better spectral resolutions whilst snapshot cameras are often used for dynamic analysis of scenes. A common method to obtain the hypercube in snapshot imagers is by pixel level filtering on the sensor chip. Pixel level filtering, where the filter is placed directly on the pixels are intergrated into the wafer-level making processing making them difficult to customize. Therefore, these sensors tend to aim for equally spaced spectral lines in order to cover many applications. This results in an often in an unnecessarily large data cube when only a few spectral lines are needed, moreover the spectral lines are not adapted to the specific application. In this work we propose a multispectral camera based on plenoptic imaging, where the filtering is done in a front-end optics module. Our camera has the usual advantages of a snapshot imager, and the added advantage that the spectral lines can be both reduced and tailored to the specific application by customizing the filter. This procedure reduces the hypercube whilst keeping performance by selecting the relevant data. Moreover, the filter is interchangeable for different applications The camera presented here is built with off-the-shelf components, shows >40 spectral channels, image sizes are 260x260 pixels, with pixel limited spatial resolution. We demonstrate this technology by fruit quality control using machine learning algorithms.

We investigate the performance of wavelet shrinkage methods for the denoising of symmetric- ${\alpha }$ -stable ( $\text{S} {\alpha }\text{S}$ ) self-similar stochastic processes corrupted by additive white Gaussian noise (AWGN), where $ {\alpha }$ is tied to the sparsity of the process. The wavelet transform is assumed to be orthonormal and the shrinkage function minimizes the mean-square approximation error (MMSE estimator). We derive the corresponding formula for the expected value of the averaged estimation error. We show that the predicted MMSE is a monotone function of a simple criterion that depends on the wavelet and the statistical parameters of the process. Using the calculus of variations, we then optimize this criterion to find the best performing wavelet within the extended family of Meyer wavelets, which are bandlimited. These are compared with the Daubechies wavelets, which are compactly supported in time. We find that the wavelets that are shorter in time (in particular, the Haar basis) are better suited to denoise the sparser processes (say, $ {\alpha <1.2}$ ), while the bandlimited ones (including the Held and Shannon wavelets) offer the best performance for $ {\alpha >1.6}$ , the limit corresponding to the Gaussian case (fBm) with $ {\alpha =2}$ .

We establish in the world of stochastic processes a theoretical relation between sparsity and wavelets. The underlying principle is to treat stochastic processes as generalized functions, which facilitates the study of their properties in a transform domain. We focus on symmetric-$\alpha$-stable (S$\alpha$S) processes, with $\alpha\in(0,2]$. They are central to a recently proposed framework for sparse stochastic processes. The case $0<\alpha<2$ corresponds to heavy-tail processes and, thus, to sparse signals with the level of sparsity being more pronounced for smaller values of $\alpha$. The limit case $\alpha=2$ yields the classical Gaussian processes, which are not considered to be sparse. In the first part of the thesis, we identify a particular class of wavelets and show that they provide an independent-component analysis for S$\alpha$S AR(1) signals, which form an important subclass of sparse processes. This contribution can be considered as the first theoretical result ever found about the optimality of wavelets in a stochastic framework. It is remarkable that, by decreasing the parameter $\alpha$, we continuously run the gamut from Fourier (for $\alpha=2$) to wavelets (for $\alpha\le1$), thus providing a unifying view that sees Fourier and wavelet transforms as two extremes of a single spectrum. In the second part of the thesis, we study the wavelet decomposition of self-similar processes---which form a second important family of sparse S$\alpha$S processes. In particular, in the context of the wavelet-based denoising of a stochastic process, we are able to theoretically predict the exact performance of an arbitrary orthonormal wavelet basis. As it turns out, our prediction takes a tractable simple form, which allows us to design the optimal Meyer wavelet as an example of application. Our theoretical results suggest that the performance depends crucially on the localization of wavelets. Consequently, in an image-processing context, we introduce a moment-based measure of localization and propose an optimization framework to design better wavelets. We end the thesis with the experimental confirmation that the proposed wavelets outperform the previously existing ones.

We propose a new statistical dictionary learning algorithm for sparse signals that is based on an α-stable innovation model. The parameters of the underlying model—that is, the atoms of the dictionary, the sparsity index α and the dispersion of the transform-domain coefficients—are recovered using a new type of probability distribution tomography. Specifically, we drive our estimator with a series of random projections of the data, which results in an efficient algorithm. Moreover, since the projections are achieved using linear combinations, we can invoke the generalized central limit theorem to justify the use of our method for sparse signals that are not necessarily α-stable. We evaluate our algorithm by performing two types of experiments: image in-painting and image denoising. In both cases, we find that our approach is competitive with state-of-the-art dictionary learning techniques. Beyond the algorithm itself, two aspects of this study are interesting in their own right. The first is our statistical formulation of the problem, which unifies the topics of dictionary learning and independent component analysis. The second is a generalization of a classical theorem about isometries of $ l _{ p } $ -norms that constitutes the foundation of our approach.

We propose two minimum-mean-square-error (MMSE) estimation methods for denoising non-Gaussian first-order auto-regressive (AR(1)) processes. The first one is based on the message passing framework and gives the exact theoretic MMSE estimator. The second is an iterative algorithm that combines standard wavelet-based thresholding with an optimized non-linearity and cycle-spinning. This method is more computationally efficient than the former and appears to provide the same optimal denoising results in practice. We illustrate the superior performance of both methods through numerical simulations by comparing them with other well-known denoising schemes.

A crucial component of steerable wavelets is the radial profile of the generating function in the frequency domain. In this paper, we present an infinite-dimensional optimization scheme that helps us find the optimal profile for a given criterion over the space of tight frames. We consider two classes of criteria that measure the localization of the wavelet. The first class specifies the spatial localization of the wavelet profile, and the second that of the resulting wavelet coefficients. From these metrics and the proposed algorithm, we construct tight wavelet frames that are optimally localized and provide their analytical expression. In particular, one of the considered criterion helps us finding back the popular Simoncelli wavelet profile. Finally, the investigation of local orientation estimation, image reconstruction from detected contours in the wavelet domain, and denoising indicate that optimizing wavelet localization improves the performance of steerable wavelets, since our new wavelets outperform the traditional ones.

The discrete cosine transform (DCT) is known to be asymptotically equivalent to the Karhunen-Loève transform (KLT) of Gaussian first-order auto-regressive (AR(1)) processes. Since being uncorrelated under the Gaussian hypothesis is synonymous with independence, it also yields an independent-component analysis (ICA) of such signals. In this paper, we present a constructive non-Gaussian generalization of this result: the characterization of the optimal orthogonal transform (ICA) for the family of symmetric-stable AR(1) processes. The degree of sparsity of these processes is controlled by the stability parameter 0 <; α ≤ 2 with the only non-sparse member of the family being the classical Gaussian AR(1) process with α = 2. Specifically, we prove that, for α <; 2, a fixed family of operator-like wavelet bases systematically outperforms the DCT in terms of compression and denoising ability. The effect is quantified with the help of two performance criteria (one based on the Kullback-Leibler divergence, and the other on Stein's formula for the minimum estimation error) that can also be viewed as statistical measures of independence. Finally, we observe that, for the sparser kind of processes with 0 <; α ≤ 1, the operator-like wavelet basis, as dictated by linear system theory, is undistinguishable from the ICA solution obtained through numerical optimization. Our framework offers a unified view that encompasses sinusoidal transforms such as the DCT and a family of orthogonal Haar-like wavelets that is linked analytically to the underlying signal model.

In this letter, we aim to identify the optimal isotropic mother wavelet for a given spatial dimension based on a localization criterion. Within the framework of the calculus of variations, we specify an Euler-Lagrange equation for this problem, and we find the unique analytic solutions. In the one- and two-dimensional cases, the derived wavelets are well known.

Reference EPFL-ARTICLE-211511 URL: http://bigwww.epfl.ch/publications/unser1403.html Record created on 2015-09-18, modified on 2017-05-10

We study the issue of localization in the context of isotropic wavelet frames. We define a variance-type measure of localization and propose an algorithm based on calculus of variations to minimize this criterion under the constraint of a tight wavelet frame. Based on these calculations, we design the variance-optimal wavelet (VOW). Finally, we demonstrate the advantage of better localization in a practical image-processing task.

Sinusoidal transforms such as the DCT are known to be optimal-that is, asymptotically equivalent to the Karhunen-Loève transform (KLT)-for the representation of Gaussian stationary processes, including the classical AR(1) processes. While the KLT remains applicable for non-Gaussian signals, it loses optimality and, is outperformed by the independent-component analysis (ICA), which aims at producing the most-decoupled representation. In this paper, we consider an extension of the classical AR(1) model that is driven by symmetric-alpha-stable (SαS) noise which is either Gaussian (α = 2) or sparse (0 <; α <; 2). For the sparse (non-Gaussian) regime, we prove that an expansion in a proper wavelet basis (including the Haar transform) is much closer to the optimal orthogonal ICA solution than the classical Fourier-type representations. Our criterion for optimality, which favors independence, is the Kullback-Leibler divergence between the joint pdf of the original signal and the product of the marginals in the transformed domain. We also observe that, for very sparse AR(1) processes (α ≤ 1), the operator-like wavelet transform is indistinguishable from the ICA solution that is determined through numerical optimization.