The problem of robust matrix completion—the recovery of a low-rank matrix and a sparse matrix from a sampling of their superposition—has been addressed extensively in prior literature. Yet, much of this work has focused exclusively on the case in which the matrix sampling is done at random, as this scenario is amenable to theoretical analysis. In contrast, sampling with an arbitrary deterministic pattern is often more accommodating to hardware implementation; consequently, the problem of robust matrix completion under deterministic sampling is considered. To this end, a restricted approximate isometry property is proposed and used, along with a modified golfing scheme and a slightly strengthened incoherence condition, to prove that the latent low-rank and sparse matrices are uniquely recoverable via convex optimization with asymptotically high probability, providing the first exact-recovery theory for robust matrix completion with arbitrary deterministic sampling. A corresponding convex-optimization algorithm, driven by a traditional nuclear norm, is developed and then subsequently generalized by substituting a convolutional nuclear norm in order to cover a broader range of application scenarios. Empirical experiments on synthetic data verify the proposed theory while a battery of results on real-world images demonstrate the practical efficacy of the generalized algorithm for robust matrix recovery.

While deep learning has been explored extensively for the classification of hyperspectral images, many prior techniques are limited to consideration of features arising in a single domain or from only a spatially local context. In contrast, a multi-branch deep-network architecture is proposed to fully exploit the complementary information arising from local spatial spectral features, global non-local spatial graph structures, as well as the local frequency domain. The proposed network consists of an initial spatial-spectral feature modeling within local cubic patches, followed by subsequent parallel branches that simultaneously extract local frequency-domain features as well as graph representations that capture global spatial topological relationships among non-local pixels. Feature fusion combines the information from the two branches to result in the final classification map. Experimental results reveal that the proposed network significantly outperforms existing methods for the classification of hyperspectral images on several public datasets, particularly for regions in which complex textures or boundary details are abundant.

Robust principal component analysis (RPCA) seeks a low-rank component and a sparse component from their summation. Yet, in many applications of interest, the sparse foreground actually replaces, or occludes, elements from the low-rank background. To address this mismatch, a new framework is proposed in which the sparse component is identified indirectly through determining its support. This approach, called robust principal component completion (RPCC), is solved via variational Bayesian inference applied to a fully probabilistic Bayesian sparse tensor factorization. Convergence to a hard classifier for the support is shown, thereby eliminating the post-hoc thresholding required of most prior RPCA-driven approaches. Experimental results reveal that the proposed approach delivers near-optimal estimates on synthetic data as well as robust foreground-extraction and anomaly-detection performance on real color video and hyperspectral datasets, respectively. Source implementation and Appendices are available at https://github.com/WongYinJ/BCP-RPCC.

Diffusion models have recently risen to prominence for a variety of inverse-imaging problems. Many such models use what is commonly known as conditional diffusion which effectively samples from the distribution of the desired image conditioned on some known side information, often a degraded or lower-resolution version of the target image. However, an alternative paradigm has recently emerged in the form of constrained diffusion in which explicit constraints between the target image and the side information are iteratively incorporated into the diffusion reconstruction during inference. While prior literature has considered conditional and constrained diffusion to effectively be mutually exclusive, a diffusion algorithm is proposed to combine the two within the widely-used denoising diffusion probabilistic models (DDPM) framework. The resulting approach—constrained conditional denoising diffusion—inputs both the target and side information into the diffusion network during both training and inference similar to conditional diffusion but also applies explicit constraints during inference like constrained diffusion. The proposed approach is evaluated for the task of fusing a hyperspectral image, possessing high spectral resolution, with a multispectral image, having high spatial resolution, to yield an image with high resolution both spatially and spectrally, an inverse-imaging problem called hyperspectral-multispectral fusion. Experimental results demonstrate that, not only can constrained and conditional diffusion operate complementarily and achieve performance superior to either used alone, but also that the proposed constrained conditional denoising diffusion outperforms other state-of-the-art approaches for hyperspectral-multispectral fusion.

While snapshot hyperspectral cameras are cheaper and faster than imagers based on pushbroom or whiskbroom spatial scanning, the output imagery from a snapshot camera typically has different spectral bands mapped to different spatial locations in a mosaic pattern, requiring a demosaicing process to be applied to generate the desired hyperspectral image with full spatial and spectral resolution. However, many existing demosaicing algorithms suffer common artifacts such as periodic striping or other forms of noise. To ameliorate these issues, a hyperspectral demosaicing framework that couples a preliminary demosaicing network with a separate multi-stage progressive denoising network is proposed, with both networks employing transformer and attention mechanisms. A multi-term loss function permits supervised network training to monitor not only performance of the preliminary demosaicing but also denoising at each stage. An extensive collection of experimental results demonstrate that the proposed approach produces demosaiced images with not only fewer visual artifacts but also improved performance with respect to several quantitative measures as compared to other state-of-the-art demosaicing methods from recent literature.

Hyperspectral super-resolution is commonly accomplished by the fusing of a hyperspectral imaging of low spatial resolution with a multispectral image of high spatial resolution, and many tensor-based approaches to this task have been recently proposed. Yet, it is assumed in such tensor-based methods that the spatial-blurring operation that creates the observed hyperspectral image from the desired super-resolved image is separable into independent horizontal and vertical blurring. Recent work has argued that such separable spatial degradation is ill-equipped to model the operation of real sensors which may exhibit, for example, anisotropic blurring. To accommodate this fact, a generalized tensor formulation based on a Kronecker decomposition is proposed to handle any general spatial-degradation matrix, including those that are not separable as previously assumed. Analysis of the generalized formulation reveals conditions under which exact recovery of the desired super-resolved image is guaranteed, and a practical algorithm for such recovery, driven by a blockwise-group-sparsity regularization, is proposed. Extensive experimental results demonstrate that the proposed generalized tensor approach outperforms not only traditional matrix-based techniques but also state-of-the-art tensor-based methods; the gains with respect to the latter are especially significant in cases of anisotropic spatial blurring.

Deep learning is being increasingly employed for hyperspectral classification, although such use is often predicated on the availability of a sufficiently large set of labeled samples for training. To improve classification performance under a limited training-set size, a semi-supervised network with end-to-end local–global active learning (AL) based on graph convolutional networks (GCNs) is proposed. The proposed AL extracts both global as well as local graph-based features to gauge the discriminative information in unlabeled samples, while semi-supervised classification expands the training set of a fully supervised classifier by attaching pseudo-labels to high-confidence unlabeled samples. Experimental results demonstrate that the proposed network outperforms not only other approaches to semi-supervised classification but also several existing fully supervised methods. The source code of this method can be found at https://github.com/XtaoS/semi-LG-AGCN .

Convolutional networks have been widely used for the classification of hyperspectral images; however, such networks are notorious for their large number of trainable parameters and high computational complexity. Additionally, traditional convolution-based methods are typically implemented as a simple cascade of a number of convolutions using a single-scale convolution kernel. In contrast, a lightweight multiscale convolutional network is proposed, capitalizing on feature extraction at multiple scales in parallel branches followed by feature fusion. In this approach, 2-D depthwise convolution is used instead of conventional convolution to reduce network complexity without sacrificing classification accuracy. Furthermore, multiscale channel attention (MSCA) is also employed to selectively exploit discriminative capability across various channels. To do so, multiple 1-D convolutions with varying kernel sizes provide channel attention at multiple scales, again with the goal of minimizing network complexity. Experimental results reveal that the proposed network not only outperforms other competing lightweight classifiers in terms of classification accuracy, but also exhibits a lower number of parameters as well as significantly less computational cost.

Twenty-five years ago, the field of computational imaging arguably did not exist, at least not as a standalone arena of research activity and technical development. Of course, the idea of using computation to form images had been around for several decades, largely thanks to the development of medical imaging-such as magnetic resonance imaging (MRI) and X-ray tomography-in the 1970s and synthetic-aperture radar (SAR) even earlier. Yet, a quarter of a century ago, such technologies would have been considered to be a subfocus of the wider field of image processing. This view started to change, however, in the late 1990s with a series of innovations that established computational imaging as a scientific and technical pursuit in its own right.

While deep learning (DL) has been widely employed for the classification of hyperspectral imagery (HSI), many scenarios arise in practice in which too few labeled samples exist to effectively train the networks. Few-shot learning has been recently used to deploy classifiers trained on source-domain datasets comprising a large number of labeled samples to datasets from a target domain with only few labeled samples. However, most techniques in this vein effectively assume that the source and target domains possess the same data distribution, whereas the distributions between the two domains often differ widely in practice. Adversarial domain adaption driven by prototype classifiers deployed independently in the source and target domains is proposed to handle such differing source and target distributions, while an attention-based feature extractor with residual skip connections is developed in order to weight spectral bands according to their importance to the hyperspectral classification task. Experimental results demonstrate improved performance for the proposed few-shot-learning framework relative to both fully supervised classifiers as well as other few-shot techniques.

Detection of moving foreground objects is essential to many image-sequence-analysis applications. However, preexisting methods tend to work best when the foreground is visually distinct from the background, suffering when objects are camouflaged. To address this shortcoming, a foreground-extraction algorithm resilient to camouflage is proposed by incorporating a redundant discrete wavelet transform into the well-known DECOLOR technique based on a sparse and low-rank model of the foreground-extraction problem. Detection of camouflaged moving objects is enhanced as a result of the combination of multiple background estimates in independent wavelet subbands into an overall estimate of the background, leveraging the known robustness of redundant wavelet transforms to additive noise. Experimental results demonstrate that the proposed method offers robustness to camouflage superior to that of other competing methods for image sequences containing snow leopards in the wild.

Convolutional neural networks have garnered increasing interest for the supervised classification of hyperspectral imagery. However, images with a wide variety ofspatial land-cover sizes can hinder the feature-extraction ability of traditional convolutional networks. Consequently, many approaches intended to extract multiscale features have emerged; these techniques typically extract features in multiple parallel branches using convolutions of differing kernel sizes with concatenation or addition employed to fuse the features resulting from the various branches. In contrast, the present work explores a multiscale spatial-spectral feature-extraction network that operates in a more granular manner. Specifically, in the proposed network, a multibranch structure expands the convolutional receptive fields through the partitioning of input feature maps, applying hierarchical connections across the partitions, crosschannel feature fusion via pointwise convolution, and depthwise three-dimensional (3-D) convolutions for feature extraction. Experimental results reveal that the proposed multiscale spatial-spectral feature-fusion network outperforms other state-of-the-art networks at the supervised classification of hyperspectral imagery while being robust to limited training data.

Tensor-based fusion that couples the high spatial resolution of a multispectral image (MSI) to the high spectral resolution of a hyperspectral image (HSI) is considered. The fusion problem is first formulated mathematically as a convex optimization of a tensor trace norm imposing low-rank spatially as well as spectrally, with an alternating-directions optimization featuring linearization providing the solution. Although prior tensor-based fusion approaches typically resort to tensor decomposition, the proposed algorithm exploits ideas from the field of tensor completion to directly impose a low-rank property spatially and spectrally while avoiding the computationally complex patch clustering and dictionary learning common to competing fusion techniques. Additionally, small modifications to the basic optimization permit a fusion process robust to missing hyperspectral values such as those that can result from dead stripes in real hyperspectral sensors. The experimental evaluations on both synthetic imagery as well as real imagery demonstrate that the resulting low-rank tensor-approximation (LRTA) fusion algorithm preserves both spatial details and texture, yielding significantly improved image quality when compared to other state-of-the-art fusion methods as well as effective restoration under conditions of missing stripes within the HSI.

Low-rank subspace representations have been observed to be well-suited to hyperspectral imagery, which tends to have a global structure composed of a small number of ground-cover signatures, and additional graph-based regularization can further incorporate local information. However, in the context of unsupervised classification, existing approaches typically limit consideration to simple graphs built on spectral information alone. In contrast, a hypergraph-based low-rank subspace clustering is proposed to capture a more complex manifold structure. In addition, basing the hypergraph on a superpixel segmentation of the image exploits structure that is meaningful both spatially as well as spectrally. The experimental results reveal performance for the proposed superpixel-hypergraph approach superior to that of competing techniques representative of several prominent classes of unsupervised classification for hyperspectral imagery.

Pushbroom acquisition of hyperspectral imagery is prone to striping artifacts in the along-track direction. A hyperspectral destriping algorithm is proposed such that the subbands of a 3-D wavelet transform most affected by pushbroom stripes-namely, those with spatially vertical orientation-are the exclusive focus of destriping. The proposed method features an iterative image decomposition composed of a low-rank model for the stripes coupled with a group-sparse prior on the wavelet coefficients of the subbands in question. While low-rank stripe models have been widely used in the past, they typically have been deployed in conjunction with a total-variation prior on the image that is prone to oversmoothing and residual stripe artifacts. On the other hand, the proposed group-sparse prior not only captures the well-known sparse nature of wavelet coefficients but also capitalizes on their vertical clustering in the subbands in question. In addition, while many prior destriping methods are wavelet-based, they employ 2-D transforms band by band. In contrast, the proposed 3-D wavelet transform provides a greater concentration of stripe information into fewer wavelet coefficients, leading to more effective destriping. Experimental results on both synthetically striped imagery as well as real striped imagery from an actual hyperspectral sensor demonstrate superior image quality for the proposed method as compared with other state-of-the-art methods.

Pushbroom acquisition of hyperspectral imagery is prone to striping artifacts in the along-track direction. A hyperspectral destriping algorithm is proposed such that subbands of a 2D wavelet transform most effected by pushbroom stripes-namely, those with spatially vertical orientation-are the exclusive focus of destriping. The proposed method features an iterative image decomposition composed of a low-rank model for the stripes coupled with a group-sparse prior on the wavelet coefficients of the subbands in question. Experimental results on both synthetically striped imagery demonstrate superior image quality for the proposed method as compared to other state-of-the-art methods.