Subspace clustering has been extensively studied from the hypothesis-and-test, algebraic, and spectral clustering-based perspectives. Most assume that only a single type/class of subspace is present. Generalizations to multiple types are non-trivial, plagued by challenges such as choice of types and numbers of models, sampling imbalance and parameter tuning. In many real world problems, data may not lie perfectly on a linear subspace and hand designed linear subspace models may not fit into these situations. In this work, we formulate the multi-type subspace clustering problem as one of learning non-linear subspace filters via deep multi-layer perceptrons (mlps). The response to the learnt subspace filters serve as the feature embedding that is clustering-friendly, i.e., points of the same clusters will be embedded closer together through the network. For inference, we apply K-means to the network output to cluster the data. Experiments are carried out on synthetic data and real world motion segmentation problems, producing state-of-the-art results.

A few methods have been proposed to estimate the CRF from a single image, however most of them tend to fail in handling general real images. For instance, EdgeCRF based on patches extracted from colour edges works effectively only when the presence of noise is insignificant, which is not the case for many real images; and, CRFNet, a recent method based on fully supervised deep learning works only for the CRFs that are in the training data, and hence fail to deal with other possible CRFs beyond the training data. To address these problems, we introduce a non-deep-learning method using prediction consistency and gradual refinement. First, we rely more on the patches of the input image that provide more consistent predictions. If the predictions from a patch are more consistent, it means that the patch is likely to be less affected by noise or any inferior colour combinations, and hence, it can be more reliable for CRF estimation. Second, we employ a gradual refinement scheme in which we start from a simple CRF model to generate a result which is more robust to noise but less accurate, and then we gradually increase the model’s complexity to improve the estimation. This is because a simple model, while being less accurate, overfits less to noise than a complex model does. Our experiments confirm that our method outperforms the existing single-image methods for both daytime and nighttime real images.

Many real-world video sequences cannot be conveniently categorized as general or degenerate; in such cases, imposing a false dichotomy in using the fundamental matrix or homography model for motion segmentation on video sequences would lead to difficulty. Even when we are confronted with a general scene-motion, the fundamental matrix approach as a model for motion segmentation still suffers from several defects, which we discuss in this paper. The full potential of the fundamental matrix approach could only be realized if we judiciously harness information from the simpler homography model. From these considerations, we propose a multi-model spectral clustering framework that synergistically combines multiple models (homography and fundamental matrix) together. We show that the performance can be substantially improved in this way. For general motion segmentation tasks, the number of independently moving objects is often unknown a priori and needs to be estimated from the observations. This is referred to as model selection and it is essentially still an open research problem. In this work, we propose a set of model selection criteria balancing data f idelity and model complexity. We perform extensive testing on existing motion segmentation datasets with both segmentation and model selection tasks, achieving state-of-the-art performance on all of them; we also put forth a more realistic and challenging dataset adapted from the KITTI benchmark, containing real-world effects such as strong perspectives and strong forward translations not seen in the traditional datasets.

Nighttime stereo depth estimation is still challenging, as assumptions associated with daytime lighting conditions do not hold any longer. Nighttime is not only about low-light and dense noise, but also about glow/glare, flares, non-uniform distribution of light, etc. One of the possible solutions is to train a network on night stereo images in a fully supervised manner. However, to obtain proper disparity ground-truths that are dense, independent from glare/glow, and have sufficiently far depth ranges is extremely intractable. To address the problem, we introduce a network joining day/night translation and stereo. In training the network, our method does not require ground-truth disparities of the night images, or paired day/night images. We utilize a translation network that can render realistic night stereo images from day stereo images. We then train a stereo network on the rendered night stereo images using the available disparity supervision from the corresponding day stereo images, and simultaneously also train the day/night translation network. We handle the fake depth problem, which occurs due to the unsupervised/unpaired translation, for light effects (e.g., glow/glare) and un-informative regions (e.g., low-light and saturated regions), by adding structure-preservation and weighted-smoothness constraints. Our experiments show that our method outperforms the baseline methods on night images.

Despite the rapid progress in discrete energy minimization, certain problems involving high connectivity and a high number of labels are considered very hard but are still very relevant in computer vision. We propose a post-processing technique to improve the sub-optimal results of the existing methods on such problems. Our core contribution is a mapping between the binary min-cut problem and finding the shortest path in a directed acyclic graph. Using this mapping, we present an algorithm to find an approximate solution for the min-cut problem. We also extend the same idea for multi-label factor-graphs in the form of an iterative move-making algorithm. The proposed algorithm is extremely fast, yet outperforms the existing techniques in terms of accuracy as well as the computational time. We demonstrate competitive or better results on problems where already high-quality work is done.

The Raffles' pitcher plant (Nepenthes rafflesiana Jack, Nepenthaceae) is a carnivorous plant found naturally across multiple habitat types both within and outside of the Central Catchment Nature Reserve (CCNR)-the largest contiguous forested habitat in Singapore. Nepenthes rafflesiana produces lower and upper pitchers during distinct ontogenetic stages, and both of these pitcher types trap highly diverse but compositionally differentiated spectra of invertebrate prey. The fluid-filled pitchers of N. rafflesiana are also habitats for many specialised aquatic invertebrate species, known as inquilines. However, neither the prey spectra nor inquiline communities of N rafflesiana pitchers are well characterised to a high taxonomic resolution. In this study, we surveyed pitcher-prey assemblages and inquiline communities in 78 N. rafflesiana pitchers within and outside of the CCNR in Singapore, as well as the plant communities in the 10 x 10 m plots in which the N. rafflesiana plants were documented. Plant communities of N. naifiesiana plots within the CCNR were found to be much richer in species and differed compositionally from those outside of the CCNR. Inquiline conununities, too, were observed to differ compositionally between pitchers within and outside of the CCNR, as well as between pitcher types. However, asymptotic inquiline species richness was not significantly lower in pitchers outside of the CCNR, as compared to those within it. Lower pitchers generally contained more inquiline species than upper pitchers. Finally, prey assemblages differ in composition only marginally between pitchers within and outside of the CCNR, but were strongly differentiated between pitcher types.

We provide a theoretical proof showing that under a proportional noise model, the discrete eight point algorithm behaves similarly to the differential eight point algorithm when the motion is small. This implies that the discrete algorithm can handle arbitrarily small motion for a general scene, as long as the noise decreases proportionally with the amount of image motion and the proportionality constant is small enough. This stability result extends to all normalized variants of the eight point algorithm. Using simulations, we show that given arbitrarily small motions and proportional noise regime, the normalized eight point algorithms outperform their differential counterparts by a large margin. Using real data, we show that in practical small motion problems involving optical flow, these discrete structure from motion (SFM) algorithms also provide better estimates than their differential counterparts, even when the motion magnitudes reach sub-pixel level. The better performance of these normalized discrete variants means that there is much to recommend them as differential SFM algorithms that are linear and normalized. The support of the DSO grant R263-000-400-592 is gratefully acknowledged. W.-Y. Lin ( ) · L.-F. Cheong Electrical and Computer Engineering Department, National University of Singapore, 4 Engineering Drive 3, Singapore 117576, Singapore e-mail: linwenyan@yahoo.com L.-F. Cheong e-mail: eleclf@nus.edu.sg G.-C. Tan Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore 637371, Singapore e-mail: gctan@ntu.edu.sg

Many methods have set state-of-the-art performance on restoring images degraded by bad weather such as rain, haze, fog, and snow, however they are designed specifically to handle one type of degradation. In this paper, we propose a method that can handle multiple bad weather degradations: rain, fog, snow and adherent raindrops using a single network. To achieve this, we first design a generator with multiple task-specific encoders, each of which is associated with a particular bad weather degradation type. We utilize a neural architecture search to optimally process the image features extracted from all encoders. Subsequently, to convert degraded image features to clean background features, we introduce a series of tensor-based operations encapsulating the underlying physics principles behind the formation of rain, fog, snow and adherent raindrops. These operations serve as the basic building blocks for our architectural search. Finally, our discriminator simultaneously assesses the correctness and classifies the degradation type of the restored image. We design a novel adversarial learning scheme that only backpropagates the loss of a degradation type to the respective task-specific encoder. Despite being designed to handle different types of bad weather, extensive experiments demonstrate that our method performs competitively to the individual and dedicated state-of-the-art image restoration methods.

Given a target prior information, our goal is to propose a method for automatically separating targets of interests from the background in hyperspectral imagery. More precisely, we regard the given hyperspectral image (HSI) as being made up of the sum of low-rank background HSI and a sparse target HSI that contains the targets based on a pre-learned target dictionary constructed from some online spectral libraries. Based on the proposed method, two strategies are briefly outlined and evaluated to realize the target detection on both synthetic and real experiments.

Self-calibration of camera intrinsics and radial distortion has a long history of research in the computer vision community. However, it remains rare to see real applications of such techniques to modern Simultaneous Localization And Mapping (SLAM) systems, especially in driving scenarios. In this paper, we revisit the geometric approach to this problem, and provide a theoretical proof that explicitly shows the ambiguity between radial distortion and scene depth when two-view geometry is used to self-calibrate the radial distortion. In view of such geometric degeneracy, we propose a learning approach that trains a convolutional neural network (CNN) on a large amount of synthetic data. We demonstrate the utility of our proposed method by applying it as a checkerboard-free calibration tool for SLAM, achieving comparable or superior performance to previous learning and hand-crafted methods.

Multi-model fitting has been extensively studied from the random sampling and clustering perspectives. Most assume that only a single type/class of model is present and their generalizations to fitting multiple types of models/structures simultaneously are non-trivial. The inherent challenges include choice of types and numbers of models, sampling imbalance and parameter tuning, all of which render conventional approaches ineffective. In this work, we formulate the multi-model multi-type fitting problem as one of learning deep feature embedding that is clustering-friendly. In other words, points of the same clusters are embedded closer together through the network. For inference, we apply K-means to cluster the data in the embedded feature space and model selection is enabled by analyzing the K-means residuals. Experiments are carried out on both synthetic and real world multi-type fitting datasets, producing state-of-the-art results. Comparisons are also made on single-type multi-model fitting tasks with promising results as well.

The contents of 147 pitchers from the pitcher plant Nepenthes ampullaria were sampled from three sites across Singapore. The primary aim of the study was to compile a comprehensive checklist of the inquiline species inhabiting Nepenthes ampullaria pitchers across the key Nepenthes habitat types in Singapore, with the secondary aims of compiling a preliminary checklist of Nepenthes ampullaria prey species, and identifying structural or floristic habitat characteristics associated with reproducing populations of Nepenthes ampullaria plants. The data suggested that Nepenthes ampullaria flowering occurs more regularly in habitats with little canopy cover—such as in urban, resam-dominated scrub, or in large treefall gaps. A total of 23 inquiline morphospecies and 72 prey morphospecies were observed, and 20 of these inquiline morphospecies could be confidently identified at least to the genus level. This also included an undescribed genus of histiostomatid mite, and one possibly undescribed taxon from the hoverfly genus Nepenthosyrphus. Among the prey species, the ant Meranoplus malaysianus represents a first record in Singapore. Inquiline diversity was not equally distributed across all samples, but was much higher in the Nee Soon Swamp Forest than in Kent Ridge Park and the urban sites, with 12 inquiline species being found only in the Nee Soon Swamp Forest. However, abiotic conditions of pitchers (fluid pH, volume and depth, pitcher dimensions and sediment volumes in pitchers) differed little between pitchers from the three different sites, thus suggesting that abiotic conditions of the larger environment, such as the absence of adult food resources in the surrounding vegetation, and/or dispersal limitations, such as the anthropogenic barriers separating inquiline source populations in the Nee Soon Swamp Forest from Kent Ridge Park and urban vegetation patches, are more important in determining inquiline community composition in Nepenthes ampullaria phytotelmata.

Subspace clustering has been extensively studied from the hypothesis-and-test, algebraic, and spectral clustering based perspectives. Most assume that only a single type/class of subspace is present. Generalizations to multiple types are non-trivial, plagued by challenges such as choice of types and numbers of models, sampling imbalance and parameter tuning. In this work, we formulate the multi-type subspace clustering problem as one of learning non-linear subspace filters via deep multi-layer perceptrons (mlps). The response to the learnt subspace filters serve as the feature embedding that is clustering-friendly, i.e., points of the same clusters will be embedded closer together through the network. For inference, we apply K-means to the network output to cluster the data. Experiments are carried out on both synthetic and real world multi-type fitting problems, producing state-of-the-art results.

An exact method of correcting the rolling shutter (RS) effect requires recovering the underlying geometry, i.e. the scene structures and the camera motions between scanlines or between views. However, the multiple-view geometry for RS cameras is much more complicated than its global shutter (GS) counterpart, with various degeneracies. In this paper, we first make a theoretical contribution by showing that RS two-view geometry is degenerate in the case of pure translational camera motion. In view of the complex RS geometry, we then propose a Convolutional Neural Network (CNN)-based method which learns the underlying geometry (camera motion and scene structure) from just a single RS image and perform RS image correction. We call our method structure-and-motion-aware RS correction because it reasons about the concealed motions between the scan-lines as well as the scene structure. Our method learns from a large-scale dataset synthesized in a geometrically meaningful way where the RS effect is generated in a manner consistent with the camera motion and scene structure. In extensive experiments, our method achieves superior performance compared to other state-of-the-art methods for single image RS correction and subsequent Structure from Motion (SfM) applications.

Optical flow in heavy rainy scenes is challenging due to the presence of both rain steaks and rain veiling effect, which break the existing optical flow constraints. Concerning this, we propose a deep-learning based optical flow method designed to handle heavy rain. We introduce a feature multiplier in our network that transforms the features of an image affected by the rain veiling effect into features that are less affected by it, which we call veiling-invariant features. We establish a new mapping operation in the feature space to produce streak-invariant features. The operation is based on a feature pyramid structure of the input images, and the basic idea is to preserve the chromatic features of the background scenes while canceling the rain-streak patterns. Both the veiling-invariant and streak-invariant features are computed and optimized automatically based on the the accuracy of our optical flow estimation. Our network is end-to-end, and handles both rain streaks and the veiling effect in an integrated framework. Extensive experiments show the effectiveness of our method, which outperforms the state of the art method and other baseline methods. We also show that our network can robustly maintain good performance on clean (no rain) images even though it is trained under rain image data.

We propose a new video representation in terms of an over-segmentation of dense trajectories covering the whole video. Trajectories are often used to encode long-temporal information in several computer vision applications. Similar to temporal superpixels, a temporal slice of super-trajectories are superpixels, but the later contains more information because it maintains the long dense pixel-wise tracking information as well. The main challenge in using trajectories for any application, is the accumulation of tracking error in the trajectory construction. For our problem, this results in disconnected superpixels. We exploit constraints for edges in addition to trajectory based color and position similarity. Analogous to superpixels as a preprocessing tool for images, the proposed representation has its applications for videos, especially in trajectory based video analysis.

The ability to identify the static background in videos captured by a moving camera is an important pre-requisite for many video applications (e.g. video stabilization, stitching, and segmentation). Existing methods usually face difficulties when the foreground objects occupy a larger area than the background in the image. Many methods also cannot scale up to handle densely sampled feature trajectories. In this paper, we propose an efficient local-to-global method to identify background, based on the assumption that as long as there is sufficient camera motion, the cumulative background features will have the largest amount of trajectories. Our motion model at the two-frame level is based on the epipolar geometry so that there will be no over-segmentation problem, another issue that plagues the 2D motion segmentation approach. Foreground objects erroneously labelled due to intermittent motions are also taken care of by checking their global consistency with the final estimated background motion. Lastly, by virtue of its efficiency, our method can deal with densely sampled trajectories. It outperforms several state-of-the-art motion segmentation methods on public datasets, both quantitatively and qualitatively.

Most deraining works focus on rain streaks removal but they cannot deal adequately with heavy rain images. In heavy rain, streaks are strongly visible, dense rain accumulation or rain veiling effect significantly washes out the image, further scenes are relatively more blurry, etc. In this paper, we propose a novel method to address these problems. We put forth a 2-stage network: a physics-based backbone followed by a depth-guided GAN refinement. The first stage estimates the rain streaks, the transmission, and the atmospheric light governed by the underlying physics. To tease out these components more reliably, a guided filtering framework is used to decompose the image into its low- and high-frequency components. This filtering is guided by a rain-free residue image --- its content is used to set the passbands for the two channels in a spatially-variant manner so that the background details do not get mixed up with the rain-streaks. For the second stage, the refinement stage, we put forth a depth-guided GAN to recover the background details failed to be retrieved by the first stage, as well as correcting artefacts introduced by that stage. We have evaluated our method against the state of the art methods. Extensive experiments show that our method outperforms them on real rain image data, recovering visually clean images with good details.

In this work, a novel target detector for hyperspectral imagery is developed. The detector is independent on the unknown covariance matrix, behaves well in large dimensions, distributional free, invariant to atmospheric effects, and does not require a background dictionary to be constructed. Based on a modification of the robust principal component analysis (RPCA), a given hyperspectral image (HSI) is regarded as being made up of the sum of a low-rank background HSI and a sparse target HSI that contains the targets based on a pre-learned target dictionary specified by the user. The sparse component is directly used for the detection, that is, the targets are simply detected at the non-zero entries of the sparse target HSI. Hence, a novel target detector is developed, which is simply a sparse HSI generated automatically from the original HSI, but containing only the targets with the background is suppressed. The detector is evaluated on real experiments, and the results of which demonstrate its effectiveness for hyperspectral target detection especially when the targets are well matched to the surroundings.