We present a novel computational puzzle solver for square-piece image jigsaw puzzles with no prior information such as piece orientation or anchor pieces. By "piece" we mean a square $d$d x $d$d block of pixels, where we investigate pieces as small as 7 × 7 pixels. To reconstruct such challenging puzzles, we propose to find maximum geometric consensus between pieces, specifically hierarchical piece loops. The proposed algorithm seeks out loops of four pieces and aggregates the smaller loops into higher order "loops of loops" in a bottom-up fashion. In contrast to previous puzzle solvers which aim to maximize compatibility measures between all pairs of pieces and thus depend heavily on the pairwise compatibility measures used, our approach reduces the dependency on the pairwise compatibility measures which become increasingly uninformative for small scales and instead exploits geometric agreement among pieces. Our contribution also includes an improved pairwise compatibility measure which exploits directional derivative information along adjoining boundaries of the pieces. We verify the proposed algorithm as well as its individual components with mathematical analysis and reconstruction experiments.

In this paper, we present a novel computational puzzle solver for square-piece image jigsaw puzzles with no prior information such as piece orientation, anchor pieces or resulting dimension of the puzzle. By "piece" we mean a square dxd block of pixels, where we investigate pieces as small as 7x7 pixels. To reconstruct such challenging puzzles, we aim to search for piece configurations which maximize the size of consensus (i.e. grid or loop) configurations which represent a geometric consensus or agreement among pieces. Pieces are considered for addition to the existing assemblies if these pieces increase the size of the consensus configurations. In contrast to previous puzzle solvers which goal for assemblies maximizing compatibility measures between all pairs of pieces and thus depend heavily on the pairwise compatibility measure used, our new approach reduces the dependency on the pairwise compatibility measures which become increasingly uninformative at small scales and instead exploits geometric agreement among pieces. Our contribution also includes an improved pairwise compatibility measure which exploits directional derivative information along adjoining boundaries of the pieces. For the challenging unknown orientation piece puzzles where the size of pieces is small, we reduce assembly error by up to 75% compared with previous algorithms for standard datasets.

Figure 4 5 6 7 and 8 show qualitative reconstruction results on the challenging unknown orientation piece puzzles from MIT dataset [1]. We varied the number of pieces and the size of each piece for the experiments. When the size of the piece is small, previous algorithms [2, 3] drastically drop their reconstruction performance whereas our proposed algorithm keeps the performance. Our proposed algorithm reduces up to 75% error from the previous work [3] on the unknown orientation piece puzzles (P=14, K=1064).

This paper proposes a new formulation and solution to image-based 3D modeling (aka “multi-view stereo”) based on generative statistical modeling and inference. The proposed new approach, named statistical inverse ray tracing, models and estimates the occlusion relationship accurately through optimizing a physically sound image generation model based on volumetric ray tracing. Together with geometric priors, they are put together into a Bayesian formulation known as Markov random field (MRF) model. This MRF model is different from typical MRFs used in image analysis in the sense that the ray clique, which models the ray-tracing process, consists of thousands of random variables instead of two to dozens. To handle the computational challenges associated with large clique size, an algorithm with linear computational complexity is developed by exploiting, using dynamic programming, the recursive chain structure of the ray clique. We further demonstrate the benefit of exact modeling and accurate estimation of the occlusion relationship by evaluating the proposed algorithm on several challenging data sets.

We present a novel algorithm based on "loop constraints" for assembling non-overlapping square-piece jigsaw puzzles where the rotation and the position of each piece are unknown. Our algorithm finds small loops of puzzle pieces which form consistent cycles. These small loops are in turn aggregated into higher order "loops of loops" in a bottom-up fashion. In contrast to previous puzzle solvers which avoid or ignore puzzle cycles, we specifically seek out and exploit these loops as a form of outlier rejection. Our algorithm significantly outperforms state-of-the-art algorithms in puzzle reconstruction accuracy. For the most challenging type of image puzzles with unknown piece rotation we reduce the reconstruction error by up to 70%. We determine an upper bound on reconstruction accuracy for various data sets and show that, in some cases, our algorithm nearly matches the upper bound.

Reasonable computation and accurate camera calibration require matching many interest points over long baselines. This is a difficult problem requiring better solutions than presently exist for urban scenes involving large buildings containing many windows since windows in a facade all have the same texture and, therefore, cannot be distinguished from one another based solely on appearance. Hence, the usual approach to feature detection and matching, such as use of SIFT, does not work in these scenes. A novel algorithm is introduced to provide correspondences for multiple repeating feature patterns seen under significant viewpoint changes. Most existing appearance-based algorithms cannot handle highly repetitive textures due to the match location ambiguity. However, the target structure provides a rich set of repeating features to be matched and tracked across multiple views, thus potentially improving camera estimation accuracy. The proposed method also exploits the geometric structure of regular grids of repeating features on planar surfaces.

A new approach is presented to vehicle-class recognition from video clips. Two new concepts introduced are: probes consisting of local 3-d curve-groups which when projected into video frames are features for recognizing vehicle classes in video clips; and Bayesian recognition based on class probability densities for groups of 3-d distances between pairs of 3-d probes. A full Bayesian recognizer is realized via Monte Carlo simulation method. Also, a sub-optimal but robust camera calibration method is employed and tested extensively.

Here we describe the statistical models of shape which will be used to represent objects in images. The shape of an object is represented by a set of n points, which may be in any dimension. Commonly the points are in two or three dimensions. Shape is usually defined as that quality of a configuration of points which is invariant under some transformation. In two or three dimensions we usually consider the Similarity transformation (translation, rotation and scaling). The shape of an object is not changed when it is moved, rotated or scaled. Recent advances in the statistics of shape allow formal statistical techniques to be applied to sets of shapes, making possible analysis of shape differences and changes [32]. Our aim is to derive models which allow us to both analyse new shapes, and to synthesise shapes similar to those in a training set. The training set typically comes from hand annotation of a set of training images, though automatic landmarking systems are being developed (see below). By analysing the variations in shape over the training set, a model is built which can mimic this variation. Much of the following will describe building models of shape in an arbitrary d-dimensional space, under a similarity transform T θ (where θ are the parameters of the transformation). Most examples will be given for two dimensional shapes under the Similarity transformation (with parameters of translation, scaling and orientation), as these are the easiest to represent on the page, and probably the most widely studied. Note however that the dimensions need not always be in space, they can equally be time or intensity in an image. For instance 3D Shapes can either be composed of points in 3D space, or could be points in 2D with a time dimension (for instance in an image sequence) 2D Shapes can either be composed of points in 2D space, or one space and one time dimension 1D Shapes can either be composed of points along a line, or, as is used below, intensity values sampled at particular positions in an image. There an numerous other possibilities. In each case a suitable transformation must be defined (eg Similarity for 2D or global scaling and offset for 1D).

This paper introduces a novel approach for reassembling pot sherds found at archaeological excavation sites, for the purpose of reconstructing clay pots that had been made on a wheel. These pots and the sherds into which they have broken are axially symmetric. The reassembly process can be viewed as 3D puzzle solving or generalized cylinder learning from broken fragments. The estimation exploits both local and semi-global geometric structure, thus making it a fundamental problem of geometry estimation from noisy fragments in computer vision and pattern recognition. The data used are densely digitized 3D laser scans of each fragment's outer surface. The proposed reassembly system is automatic and functions when the pile of available fragments is from one or multiple pots, and even when pieces are missing from any pot. The geometric structure used are curves on the pot along which the surface had broken and the silhouette of a pot with respect to an axis, called axis-profile curve (APC). For reassembling multiple pots with or without missing pieces, our algorithm estimates the APC from each fragment, then reassembles into configurations the ones having distinctive APC. Further growth of configurations is based on adding remaining fragments such that their APC and break curves are consistent with those of a configuration. The method is novel, more robust and handles the largest numbers of fragments to date.

This paper proposes a new method for estimating the symmetric axis of a pottery from its small fragment using surface geometry. For the automatic assembly of broken sherds, the axis estimation is an important measure [2]. When a fragment is small, it is difficult to estimate axis orientation since it looks like a patch of a sphere and conventional methods mostly fail, but the proposed method provides reliable axis estimation by using multiple constraints. The computational cost is also much lowered. To estimate the symmetric axis, the proposed algorithm uses three constraints: (1) the curvature is constant on a circumference C H . (2) the curvature is invariant in any scale. (3) also the principal curvatures does not vary on C H . C H is a planar circle which is one of all the possible circumferences of a pottery or sherd. A hypothesis test for axis is performed using maximum likelihood. The variance of curvature, multi-scale curvature and principal curvatures are computed in the likelihood function. We also show that the principal curvatures can be used for grouping of sherds. The grouping of sherds will reduce the computation significantly by omitting impossible configurations in pottery assembly.

A new method for structure-from-motion (SfM) is developed for moving vehicles on the street using a video camera Outdoor vehicle is one of the most difficult objects for tracking or three-dimensional (3D) reconstruction because most vehicle surfaces are specular and reflect background scenes. Therefore conventional point feature tracking method is not appropriate since spurious features are often tracked and the numbers of the correct features are too small. To overcome such difficulties, epipolar constraint, cross-ratio histogram and 3D curve reconstruction are employed in this method. The proposed method is also computationally efficient since it does not require expensive feature tracking process, which is used in most conventional SfM methods. Finally, to evaluate the error, experiments over 200 vehicles are performed under various viewing conditions and they show significant correlation among the number of frames, distance from the camera and accuracy of reconstruction.

We present an approach to 3D vehicle class recognition (which of SUV, mini-van, sedan, pickup truck) with one or more fixed video-cameras in arbitrary positions with respect to a road. The vehicle motion is assumed to be straight. We propose an efficient method of Structure from Motion (SfM) for camera calibration and 3D reconstruction. 3D geometry such as vehicle and cabin length, width, height, and functions of these are computed and become features for use in a classifier. Classification is done by a minimum probability of error recognizer. Finally, when additional video clips taken elsewhere are available, we design classifiers based on two or more video clips, and this results in significant classification-error reduction.

This paper presents a complete solution to estimating a scene's 3D geometry and appearance from multiple 2D images by using a statistical inverse ray tracing method. Instead of matching image features/pixels across images, the inverse ray tracing approach models the image generation process directly and searches for the best 3D geometry and surface reflectance model to explain all the observations. Here the image generation process is modeled through volumetric ray tracing, where the occlusion/visibility is exactly modeled. All the constraints (including ray constraints and prior knowledge about the geometry) are put into the Ray Markov Random Field (Ray MRF) formulation, developed in [10]. Differently from [10], where the voxel colors are estimated independently of the voxel occupancies, in this work, both voxel occupancies and colors (i.e., both geometry and appearance) are modeled and estimated jointly in the same inversey ray tracing framework (Ray MRF + deep belief propagation) and implemented in a common message passing scheme, which improves the accuracy significantly as verified by extensive experiments. The complete inverse ray tracing approach can better handle difficult problems in multi-view stereo than do traditional methods, including large camera baseline, occlusion, matching ambiguities, color constant or slowly changing regions, etc., without additional information and assumptions, such as initial surface estimate or simple background assumption. A prototype system is built and tested over several challenging datasets and compared with the state-of-the-art systems, which demonstrates its good performance and wide applicability.

REVEAL, (Reconstruction and Exploratory Visualization: Engineering meets Art / ArchaeoLogy) is a four year NSF-funded project promoting paradigm shifts in archaeology, currently at the 1.5 year point This is a project to create an environment for acquiring and presenting archaeological data in a way that streamlines the excavation process and supports and enhances the expert's understanding of the data. REVEAL leverages three aspects of technology: using vision algorithms to speed up or replace measurement and documentation tasks, using computer automation to speed up data entry tasks, using integrated 2D and 3D media to enhance data comprehension. This paper is an update on what the project has accomplished, what has been learned, and what is planned for the rest of the project.

The increasing availability of very high resolution satellite imagery has spurred interest in automatically detecting very fine detailed changes in an area over time, a particularly useful tool for analyzing activity in dense urban areas. However, attempting automated change detection at this resolution is difficult due to the motion parallax of elevated structures. This paper presents a comprehensive solution to change detection in areas of significant 3D relief using a new framework called volumetric appearance modeling (VAM). This approach can manage the complications of unknown and changing world surfaces by maintaining a 3D voxel-based model, where probability distributions for surface occupancy and image appearance are stored in each voxel. These distributions are continuously updated as new images are received using an adaptive learning procedure. This representation is demonstrated to produce accurate change detection results under conditions of variable illumination and viewpoint as well as haze conditions present in satellite imagery. The volumetric representation also supports automatic sensor model correction to align incoming imagery to a common geographic reference. This registration approach is demonstrated to achieve geo-positioning accuracy on the order of the ground sampling distance (GSD) or better.

In this paper, we present an approach to multi-view image-based 3D reconstruction by statistically inversing the ray-tracing based image generation process. The proposed algorithm is fast, accurate and does not need any initialization. The geometric representation is a discrete volume divided into voxels, with each voxel associated with two properties: opacity (shape) and color (appearance). The problem is then formulated as inferring each voxel's most probable opacity and color through MAP estimation of the developed Ray Markov Random Fields (RayMRF). RayMRF is constructed with three kinds of cliques: the usual unary and pairwise cliques favoring connected voxel regions, and most importantly ray-cliques modelling the ray-tracing based image generation process. Each ray-clique connects the voxels that the viewing ray passes through. It provides a principled way of modeling the occlusion without approximation. The inference problem involved in the MAP estimation is handled by an optimized belief propagation algorithm. One unusual structure of the proposed MRF is that each ray-clique usually involves hundreds/thousands of random variables, which seems to make the inference computationally formidable. Thanks to the special property of the ray-clique functional form, we investigate the deep factorization property of ray-clique energy and get a highly efficient algorithm based on the general loopy belief propagation, which has reduced the computational complexity from exponential to linear. Both of the efficient inference algorithm and the overall system concept are new. Combining these results in an algorithm that can reverse the image generation process very fast. 3D surface reconstruction in a 100×100×100, i.e., 106 voxel space with 10 images requires roughly 3 minutes on a 3.0 GHz single-core CPU. The running time grows linearly with respect to the number of voxels and the number of images. And the speed could be further improved with a hierarchical sparse representation of the volume, like octree. Experiments on several standard datasets show the quality and speed of the proposed models and algorithms.

Central to many problems in scene understanding based on using a network of tens, hundreds or even thousands of randomly distributed cameras with on-board processing and wireless communication capability is the “efficient” reconstruction of the 3D geometry structure in the scene. What is meant by “efficient” reconstruction? In this paper we investigate this from different aspects in the context of visual sensor networks and offer a distributed reconstruction algorithm roughly meeting the following goals: 1. Close to achievable 3D reconstruction accuracy and robustness; 2. Minimization of the processing time by adaptive computing-job distribution among all the cameras in the network and asynchronous parallel processing; 3. Communication Optimization and minimization of the (battery-stored) energy, by reducing and localizing the communications between cameras. A volumetric representation of the scene is reconstructed with a shape from apparent contour algorithm, which is suitable for distributed processing because it is essentially a local operation in terms of the involved cameras, and apparent contours are robust to ourdoor illumination conditions. Each camera processes its own image and performs the computation for a small subset of voxels, and updates the voxels through collaborating with its neighbor cameras. By exploring the structure of the reconstruction algorithm, we design the minimum-spanning-tree (MST) message passing protocol in order to minimize the communication. Of interest is that the resulting system is an example of “swarm behavior”. 3D reconstruction is illustrated using two real image sets, running on a single computer. The iterative computations used in the single processor experiment are exactly the same as are those used in the network computations. Distributed concepts and algorithms for network control and communication performance are theoretical designs and estimates.