We present a novel approach, Augmented Photogrammetry, for scanning and editing the appearance of physical objects in augmented reality (AR). Our work provides a user-friendly and efficient technique for enabling customizable appearance modifications in real time on arbitrary objects scanned from a user’s physical environment. We accomplish this by integrating Structure from Motion (SfM), instance segmentation, and machine learning into a unified pipeline. Our streamlined process enables users to easily select a physical object and specify its desired appearance. We believe our mobile AR approach holds promise for applications in interior design, virtual prototyping, and content creation.

Synthetic data is highly useful for training machine learning systems performing image-based 3D reconstruction, as synthetic data has applications in both extending existing generalizable datasets and being tailored to train neural networks for specific learning tasks of interest. In this paper, we introduce and utilize a synthetic data generation suite capable of generating data given existing 3D scene models as input. Specifically, we use our tool to generate image sequences for use with Multi-View Stereo (MVS), moving a camera through the virtual space according to user-chosen camera parameters. We evaluate how the given camera parameters and type of 3D environment affect how applicable the generated image sequences are to the MVS task using five pre-trained neural networks on image sequences generated from three different 3D scene datasets. We obtain generated predictions for each combination of parameter value and input image sequence, using standard error metrics to analyze the differences in depth predictions on image sequences across 3D datasets, parameters, and networks. Among other results, we find that camera height and vertical camera viewing angle are the parameters that cause the most variation in depth prediction errors on these image sequences.