Joint Triangulation and Mapping via Differentiable Sensor Fusion

The challenge of sensor fusion is prevalent in route planning, robotics, and autonomous vehicles. We leverage automatic differentiation (AD) and probabilistic programming to develop an end-to-end stochastic optimization algorithm for sensor fusion and triangulation of a large number of unknown objects. Our algorithm uses a generative model to train a Expectation Maximization (EM) clustering solver. We validate our method on street sign detections extracted from noisily geo-located street level imagery without depth information by jointly estimating the number and location of objects of different types, together with parameters for sensor noise characteristics and prior distribution of objects. We find that our model is more robust to upstream misclassifications than current methods and generalizes across sign types.