ESPReSSo: Efficient Slanted PatchMatch for Real-Time Spacetime Stereo

We present ESPReSSo, the first real-time implementation of spacetime stereo, offering improved quality vs. existing real-time systems. ESPReSSo is a local stereo reconstruction algorithm that precomputes subpixel-shifted binary descriptors, then iteratively samples them along slanted disparity plane hypotheses, applying an edge-aware filter for spatial cost aggregation. Plane hypotheses are shared within rectangular tiles, but every pixel selects a different winner from among these candidates, as in PatchMatch Filter. This architecture performs few descriptor computations but many cost aggregations, and we tune our choice of descriptor and filter accordingly: We propose a new 32-bit binary spacetime descriptor "breve" combining the benefits of small spatial extent with robustness to scene motion, and the system aggregates costs using the permeability filter, a very efficient edge-aware filter. Our prototype outputs 60 depth frames per second on a desktop GPU, using less than 11ms total computation per frame.