Semantic-aware self-supervised depth estimation for stereo 3D detection

•A semantic-aware depth self-supervision framework for stereo-based 3D detection.•Adapting self-supervised losses defined in the image frustum space to the 3D space.•Avoiding the dependency on ground-truth and pseudo disparity supervisions.•Sampling the supervisions in an unbalanced manner provides more useful information.•A state-of-the-art stereo-based detector supervised only by 3D boxes.