Residual objectness for imbalance reduction

•We discover that the foreground-background imbalance in object detection could be addressed in a learning-based manner, without any hard-crafted resampling and reweighting schemes.•We propose a novel Residual Objectness (ResObj) mechanism to address the foreground-background imbalance in training object detectors. With a cascade architecture to gradually refine the objectness estimation, our ResObj module could address the imbalance in an endto- end way, thus avoiding laborious hyper-parameters tuning required by resampling and reweighting schemes.•We validate the proposed method on the COCO dataset with thorough ablation studies. For various detectors, our Residual Objectness steadily improves relative 3%∼4% detection accuracy.