AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
We proposed a unified perspective and algorithm to deal with label prior shift, and combine it with methods for non-semantic likelihood to tackle both types of shifts simultaneously

Posterior Re-calibration for Imbalanced Datasets

NIPS 2020, (2020)

Cited by: 0|Views29
EI
Full Text
Bibtex
Weibo

Abstract

Neural Networks can perform poorly when the training label distribution is heavily imbalanced, as well as when the testing data differs from the training distribution. In order to deal with shift in the testing label distribution, which imbalance causes, we motivate the problem from the perspective of an optimal Bayes classifier and der...More

Code:

Data:

0
Introduction
  • Applications of deep learning algorithms in the real world have fueled new interest in research beyond well-constructed datasets where the classes are balanced and the training distribution faithfully reflects the true distribution.

    it is unlikely that one can anticipate all possible scenarios and construct well-curated datasets consistently.
  • Applications of deep learning algorithms in the real world have fueled new interest in research beyond well-constructed datasets where the classes are balanced and the training distribution faithfully reflects the true distribution.
  • The aforementioned problems can be categorized as distributional shift between the training and testing conditions, :label prior shift and non-semantic likelihood shift.
  • The authors focus on label prior shift, which can arise when i) the training class distribution does not match the true class distribution due to the inherent difficulty in obtaining samples from certain classes, or ii) the distribution of classes does not reflect their relative importance.
  • In semantic segmentation, the pixel percentage of pedestrains does not reflect their importance in prediction
Highlights
  • Applications of deep learning algorithms in the real world have fueled new interest in research beyond well-constructed datasets where the classes are balanced and the training distribution faithfully reflects the true distribution.

    it is unlikely that one can anticipate all possible scenarios and construct well-curated datasets consistently
  • We explored a probabilistic way to deal with prior shift
  • We proposed a unified perspective and algorithm to deal with label prior shift, and combine it with methods for non-semantic likelihood to tackle both types of shifts simultaneously
  • The major contribution is a novel imbalance calibration technique which is motivated from the concept of optimal Bayes classifier and optimization of KL divergence
  • The final algorithm can be used in probabilistic classification tasks regardless of underlying architectures
  • We have shown the generality of the imbalanced calibration technique on six datasets and a number of different architectures, showing effectiveness of the unified algorithm against both extreme dataset imbalance and unseen degredations in multi-modal semantic segmentation
Methods
  • Label prior shift refers to the prior distribution changing between training and testing, i.e, Ps(Y ) = Pt(Y ); in this paper, the authors focus on the case where the training distributions have varying degrees of imbalance and the testing distribution shifts.
  • Non-semantic likelihood shift refers to shift without introducing new semantic labels such as sensor degradations, changes in lighting, or presentation of the same categories in a different modality, i.e. fs(X|Y ) = ft(X|Y )
  • Note that some metrics during testing implicitly emphasize a uniform distribution over labels, regardless of the actual label distribution in the testing dataset; see Appendix Section 6.3 for a proof for one popular class of metrics, namely class-averaged metrics.
Results
  • Qualitative Results for Synthia RGBD fusion Experiments

    Fig. 4 shows qualitative results for the Synthia fusion experiments in Sec. 4.4.2.
  • Qualitative Results for Synthia RGBD fusion Experiments.
  • Fig. 4 shows qualitative results for the Synthia fusion experiments in Sec. 4.4.2.
  • The authors' method UNOIC is able to capture small details such as pedestrians in the distance even under severe visual degradations.
  • Because the rain condition is not encountered during training, the rgb channel failed to cope with the degradation.
  • UNO [19] dynamically shifts the fusion algorithms weight to the depth channel while the imbalance calibration method gives more weights to small objects.
  • The qualitative and quatitative results in Sec 4.4.2 demonstrate the effectiveness of the unifed perspective 3.1 and Alg. 1
Conclusion
  • The authors proposed a unified perspective and algorithm to deal with label prior shift, and combine it with methods for non-semantic likelihood to tackle both types of shifts simultaneously.
  • To accompany the formula, the authors introduced a modified binary search algorithm for the hyperparameter based on the empirical observation of a concave performance function as it is varied.
  • The authors have shown the generality of the imbalanced calibration technique on six datasets and a number of different architectures, showing effectiveness of the unified algorithm against both extreme dataset imbalance and unseen degredations in multi-modal semantic segmentation
Summary
  • Introduction:

    Applications of deep learning algorithms in the real world have fueled new interest in research beyond well-constructed datasets where the classes are balanced and the training distribution faithfully reflects the true distribution.

    it is unlikely that one can anticipate all possible scenarios and construct well-curated datasets consistently.
  • Applications of deep learning algorithms in the real world have fueled new interest in research beyond well-constructed datasets where the classes are balanced and the training distribution faithfully reflects the true distribution.
  • The aforementioned problems can be categorized as distributional shift between the training and testing conditions, :label prior shift and non-semantic likelihood shift.
  • The authors focus on label prior shift, which can arise when i) the training class distribution does not match the true class distribution due to the inherent difficulty in obtaining samples from certain classes, or ii) the distribution of classes does not reflect their relative importance.
  • In semantic segmentation, the pixel percentage of pedestrains does not reflect their importance in prediction
  • Methods:

    Label prior shift refers to the prior distribution changing between training and testing, i.e, Ps(Y ) = Pt(Y ); in this paper, the authors focus on the case where the training distributions have varying degrees of imbalance and the testing distribution shifts.
  • Non-semantic likelihood shift refers to shift without introducing new semantic labels such as sensor degradations, changes in lighting, or presentation of the same categories in a different modality, i.e. fs(X|Y ) = ft(X|Y )
  • Note that some metrics during testing implicitly emphasize a uniform distribution over labels, regardless of the actual label distribution in the testing dataset; see Appendix Section 6.3 for a proof for one popular class of metrics, namely class-averaged metrics.
  • Results:

    Qualitative Results for Synthia RGBD fusion Experiments

    Fig. 4 shows qualitative results for the Synthia fusion experiments in Sec. 4.4.2.
  • Qualitative Results for Synthia RGBD fusion Experiments.
  • Fig. 4 shows qualitative results for the Synthia fusion experiments in Sec. 4.4.2.
  • The authors' method UNOIC is able to capture small details such as pedestrians in the distance even under severe visual degradations.
  • Because the rain condition is not encountered during training, the rgb channel failed to cope with the degradation.
  • UNO [19] dynamically shifts the fusion algorithms weight to the depth channel while the imbalance calibration method gives more weights to small objects.
  • The qualitative and quatitative results in Sec 4.4.2 demonstrate the effectiveness of the unifed perspective 3.1 and Alg. 1
  • Conclusion:

    The authors proposed a unified perspective and algorithm to deal with label prior shift, and combine it with methods for non-semantic likelihood to tackle both types of shifts simultaneously.
  • To accompany the formula, the authors introduced a modified binary search algorithm for the hyperparameter based on the empirical observation of a concave performance function as it is varied.
  • The authors have shown the generality of the imbalanced calibration technique on six datasets and a number of different architectures, showing effectiveness of the unified algorithm against both extreme dataset imbalance and unseen degredations in multi-modal semantic segmentation
Tables
  • Table1: Summary of datasets and architectures: Imbalance Ratio is the ratio of class size between the largest and smallest class
  • Table2: Top1 validation error↓ * indicates the reported results from [<a class="ref-link" id="c10" href="#r10">10</a>]. Our model (-IC) achieves the best performance overall. CE is a baseline trained with unmodified cross-entropy loss. CB refers to class balanced loss in [<a class="ref-link" id="c1" href="#r1">1</a>]. Combining our method with techniques such as Deferred Reweighting (DRW) [<a class="ref-link" id="c2" href="#r2">2</a>] yields the best performance
  • Table3: Validation error↓ on iNaturalist2018. Our method (-IC) yields the best performance and is effective for extreme large number of classes and severe imbalance. We use 3-fold cross validation for tuning λ and testing. Subscript value represents the λ for that experiment. * indicates the reported results from [<a class="ref-link" id="c10" href="#r10">10</a>]
  • Table4: Test mean IOU, mean accuracy on Synthia in distribution splits (RGB only). Our method (IC) with cross entropy or Focal loss achieves significant improvement in mean accuracy while maintaining competitive mean IOU
  • Table5: Test mean IOU and mean accuracy (mIOU | mACC) of the joint algorithm 1 on Synthia out-of-distribution splits.The listed conditions have not been encountered during training
  • Table6: Top1 validation error↓ for Imbalance Calibration. CE is a baseline trained with unmodified cross-entropy loss. CB refers to class balanced loss in [<a class="ref-link" id="c1" href="#r1">1</a>]. The subscript denotes the λ values for the experiments
Download tables as Excel
Related work
  • In this paper, we focus on prior (label) distribution shift resulting from various degrees of imbalance in the training label distributions, and testing on a different distribution or emphasizing a different distribution in the testing metrics (e.g. class-averaged accuracy). We therefore summarize methods for dealing with such imbalance.

    Imbalance - Data Level Methods The simplest methods for dealing with label imbalance during training are random under-sampling (RUS) which discards samples from the majority classes and random over-sampling (ROS) which re-samples from the minority classes [3]. While ROS is infeasible when the data imbalance is extreme, RUS tends to overfit the minority classes [4]. Synthetic generation [5] or interpolation [6] to increase the number of samples in the minority class are also used. However, these methods are sensitive to imperfections in the generated data.
Funding
  • We then demonstrate that our method can achieve state of the art results on imbalanced variants of CIFAR-10 and CIFAR-100, and can scale to larger datasets such as iNaturalist which has extreme imbalance
Study subjects and analysis
datasets: 6
We re-interpret these methods from the same Bayesian perspective, and demonstrate that our method can deal with both problems in a unified way. We demonstrate our method on six datasets and five different neural network architectures, across two different imbalanced tasks: classification and semantic segmentation. Using a toy dataset that allows for visualizations of the classifier margins, we show that our method effectively shifts the decision boundary away from the minority class towards the over-represented class

datasets: 6
Unlike cost sensitive learning, an optimal hyperparameter can be searched efficiently on a validation set posttraining. • We test the algorithm on six datasets and five different architectures and outperforms state-of-the-art models on classification accuracy (recall) across all tasks and models while maintaining good precision. • We further combine our method with non-semantic likelihood shift methods, re-motivate it from the Bayesian perspective, and show that we can tackle both problems in a unified way on a RGB-D semantic segmentation dataset with unseen weather conditions

datasets with five different architectures: 6
[20] DeepLab [24]. We conduct experiments to test our imbalance calibration (IC) algorithm described in sec. 3.2 on six datasets with five different architectures. A summary of datasets and architectures used for each one is shown in table 1

datasets: 6
The final algorithm can be used in probabilistic classification tasks regardless of underlying architectures. We have shown the generality of the imbalanced calibration technique on six datasets and a number of different architectures, showing effectiveness of the unified algorithm against both extreme dataset imbalance and unseen degredations in multi-modal semantic segmentation.

Reference
  • Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9268–9277, 2019.
    Google ScholarLocate open access versionFindings
  • Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. In Advances in Neural Information Processing Systems, pages 1565–1576, 2019.
    Google ScholarLocate open access versionFindings
  • Jason Van Hulse, Taghi M Khoshgoftaar, and Amri Napolitano. Experimental perspectives on learning from imbalanced data. In Proceedings of the 24th international conference on Machine learning, pages 935–942, 2007.
    Google ScholarLocate open access versionFindings
  • Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):27, 2019.
    Google ScholarLocate open access versionFindings
  • Haibo He, Yang Bai, Edwardo A Garcia, and Shutao Li. Adasyn: Adaptive synthetic sampling approach for imbalanced learning. In 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence), pages 1322–1328. IEEE, 2008.
    Google ScholarLocate open access versionFindings
  • Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321– 357, 2002.
    Google ScholarLocate open access versionFindings
  • David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In Proceedings of the IEEE international conference on computer vision, pages 2650–2658, 2015.
    Google ScholarLocate open access versionFindings
  • Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(12):2481–2495, 2017.
    Google ScholarLocate open access versionFindings
  • Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
    Google ScholarLocate open access versionFindings
  • Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. arXiv preprint arXiv:1912.02413, 2019.
    Findings
  • Steve Lawrence, Ian Burns, Andrew Back, Ah Chung Tsoi, and C Lee Giles. Neural network classification and prior class probabilities. In Neural networks: tricks of the trade, pages 299–313.
    Google ScholarLocate open access versionFindings
  • Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249–259, 2018.
    Google ScholarLocate open access versionFindings
  • Robin Chan, Matthias Rottmann, Fabian Hüger, Peter Schlicht, and Hanno Gottschalk. Application of decision rules for handling class imbalance in semantic segmentation. arXiv preprint arXiv:1901.08394, 2019.
    Findings
  • John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61–74, 1999.
    Google ScholarLocate open access versionFindings
  • Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure. Neural computation, 14(1):21–41, 2002.
    Google ScholarLocate open access versionFindings
  • Ali E Abbas. A kullback-leibler view of linear and log-linear pools. Decision Analysis, 6(1):25–37, 2009.
    Google ScholarLocate open access versionFindings
  • Juliana Tolles and William J Meurer. Logistic regression: relating patient characteristics to outcomes. Jama, 316(5):533–534, 2016.
    Google ScholarLocate open access versionFindings
  • Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1321–1330. JMLR. org, 2017.
    Google ScholarLocate open access versionFindings
  • Junjiao Tian, Wesley Cheung, Nathan Glaser, Yen-Cheng Liu, and Zsolt Kira. Uno: Uncertaintyaware noisy-or multimodal fusion for unanticipated input degradation. arXiv preprint arXiv:1911.05611, 2019.
    Findings
  • Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015.
    Google ScholarLocate open access versionFindings
  • Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769–8778, 2018.
    Google ScholarLocate open access versionFindings
  • Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
    Google ScholarLocate open access versionFindings
  • German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3234–3243, 2016.
    Google ScholarLocate open access versionFindings
  • Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
    Google ScholarLocate open access versionFindings
  • German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio Lopez. The SYNTHIA Dataset: A large collection of synthetic images for semantic segmentation of urban scenes. 2016.
    Google ScholarFindings
  • Jose G Moreno-Torres, Troy Raeder, RocíO Alaiz-RodríGuez, Nitesh V Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern recognition, 45(1):521–530, 2012.
    Google ScholarLocate open access versionFindings
Author
Junjiao Tian
Junjiao Tian
Yen-Cheng Liu
Yen-Cheng Liu
Nathaniel Glaser
Nathaniel Glaser
Your rating :
0

 

Tags
Comments
小科