Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels

ICML, pp. 4804-4815, 2020.

Cited by: 0|Bibtex|Views60
EI
Other Links: academic.microsoft.com|dblp.uni-trier.de
Weibo:
By comparing six robust deep learning methods, we found that real-world noise is more difficult to improve and methods that work well on synthetic noise may not work as well on realworld noise, and vice versa

Abstract:

Performing controlled experiments on noisy data is essential in thoroughly understanding deep learning across a spectrum of noise levels. Due to the lack of suitable datasets, previous research have only examined deep learning on controlled synthetic noise, and real-world noise has never been systematically studied in a controlled setting...More

Code:

Data:

0
Introduction
  • You take the blue pill you wake up in your bed and believe whatever you want to believe.
  • Deep Neural Networks (DNNs) trained on noisy data demonstrate intriguing properties.
  • When trained with stochastic gradient descent, DNNs learn patterns first before memorizing the label noise Arpit et al (2017).
  • These findings inspired recent research on noisy data.
  • As training data are usually noisy, the fact that DNNs are able to memorize the noisy labels highlights the importance of deep learning research on noisy data
Highlights
  • You take the blue pill you wake up in your bed and believe whatever you want to believe
  • As training data are usually noisy, the fact that Deep Neural Networks (DNNs) are able to memorize the noisy labels highlights the importance of deep learning research on noisy data
  • We find that real-world noise is more difficult for robust DNNs to improve
  • As the noise levels span across a wide range from 0% to 80%, we find the hyperparameters of robust DNNs are important
  • By comparing six robust deep learning methods, we found that real-world noise is more difficult to improve and methods that work well on synthetic noise may not work as well on realworld noise, and vice versa
Methods
  • The authors select six methods from four directions that deal with noisy training data: (a) regularization, (b) label/prediction correction, (c) example weighting, and (d) vicinal risk minimization
  • These methods are selected because they (i) represent a reasonable coverage of recent work, (ii) are competitive to the state-ofthe-art on the common CIFAR-100 dataset with synthetic noise.
  • The authors' study seeks the answer to the following questions: 1
  • How does their performance differ on synthetic versus real-world noise?
  • What is their real performance gap when each method is extensively tuned for every noise level?
Results
  • Fig. 2 plots the training curve and the test curve on Blue and Red noisy benchmarks using vanilla training.
  • 2 and Fig. 3b suggest that DNNs generalize better on real-world noisy data.
  • This phenomenon is probably due to the two properties discussed in Section 3: (i) red noisy images are similar to clean training images, and bring less change to the training (ii) red noisy images are often sampled out of the training classes.
  • This may make them less confusing for the fixed training classes.
Conclusion
  • The authors established a benchmark for controlled real-world noise. On the benchmark, the authors conducted a large-scale study to understand deep learning on noisy data across a variety of settings.
  • By comparing six robust deep learning methods, the authors found that real-world noise is more difficult to improve and methods that work well on synthetic noise may not work as well on realworld noise, and vice versa.
  • This encourages future research to be carried out on controlled real-world noise.
  • As well as the findings, will facilitate deep learning research on noisy data
Summary
  • Introduction:

    You take the blue pill you wake up in your bed and believe whatever you want to believe.
  • Deep Neural Networks (DNNs) trained on noisy data demonstrate intriguing properties.
  • When trained with stochastic gradient descent, DNNs learn patterns first before memorizing the label noise Arpit et al (2017).
  • These findings inspired recent research on noisy data.
  • As training data are usually noisy, the fact that DNNs are able to memorize the noisy labels highlights the importance of deep learning research on noisy data
  • Objectives:

    The authors' goal is to verify whether the findings still hold on this new data subset.
  • Methods:

    The authors select six methods from four directions that deal with noisy training data: (a) regularization, (b) label/prediction correction, (c) example weighting, and (d) vicinal risk minimization
  • These methods are selected because they (i) represent a reasonable coverage of recent work, (ii) are competitive to the state-ofthe-art on the common CIFAR-100 dataset with synthetic noise.
  • The authors' study seeks the answer to the following questions: 1
  • How does their performance differ on synthetic versus real-world noise?
  • What is their real performance gap when each method is extensively tuned for every noise level?
  • Results:

    Fig. 2 plots the training curve and the test curve on Blue and Red noisy benchmarks using vanilla training.
  • 2 and Fig. 3b suggest that DNNs generalize better on real-world noisy data.
  • This phenomenon is probably due to the two properties discussed in Section 3: (i) red noisy images are similar to clean training images, and bring less change to the training (ii) red noisy images are often sampled out of the training classes.
  • This may make them less confusing for the fixed training classes.
  • Conclusion:

    The authors established a benchmark for controlled real-world noise. On the benchmark, the authors conducted a large-scale study to understand deep learning on noisy data across a variety of settings.
  • By comparing six robust deep learning methods, the authors found that real-world noise is more difficult to improve and methods that work well on synthetic noise may not work as well on realworld noise, and vice versa.
  • This encourages future research to be carried out on controlled real-world noise.
  • As well as the findings, will facilitate deep learning research on noisy data
Tables
  • Table1: Overview of the datasets. The same test set, on each dataset, is shared in evaluation
  • Table2: Overview of the ImageNet architectures used in our study
  • Table3: The peak test accuracy (%) on CIFAR-100 with synthetic noise. † marks our implementation of MentorNet and Mixup under the best hyperparameter setting. “-” indicates the number is unavailable in the published paper. The best accuracy is in bold
  • Table4: Peak accuracy (%) of the best trial for each method, fine-tuned on Mini-ImageNet. The peak and converged test accuracies are shown in the format of XXX/YYY
  • Table5: Peak accuracy (%) of the best trial of for method, trained from scratch on Mini-ImageNet. ’-’ denotes the method that is failed to converge
  • Table6: Peak accuracy (%) of the best trial for each method, fine-tuned on Stanford Cars. The peak and converged test accuracies are shown in the format of XXX/YYY
  • Table7: Peak accuracy (%) of the best trial for each method, trained from scratch on Stanford Cars. ’-’ denotes the method that is failed to converge
Download tables as Excel
Related work
  • Noisy Datasets: to understand deep learning’s properties on noisy training data, research often conducted experiments across a series of levels of synthetic noises. The most common one is uniform label-flipping noise (aka. symmetric noise), in which the label of each example is independently and uniformly changed to a random (incorrect) class with a probability (Zhang et al, 2017; Arpit et al, 2017; Vahdat, 2017; Shu et al, 2019; Jiang et al, 2018; Ma et al, 2018; Han et al, 2018; Li et al, 2019; Arazo et al, 2019). The synthetic noisy dataset enables us to experiment on controlled noise levels, and drive the development of theory and methodology in this field. Research have also examined other types of noise to better approximate the real-world noise distribution, including class-conditional noises (Patrini et al, 2017; Rolnick et al, 2017), noises from other datasets (Wang et al, 2018), etc. However, these noises are still synthetic, generated from artificial distributions. Furthermore, different types of synthetic noises may lead to inconsistent or even contradicting observations. For example, Rolnick et al (2017) experimented on a slightly different type of uniform noise and surprisingly found that DNNs are robust to massive label noise.
Reference
  • Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In ICML, 2019.
    Google ScholarLocate open access versionFindings
Full Text
Your rating :
0

 

Tags
Comments