Observational Overfitting in Reinforcement Learning

international conference on learning representations, (2020)

被引用：0|浏览50

下载 PDF 全文

其它链接

academic.microsoft.com arxiv.org

引用

微博一下

关键词

observational overfitting reinforcement learning generalization更多(3+)

摘要

A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to design multiple synthe...更多

代码：

数据：

精读

下载 PDF 全文

PPT (上传PPT)

简介

Generalization for RL has recently grown to be an important topic for agents to perform well in unseen environments.
One particular framework, which the authors denote by zero-shot supervised framework (Zhang et al, 2018a;c; Nichol et al, 2018; Justesen et al, 2018) and is used to study RL generalization, is to treat it analogous to a classical supervised learning (SL) problem – i.e. assume there exists a distribution of MDP’s, train jointly on a finite “training set” sampled from this distribution, and check expected performance on the entire distribution, with the fixed trained policy
In this framework, there is a spectrum of analysis, ranging from almost purely theoretical analysis (Wang et al, 2019; Asadi et al, 2018) to full empirical results on diverse environments (Zhang et al, 2018c; Packer et al, 2018).
Various hyperparameters such as the batch-size in SGD (Smith et al, 2018), choice of optimizer (Kingma & Ba, 2014), discount factor γ (Jiang et al, 2015) and regularizations such as entropy (Ahmed et al, 2018) and weight norms (Cobbe et al, 2018) can affect generalization

重点内容

Generalization for reinforcement learning has recently grown to be an important topic for agents to perform well in unseen environments
In order to isolate these factors, we study one broad factor affecting generalization that is most correlated with themes in supervised learning, observational overfitting, where an agent overfits due to properties of the observation which are irrelevant to the latent dynamics of the Markov Decision Process family
We have identified and isolated a key component of overfitting in reinforcement learning as the particular case of “observational overfitting”, which is attractive for studying architectural implicit regularizations
The analytical case of linear quadratic regulators and linear policies under exact gradient descent, which lays the foundation for understanding theoretical properties of networks in reinforcement learning generalization
The empirical but principled Projected-Gym case for both multi-layer perceptrons and convolutional networks which demonstrates the effects of neural network policies under nonlinear environments
We noted that current network policy bounds using ideas from supervised learning are unable to explain overparametrization effects in reinforcement learning, which is an important further direction. This area of reinforcement learning generalization is an extension of static supervised learning classification from adding extra reinforcement learning components

方法

The authors first analyze the case of the LQR as a surrogate for what may occur in deep RL, which has been done before for various topics such as sample complexity (Dean et al, 2019) and model-based RL (Tu & Recht, 2019)
This is analogous to analyzing linear/logistic regression (Kakade et al, 2008; McAllester, 2003) as a surrogate to understanding extensions to deep SL techniques (Neyshabur et al, 2018a; Bartlett et al, 2017).
There have been multiple recent works on this linear-layer stacking in SL and other theoretical problems such as matrix factorization and matrix completion (Arora et al, 2018b;a; Gunasekar et al, 2017), but to the knowledge, the authors are the first to analyze this case in the context of RL generalization

结论

The authors have identified and isolated a key component of overfitting in RL as the particular case of “observational overfitting”, which is attractive for studying architectural implicit regularizations.
In terms of architectural design, recent works (Jacot et al, 2018; Garriga-Alonso et al, 2019; Lee et al, 2019) have shed light on the properties of asymptotically overparametrized neural networks in the infinite width and depth cases and their performance in SL
Such architectures may be used in the RL setting which can possibly provide benefits, one of which is generalization as shown in this paper.
The authors believe that this work provides an important initial step towards solving these future problems

总结

Introduction:
Generalization for RL has recently grown to be an important topic for agents to perform well in unseen environments.
One particular framework, which the authors denote by zero-shot supervised framework (Zhang et al, 2018a;c; Nichol et al, 2018; Justesen et al, 2018) and is used to study RL generalization, is to treat it analogous to a classical supervised learning (SL) problem – i.e. assume there exists a distribution of MDP’s, train jointly on a finite “training set” sampled from this distribution, and check expected performance on the entire distribution, with the fixed trained policy
In this framework, there is a spectrum of analysis, ranging from almost purely theoretical analysis (Wang et al, 2019; Asadi et al, 2018) to full empirical results on diverse environments (Zhang et al, 2018c; Packer et al, 2018).
Various hyperparameters such as the batch-size in SGD (Smith et al, 2018), choice of optimizer (Kingma & Ba, 2014), discount factor γ (Jiang et al, 2015) and regularizations such as entropy (Ahmed et al, 2018) and weight norms (Cobbe et al, 2018) can affect generalization
Methods:
The authors first analyze the case of the LQR as a surrogate for what may occur in deep RL, which has been done before for various topics such as sample complexity (Dean et al, 2019) and model-based RL (Tu & Recht, 2019)
This is analogous to analyzing linear/logistic regression (Kakade et al, 2008; McAllester, 2003) as a surrogate to understanding extensions to deep SL techniques (Neyshabur et al, 2018a; Bartlett et al, 2017).
There have been multiple recent works on this linear-layer stacking in SL and other theoretical problems such as matrix factorization and matrix completion (Arora et al, 2018b;a; Gunasekar et al, 2017), but to the knowledge, the authors are the first to analyze this case in the context of RL generalization
Conclusion:
The authors have identified and isolated a key component of overfitting in RL as the particular case of “observational overfitting”, which is attractive for studying architectural implicit regularizations.
In terms of architectural design, recent works (Jacot et al, 2018; Garriga-Alonso et al, 2019; Lee et al, 2019) have shed light on the properties of asymptotically overparametrized neural networks in the infinite width and depth cases and their performance in SL
Such architectures may be used in the RL setting which can possibly provide benefits, one of which is generalization as shown in this paper.
The authors believe that this work provides an important initial step towards solving these future problems

表格

Table1: Raw Network Performance (rounded to nearest 0.5) on CoinRun, 100 levels. Images scaled to default image sizes (32 × 32 or 224 × 224) depending on network input requirement. See Appendix A.2.1 for training curves
Table2: IMPALA vs NatureCNN test rewards, with and without Blackout
Table3: Hyperparameters for LQR

Download tables as Excel

基金

Studies one broad factor affecting generalization that is most correlated with themes in SL, observational overfitting, where an agent overfits due to properties of the observation which are irrelevant to the latent dynamics of the MDP family
Studies observational overfitting with linear quadratic regulators in a synthetic environment and neural networks such as multi-layer perceptrons and convolutions in classic Gym environments
A primary novel result demonstrates for all cases is that implicit regularization occurs in this setting in RL
Provides an extensive analysis of the convex one-step LQR case under the observational overfitting regime, showing that under Gaussian initialization of the policy and using gradient descent on the training cost, a generalization gap must necessarily exist

引用论文

Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. CoRR, abs/1811.11214, 2018.
Martın Arjovsky, Leon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. CoRR, abs/1907.02893, 2019. URL http://arxiv.org/abs/1907.02893.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. CoRR, abs/1802.06509, 2018a. URL http://arxiv.org/abs/1802.06509.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018, pp. 244–253, 2018b.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018, pp. 254–263, 2018c.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. CoRR, abs/1905.13655, 2019. URL http://arxiv.org/abs/1905.13655.
Kavosh Asadi, Dipendra Misra, and Michael L. Littman. Lipschitz continuity in model-based reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018, pp. 264–273, 2018.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463–482, 2002.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6240–624Curran Associates, Inc., 2017.
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. CoRR, abs/1207.4708, 2012.
Joseph K. Bradley, Aapo Kyrola, Danny Bickson, and Carlos Guestrin. Parallel coordinate descent for l1-regularized loss minimization. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July 2, 2011, pp. 321–328, 20URL https://icml.cc/2011/papers/231_icmlpaper.pdf.
Krzysztof Choromanski, Mark Rowland, Vikas Sindhwani, Richard E. Turner, and Adrian Weller. Structured evolution with compact architectures for scalable policy optimization. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018, pp. 969–977, 2018. URL http://proceedings.mlr.press/v80/choromanski18a.html.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. CoRR, abs/1812.02341, 2018.
Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu. On the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics, Aug 2019.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018, pp. 1406–1415, 2018.
Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018, pp. 1466–1475, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1126–1135, 2017. URL http://proceedings.mlr.press/v70/finn17a.html.
Adam Gaier and David Ha. Weight agnostic neural networks. CoRR, abs/1906.04358, 20URL http://arxiv.org/abs/1906.04358.
Shani Gamrian and Yoav Goldberg. Transfer learning for related reinforcement learning tasks via image-to-image translation. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 2063–2072, 2019. URL http://proceedings.mlr.press/v97/gamrian19a.html.
Adria Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks as shallow gaussian processes. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL https://openreview.net/forum?id=Bklfsi0cKm.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory, COLT 2018, Stockholm, Sweden, 6-9 July 2018., pp. 297–299, 2018. URL http://proceedings.mlr.press/v75/golowich18a.html.
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018, pp. 1787–1796, 2018.
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. In Advances in Neural Information Processing Systems, pp. 6151–6159, 2017.
Christina Heinze-Deml and Nicolai Meinshausen. Conditional variance penalties and domain shift robustness. CoRR, abs/1710.11469, 2019. URL https://arxiv.org/abs/1710.11469.
Christina Heinze-Deml, Jonas Peters, and Nicolai Meinshausen. Invariant causal prediction for nonlinear models. CoRR, abs/1706.08576, 2019. URL https://arxiv.org/abs/1706.08576.
Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2019,Long Beach, CA, USA, 2019.
Arthur Jacot, Clement Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montreal, Canada., pp. 8580–8589, 2018.
Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’15, pp. 1181–1189, Richland, SC, 2015. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978-1-4503-3413-6.
Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. CoRR, abs/1810.00113, 2018. URL http://arxiv.org/abs/1810.00113.
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Procedural level generation improves generality of deep reinforcement learning. CoRR, abs/1806.10729, 2018. URL http://arxiv.org/abs/1806.10729.
Sham M. Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction: Risk bounds, margin bounds, and regularization. In Advances in Neural Information Processing Systems 21, Proceedings of the Twenty-Second Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 8-11, 2008, pp. 793–800, 2008.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. CoRR, abs/1902.06720, 2019. URL http://arxiv.org/abs/1902.06720.
Tengyuan Liang, Tomaso A. Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, pp. 888–896, 2019. URL http://proceedings.mlr.press/v89/liang19a.html.
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. J. Artif. Intell. Res., 61:523–562, 2018. doi: 10.1613/jair.5699.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides a competitive approach to reinforcement learning. CoRR, abs/1803.07055, 2018.
David A. McAllester. Simplified pac-bayesian margin bounds. In Computational Learning Theory and Kernel Machines, 16th Annual Conference on Computational Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003, Proceedings, pp. 203–215, 2003. doi: 10.1007/978-3-540-45167-9\ 16. URL https://doi.org/10.1007/978-3-540-45167-9_16.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
Vaishnavh Nagarajan and J Zico Kolter. Generalization in deep networks: The role of distance from initialization. arXiv preprint arXiv:1901.01672, 2019.
Behnam Neyshabur. Implicit regularization in deep learning. CoRR, abs/1709.01953, 2017. URL http://arxiv.org/abs/1709.01953.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. ICLR, 2015a.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pp. 1376–1401, 2015b.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5949–5958, 2017.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrallynormalized margin bounds for neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, 2018a. URL https://openreview.net/forum?id=Skz_WfbCZ.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of over-parametrization in generalization of neural networks. CoRR, abs/1805.12076, 2018b. URL http://arxiv.org/abs/1805.12076.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in RL. CoRR, abs/1804.03720, 2018.
OpenAI. Openai five. CoRR, 2018. URL https://openai.com/blog/openai-five/.
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krahenbuhl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. CoRR, abs/1810.12282, 2018. URL http://arxiv.org/abs/1810.12282.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, pp. 1–8, 2018. doi: 10.1109/ ICRA.2018.8460528. URL https://doi.org/10.1109/ICRA.2018.8460528.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 2817–2826, 2017.
Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham M. Kakade. Towards generalization and simplicity in continuous control. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 6553–6564, 2017.
Adam Santoro, Ryan Faulkner, David Raposo, Jack W. Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy P. Lillicrap. Relational recurrent neural networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montreal, Canada., pp. 7310–7321, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.
Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V. Le. Don’t decay the learning rate, increase the batch size. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, 2018. URL https://openreview.net/forum?id=B1Yy1BxCZ.
Felipe Petroski Such, Vashisht Madhavan, Rosanne Liu, Rui Wang, Pablo Samuel Castro, Yulun Li, Ludwig Schubert, Marc G. Bellemare, Jeff Clune, and Joel Lehman. An atari model zoo for analyzing, visualizing, and comparing deep reinforcement learning agents. CoRR, abs/1812.07069, 2018.
Stephen Tu and Benjamin Recht. The gap between model-based and model-free methods on the linear quadratic regulator: An asymptotic viewpoint. In Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA, pp. 3036–3083, 2019. URL http://proceedings.mlr.press/v99/tu19a.html.
Roman Vershynin. Lectures in geometric functional analysis. CoRR, 2009. Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle