We develop consensus-based distributed algorithms and establish the regret bounds for both convex and strongly convex losses, which match those of the centralized online optimization in the literature
Online Convex Optimization Over Erdos-Renyi Random Networks
NIPS 2020, (2020)
The work studies how node-to-node communications over an Erdos-Rényi random network influence distributed online convex optimization, which is vital in solving large-scale machine learning in antagonistic or changing environments. At per step, each node (computing unit) makes a local decision, experiences a loss evaluated with a convex fu...更多
下载 PDF 全文
- The online convex optimization paradigm has become a central and canonical solution for machine learning where data is generated sequentially over time, e.g., online routing, ad. selection for search engines, and spam filtering ([1,2,3,4]).
- The goal of the learner is to minimize its regret by adapting the decisions along the streaming data, measured by the difference between the cumulative loss of online decisions and the loss of the best decision chosen in hindsight.
- The gradient descent algorithm was proved to guarantee regret bounds O( T ) and O(ln(T )) for convex and strongly convex loss functions ([5, 6]), respectively, which were later shown to be minimax optimal ([7, 8])
- The online convex optimization paradigm has become a central and canonical solution for machine learning where data is generated sequentially over time, e.g., online routing, ad. selection for search engines, and spam filtering ([1,2,3,4])
- When the nodes can only observe loss function values, the one-point bandit or two-points bandit around the current decision is used to get a randomized approximation of the gradient. √
- We develop consensus-based distributed algorithms and establish the regret bounds for both convex and strongly convex losses, which match those of the centralized online optimization in the literature
- We further quantitatively characterize the influence of the algebraic network connectivity on the regret bounds, and show that the link connection probability can be used to tune a trade-off between the communication overhead and the computation accuracy
- Future directions include closing the gap of the regret bounds and extending the kernel-based methods to bandit online convex optimization over networks
- The work provides the theoretical understanding of the performance limits about distributed online convex optimization over random networks, and could be applied in processing streaming data to various Internet of Things systems, such as machine learning with personal wearable devices
- The authors consider online distributed gradient descents under Erdos-Rényi graphs, and establish the regret bounds explicitly in term of time horizon T , the underlying graph G, the probability p, and the decision complexity d.
- Algorithms: The node adapts its decision with gradient descents and local averaging over Erdos-.
- When the nodes can only observe loss function values, the one-point bandit or two-points bandit around the current decision is used to get a randomized approximation of the gradient.
- It is shown that the regrets scale with network size by a magnitude of
- The authors consider the online convex optimization over Erdos-Rényi random graphs under the full information feedback, one-point and two-points bandit feedback.
- The work provides the theoretical understanding of the performance limits about distributed online convex optimization over random networks, and could be applied in processing streaming data to various Internet of Things systems, such as machine learning with personal wearable devices.
- It does not present any foreseeable societal consequence
- Table1: Regret bounds and communication complexity over classical Erdos-Rényi graphs
- The early works  and  a√bout the centralized online convex optimization in the full information feedback obtained regrets O( T ) and O(ln(T )) for convex and strongly convex losses, respectively. With one-point bandit feedback, the seminal work  modified the gradient descent algorithm by replacing the gradient with a randomized estimate, and showed that the expected regret was O(T 3/4) for bounded and Lipschitz-continuous convex losses, whereas the regret O(T 2/3) was obtained in  for the setting of Lipschitz and strongly convex losses. It remains an open problem to design an optimal algorithm for the one-point band√it online convex optimization, whereas  proved that the optimal regret can not be better than Ω( T ) e√ven for strongly convex losses. In some special cases, the minimax regret is exactly O(poly(ln(T )) T ) , e.g., the losses are Lipschitz and linear [17, 18], or they are both smooth and str√ongly-convex . The recent work  designed some kernel-based methods with O(poly(ln(T )) T ) regret and polynomial computing time.  extended the onepoint bandit feedback to the multi-points bandit feedback where loss can b√e observed at multiple points around the decision, and established the expected regret bounds O( T ) and O(ln(T )) for convex and strongly convex losses. In this work, we design distributed algorithms over random graphs with full gradient feedback, one-point bandit feedback, and two-points bandit feedback, which can recover the regret bounds in the centralized methods (,  and ).
- The work was sponsored by Shanghai Sailing Program (No 20YF1453000, No 20YF1452800) and the Fundamental Research Funds for the Central Universities, China (No 22120200047, No 22120200048)
- Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends R in Optimization, 2(3-4):157–325, 2016.
- Ashok Cutkosky and Kwabena A Boahen. Online convex optimization with unconstrained domains and losses. In Advances in Neural Information Processing Systems (NIPS), pages 748–756, 2016.
- Jianjun Yuan and Andrew Lamperski. Online convex optimization for cumulative constraints. In Advances in Neural Information Processing Systems (NIPS), pages 6137–6146, 2018.
- Sébastien Arnold, Pierre-Antoine Manzagol, Reza Babanezhad Harikandeh, Ioannis Mitliagkas, and Nicolas Le Roux. Reducing the variance in online optimization by transporting past gradients. In Advances in Neural Information Processing Systems (NIPS), pages 5392–5403, 2019.
- Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (ICML-03), pages 928–936, 2003.
- Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning, 69(2-3):169–192, 2007.
- Jacob Abernethy, Alekh Agarwal, and Peter L Bartlett. A stochastic view of optimal regret through minimax duality. In Proceedings of the 22nd Annual Conference on Learning Theory (COLT), 2009.
- Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
- Wenpeng Zhang, Peilin Zhao, Wenwu Zhu, Steven CH Hoi, and Tong Zhang. Projection-free distributed online learning in networks. In Proceedings of the 34th International Conference on Machine LearningVolume 70 (ICML), pages 4054–4062. JMLR. org, 2017.
- Nuwan Ferdinand, Haider Al-Lawati, Stark Draper, and Matthew Nokleby. Anytime minibatch: Exploiting stragglers in online distributed optimization. In Seventh International Conference on Learning Representations (ICLR), 2019.
- Paul Erdos and Alfréd Rényi. On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci, 5(1): 17–60, 1960.
- Osman Yagan and Armand M Makowski. Modeling the pairwise key predistribution scheme in the presence of unreliable links. IEEE Transactions on Information Theory, 59(3):1740–1760, 2012.
- Osman Yagan. Performance of the eschenauer–gligor key distribution scheme under an on/off channel. IEEE Transactions on Information Theory, 58(6):3821–3835, 2012.
- Dhaval Adjodah, Dan Calacci, Abhimanyu Dubey, Anirudh Goyal, Peter Krafft, Esteban Moro, and Alex Pentland. Communication topologies between learning agents in deep reinforcement learning. arXiv preprint arXiv:1902.06740, 2019.
- Abraham D Flaxman, Adam Tauman Kalai, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proceedings of the sixteenth annual ACM-SIAM symposium on Discrete algorithms, pages 385–394. Society for Industrial and Applied Mathematics, 2005.
- Alekh Agarwal, Ofer Dekel, and Lin Xiao. Optimal algorithms for online convex optimization with multi-point bandit feedback. In Proceedings of the 23rd Annual Conference on Learning Theory (COLT), pages 28–40, 2010.
- Varsha Dani, Sham M Kakade, and Thomas P Hayes. The price of bandit information for online optimization. In Advances in Neural Information Processing Systems (NIPS), pages 345–352, 2008.
- Jacob Abernethy, Elad E Hazan, and Alexander Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In 21st Annual Conference on Learning Theory, COLT 2008, pages 263–273, 2008.
- Elad Hazan and Kfir Levy. Bandit convex optimization: Towards tight bounds. In Advances in Neural Information Processing Systems, pages 784–792, 2014.
- Sébastien Bubeck, Yin Tat Lee, and Ronen Eldan. Kernel-based methods for bandit convex optimization. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing, pages 72–85, 2017.
- Deming Yuan, Alexandre Proutiere, and Guodong Shi. Distributed online linear regression. IEEE Transactions on Information Theory, in press, Preprint arXiv:1902.04774, 2019.
- Feng Yan, Shreyas Sundaram, SVN Vishwanathan, and Yuan Qi. Distributed autonomous online learning: Regrets and intrinsic privacy-preserving properties. IEEE Transactions on Knowledge and Data Engineering, 25(11):2483–2493, 2012.
- Saghar Hosseini, Airlie Chapman, and Mehran Mesbahi. Online distributed optimization via dual averaging. In 52nd IEEE Conference on Decision and Control, pages 1484–1489. IEEE, 2013.
- David Mateos-Nunez and Jorge Cortés. Distributed online convex optimization over jointly connected digraphs. IEEE Transactions on Network Science and Engineering, 1(1):23–37, 2014.
- Deming Yuan, Alexandre Proutiere, and Guodong Shi. Distributed online optimization with long-term constraints. Preprint arXiv:1912.09705, 2019.
- Mohammad Akbari, Bahman Gharesifard, and Tamás Linder. Distributed online convex optimization on time-varying directed graphs. IEEE Transactions on Control of Network Systems, 4(3):417–428, 2015.
- Soomin Lee and Michael M Zavlanos. Distributed primal-dual methods for online constrained optimization. In 2016 American Control Conference (ACC), pages 7171–7176. IEEE, 2016.
- Elad Hazan, Alexander Rakhlin, and Peter L Bartlett. Adaptive online gradient descent. In Advances in Neural Information Processing Systems (NIPS), pages 65–72, 2008.
- Fabio Fagnani and Sandro Zampieri. Randomized consensus algorithms over large scale networks. IEEE Journal on Selected Areas in Communications, 26(4):634–649, 2008.
- Theodora S Brisimi, Ruidi Chen, Theofanie Mela, Alex Olshevsky, Ioannis Ch Paschalidis, and Wei Shi. Federated learning of predictive models from federated electronic health records. International journal of medical informatics, 112:59–67, 2018.