Issue |
MATEC Web Conf.
Volume 232, 2018
2018 2nd International Conference on Electronic Information Technology and Computer Engineering (EITCE 2018)
|
|
---|---|---|
Article Number | 03007 | |
Number of page(s) | 4 | |
Section | Algorithm Study and Mathematical Application | |
DOI | https://doi.org/10.1051/matecconf/201823203007 | |
Published online | 19 November 2018 |
An Optimization Strategy Based on Hybrid Algorithm of Adam and SGD
1,2
Software Engineering, Central South University, 410075 Changsha, China
3
Biological Engineering, Nanyang Normal University, 473000 Nanyang, China
a Corresponding author: pyzhoucsu@163.com
Despite superior training outcomes, adaptive optimization methods such as Adam, Adagrad or RMSprop have been found to generalize poorly compared to stochastic gradient descent (SGD). So scholars (Nitish Shirish Keskar et al., 2017) proposed a hybrid strategy to start training with Adam and switch to SGD at the right time. In the learning task with a large output space, it was observed that Adam could not converge to an optimal solution (or could not converge to an extreme point in a non-convex scene) [1]. Therefore, this paper proposes a new variant of the ADAM algorithm (AMSGRAD), which not only solves the convergence problem, but also improves the empirical performance.
© The Authors, published by EDP Sciences, 2018
This is an open access article distributed under the terms of the Creative Commons Attribution License 4.0 (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Current usage metrics show cumulative count of Article Views (full-text article views including HTML views, PDF and ePub downloads, according to the available data) and Abstracts Views on Vision4Press platform.
Data correspond to usage on the plateform after 2015. The current usage metrics is available 48-96 hours after online publication and is updated daily on week days.
Initial download of the metrics may take a while.