Optimising neural network training efficiency through spectral parameter-based multiple adaptive learning rates

The process of training neural networks heavily involves solving optimization problems. Most optimization algorithms use a !xed learning rate or a simpli!ed adaptive updating scheme in every iteration. In this paper, we propose a stochastic gradient descent method with multiple adaptive learning rates (MAdaGrad) and Adam with multiple adaptive learning rates (MAdaGrad Adam). The proposed algorithm updates the learning rate in every iteration based on the approximated spectrum of the Hessian of the loss function. The method is compared to the existing optimization methods in machine learning, namely stochastic gradient descent method (SGD) and Adam. Selected datasets are used to evaluate the performance of the proposed method. The proposed algorithm is used to train the neural networks with di"erent hidden layer sizes and di"erent neurons. The numerical results prove that the proposed methods perform better than SGD and Adam.

Text 121559.pdf - Accepted Version Restricted to Repository staff only Download (6MB) Official URL or Download Paper: https://dl.acm.org/doi/proceedings/10.1145/3708778

Actions (login required)

View Item