Efficient computation of the Levenberg-Marquardt algorithm for feedforward networks with linear outputs

Abstract

An efficient algorithm for the calculation of the approximate Hessian matrix for the Levenberg-Marquardt (LM) optimization algorithm for training a single-hidden-layer feedforward network with linear outputs is presented. The algorithm avoids explicit calculation of the Jacobian matrix and computes the gradient vector and approximate Hessian matrix directly. It requires approximately 1/N the floating point operations of other published algorithms, where N is the number of network outputs. The required memory for the algorithm is also less than 1/N of the memory required for algorithms explicitly computing the Jacobian matrix. We applied our algorithm to two large-scale classification problems - the MNIST and the Forest Cover Type databases. Our results were within 0.5% of the best performance of systems using pixel values as inputs to a feedforward network for the MNIST database. Our results were achieved with a much smaller network than other published results. We achieved state-of-the-art performance for the Forest Cover Type database.