Regularized training of the extreme learning machine using the conjugate gradient method

Abstract

We describe a new algorithm providing regularized training of the extreme learning machine (ELM) that uses a modified conjugate gradient (CG) method to determine the network hidden to output weights. The CG method is modified to include a validation set performance calculation at each iteration step. The solution is initialized to zero and during the CG iterations, we monitor the validation set error. When the error begins to rise we terminate the CG algorithm. The operations per iteration is O(P2), where P is the number of output weights, which is significantly faster than the O(P3) operations per iteration required by ridge regression regularization methods. We demonstrate the effectiveness of our method by classifying the MNIST database and achieve an accuracy of 99.2% using an ELM classifier processing the unmodified pixel values.