Please use this identifier to cite or link to this item:
Scopus Web of Science® Altmetric
Type: Conference paper
Title: On the importance of normalisation layers in deep learning with piecewise linear activation units
Author: Liao, Z.
Carneiro, G.
Citation: Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision, 2016, pp.1-8
Publisher: IEEE
Issue Date: 2016
Series/Report no.: IEEE Winter Conference on Applications of Computer Vision
ISBN: 9781509006410
ISSN: 2472-6737
Conference Name: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV 2016) (7 Mar 2016 - 10 Mar 2016 : Lake Placid, NY)
Statement of
Zhibin Liao, Gustavo Carneiro
Abstract: Deep feedforward neural networks with piecewise linear activations are currently producing the state-of-the-art results in several public datasets (e.g., CIFAR-10, CIFAR- 100, MNIST, and SVHN). The combination of deep learning models and piecewise linear activation functions allows for the estimation of exponentially complex functions with the use of a large number of subnetworks specialized in the classification of similar input examples. During the training process, these subnetworks avoid overfitting with an implicit regularization scheme based on the fact that they must share their parameters with other subnetworks. Using this framework, we have made an empirical observation that can improve even more the performance of such models. We notice that these models assume a balanced initial distribution of data points with respect to the domain of the piecewise linear activation function. If that assumption is violated, then the piecewise linear activation units can degenerate into purely linear activation units, which can result in a significant reduction of their capacity to learn complex functions. Furthermore, as the number of model layers increases, this unbalanced initial distribution makes the model ill-conditioned. Therefore, we propose the introduction of batch normalisation units into deep feedforward neural networks with piecewise linear activations, which drives a more balanced use of these activation units, where each region of the activation function is trained with a relatively large proportion of training samples. Also, this batch normalisation promotes the pre-conditioning of very deep learning models. We show that by introducing maxout and batch normalisation units to the network in network model results in a model that produces classification results that are better than or comparable to the current state of the art in CIFAR-10, CIFAR-100, MNIST, and SVHN datasets.
Keywords: Training, data models, machine learning, feedforward neural networks, ear, image color analysis.
Rights: Copyright © 2016, IEEE
DOI: 10.1109/WACV.2016.7477624
Grant ID:
Appears in Collections:Aurora harvest 8
Computer Science publications

Files in This Item:
File Description SizeFormat 
  Restricted Access
Restricted Access2.11 MBAdobe PDFView/Open

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.