Analysis of Gradient Degradation and Feature Map Quality in Deep All-Convolutional Neural Networks Compared to Deep Residual Networks
Date
2017
Authors
Gao, W.
McDonnell, M.D.
Editors
Liu, D.
Xie, S.
Li, Y.
Zhao, D.
ElAlfy, E.S.M.
Xie, S.
Li, Y.
Zhao, D.
ElAlfy, E.S.M.
Advisors
Journal Title
Journal ISSN
Volume Title
Type:
Conference paper
Citation
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2017 / Liu, D., Xie, S., Li, Y., Zhao, D., ElAlfy, E.S.M. (ed./s), vol.10635, pp.612-621
Statement of Responsibility
Conference Name
24th International Conference on Neural Information Processing (ICONIP) (14 Nov 2017 - 18 Nov 2017 : Guangzhou, PEOPLES R CHINA)
Abstract
The introduction of skip connections used for summing feature maps in deep residual networks (ResNets) were crucially important for overcoming gradient degradation in very deep convolutional neural networks (CNNs). Due to the strong results of ResNets, it is a natural choice to use features that it produces at various layers in transfer learning or for other feature extraction tasks. In order to analyse how the gradient degradation problem is solved by ResNets, we empirically investigate how discriminability changes as inputs propagate through the intermediate layers of two CNN variants: all-convolutional CNNs and ResNets. We found that the feature maps produced by residual-sum layers exhibit increasing discriminability with layer-distance from the input, but that feature maps produced by convolutional layers do not. We also studied how discriminability varies with training duration and the placement of convolutional layers. Our method suggests a way to determine whether adding extra layers will improve performance and show how gradient degradation impacts on which layers contribute increased discriminability.
School/Discipline
Dissertation Note
Provenance
Description
Access Status
Rights
Copyright 2017 Springer