Approximate Fisher information matrix to characterise the training of deep neural networks

Simple item page
dc.contributor.authorLiao, Z.
dc.contributor.authorDrummond, T.
dc.contributor.authorReid, I.
dc.contributor.authorCarneiro, G.
dc.date.issued2020
dc.descriptionDate of Publication: 16 October 2018
dc.description.abstractIn this paper, we introduce a novel methodology for characterising the performance of deep learning networks (ResNets and DenseNet) with respect to training convergence and generalisation as a function of mini-batch size and learning rate for image classification. This methodology is based on novel measurements derived from the eigenvalues of the approximate Fisher information matrix, which can be efficiently computed even for high capacity deep models. Our proposed measurements can help practitioners to monitor and control the training process (by actively tuning the mini-batch size and learning rate) to allow for good training convergence and generalisation. Furthermore, the proposed measurements also allow us to show that it is possible to optimise the training process with a new dynamic sampling training approach that continuously and automatically change the mini-batch size and learning rate during the training process. Finally, we show that the proposed dynamic sampling training approach has a faster training time and a competitive classification accuracy compared to the current state of the art.
dc.description.statementofresponsibilityZhibin Liao, Tom Drummond, Ian Reid, Gustavo Carneiro,
dc.identifier.citationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020; 42(1):15-26
dc.identifier.doi10.1109/TPAMI.2018.2876413
dc.identifier.issn0162-8828
dc.identifier.issn1939-3539
dc.identifier.orcidLiao, Z. [0000-0001-9965-4511]
dc.identifier.orcidReid, I. [0000-0001-7790-6423]
dc.identifier.orcidCarneiro, G. [0000-0002-5571-6220]
dc.identifier.urihttp://hdl.handle.net/2440/117297
dc.language.isoen
dc.publisherIEEE
dc.relation.granthttp://purl.org/au-research/grants/arc/DP180103232
dc.relation.granthttp://purl.org/au-research/grants/arc/CE140100016
dc.relation.granthttp://purl.org/au-research/grants/arc/FL130100102
dc.rights© IEEE
dc.source.urihttps://doi.org/10.1109/tpami.2018.2876413
dc.titleApproximate Fisher information matrix to characterise the training of deep neural networks
dc.typeJournal article
pubs.publication-statusPublished

Files

Collections

Computer Science publications
Australian Institute for Machine Learning publications