Temporal pyramid pooling based convolutional neural network for action recognition

Simple item page
dc.contributor.authorWang, P.
dc.contributor.authorCao, Y.
dc.contributor.authorShen, C.
dc.contributor.authorLiu, L.
dc.contributor.authorShen, H.T.
dc.date.issued2016
dc.description.abstractEncouraged by the success of Convolutional Neural Networks (CNNs) in image classification, recently much effort is spent on applying CNNs to video based action recognition problems. One challenge is that video contains a varying number of frames which is incompatible to the standard input format of CNNs. Existing methods handle this issue either by directly sampling a fixed number of frames or bypassing this issue by introducing a 3D convolutional layer which conducts convolution in spatial-temporal domain. In this paper we propose a novel network structure which allows an arbitrary number of frames as the network input. The key of our solution is to introduce a module consisting of an encoding layer and a temporal pyramid pooling layer. The encoding layer maps the activation from previous layers to a feature vector suitable for pooling while the temporal pyramid pooling layer converts multiple frame-level activations into a fixed-length video-level representation. In addition, we adopt a feature concatenation layer which combines appearance information and motion information. Compared with the frame sampling strategy, our method avoids the risk of missing any important frames. Compared with the 3D convolutional method which requires a huge video dataset for network training, our model can be learned on a small target dataset because we can leverage the off-the-shelf image-level CNN for model parameter initialization. Experiments on three challenging datasets, Hollywood2, HMDB51 and UCF101 demonstrate the effectiveness of the proposed network.
dc.description.statementofresponsibilityPeng Wang, Yuanzhouhan Cao, Chunhua Shen, Lingqiao Liu, Heng Tao Shen
dc.identifier.citationIEEE Transactions on Circuits and Systems for Video Technology, 2016; 27(99):1-8
dc.identifier.doi10.1109/TCSVT.2016.2576761
dc.identifier.issn1051-8215
dc.identifier.issn1558-2205
dc.identifier.orcidShen, C. [0000-0002-8648-8718]
dc.identifier.urihttp://hdl.handle.net/2440/108774
dc.language.isoen
dc.publisherInstitute of Electrical and Electronics Engineers
dc.rights© IEEE
dc.source.urihttps://doi.org/10.1109/tcsvt.2016.2576761
dc.subjectTemporal pyramid pooling; action recognition; convolutional neural network
dc.titleTemporal pyramid pooling based convolutional neural network for action recognition
dc.typeJournal article
pubs.publication-statusPublished

Files

Original bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
RA_hdl_108774.pdf
Size:
5.88 MB
Format:
Adobe Portable Document Format
Description:
Restricted Access
Download

Collections

Electrical and Electronic Engineering publications