Sketch, ground, and refine: top-down dense video captioning

Simple item page
dc.contributor.authorDeng, C.
dc.contributor.authorChen, S.
dc.contributor.authorChen, D.
dc.contributor.authorHe, Y.
dc.contributor.authorWu, Q.
dc.contributor.conference2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (19 Jun 2021 - 25 Jun 2021 : virtual online)
dc.date.issued2021
dc.description.abstractThe dense video captioning task aims to detect and describe a sequence of events in a video for detailed and coherent storytelling. Previous works mainly adopt a "detect-then-describe" framework, which firstly detects event proposals in the video and then generates descriptions for the detected events. However, the definitions of events are diverse which could be as simple as a single action or as complex as a set of events, depending on different semantic con-texts. Therefore, directly detecting events based on video information is ill-defined and hurts the coherency and accuracy of generated dense captions. In this work, we reverse the predominant "detect-then-describe" fashion, proposing a top-down way to first generate paragraphs from a global view and then ground each event description to a video segment for detailed refinement. It is formulated as a Sketch, Ground, and Refine process (SGR). The sketch stage first generates a coarse-grained multi-sentence paragraph to describe the whole video, where each sentence is treated as an event and gets localised in the grounding stage. In the re-fining stage, we improve captioning quality via refinement-enhanced training and dual-path cross attention on both coarse-grained event captions and aligned event segments. The updated event caption can further adjust its segment boundaries. Our SGR model outperforms state-of-the-art methods on ActivityNet Captioning benchmark under traditional and story-oriented dense caption evaluations. Code will be released at ithub.com/bearcatt/SGR.
dc.description.statementofresponsibilityChaorui Deng, Shizhe Chen, Da Chen, Yuan He, Qi Wu
dc.identifier.citationProceedings / CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2021, pp.234-243
dc.identifier.doi10.1109/CVPR46437.2021.00030
dc.identifier.isbn9781665445092
dc.identifier.issn1063-6919
dc.identifier.issn2575-7075
dc.identifier.orcidDeng, C. [0000-0002-8587-9047]
dc.identifier.orcidWu, Q. [0000-0003-3631-256X]
dc.identifier.urihttps://hdl.handle.net/2440/134309
dc.language.isoen
dc.publisherIEEE
dc.publisher.placeonline
dc.relation.granthttp://purl.org/au-research/grants/arc/DE190100539
dc.relation.ispartofseriesIEEE Conference on Computer Vision and Pattern Recognition
dc.rights© 2021 IEEE.
dc.source.urihttps://ieeexplore.ieee.org/xpl/conhome/9577055/proceeding
dc.titleSketch, ground, and refine: top-down dense video captioning
dc.typeConference paper
pubs.publication-statusPublished

Files

Collections

Computer Science publications