Temporally Consistent Referring Video Object Segmentation With Hybrid Memory

Simple item page
dc.contributor.authorMiao, B.
dc.contributor.authorBennamoun, M.
dc.contributor.authorGao, Y.
dc.contributor.authorShah, M.
dc.contributor.authorMian, A.
dc.date.issued2024
dc.description.abstractReferring Video Object Segmentation (R-VOS) methods face challenges in maintaining consistent object segmentation due to temporal context variability and the presence of other visually similar objects. We propose an end-to-end R-VOS paradigm that explicitly models temporal instance consistency alongside the referring segmentation. Specifically, we introduce a novel hybrid memory that facilitates inter-frame collaboration for robust spatio-temporal matching and propagation. Features of frames with automatically generated high-quality reference masks are propagated to segment the remaining frames based on multi-granularity association to achieve temporally consistent R-VOS. Furthermore, we propose a new Mask Consistency Score (MCS) metric to evaluate the temporal consistency of video segmentation. Extensive experiments demonstrate that our approach enhances temporal consistency by a significant margin, leading to top-ranked performance on popular R-VOS benchmarks, i.e., Ref-YouTube-VOS (67.1%) and Ref-DAVIS17 (65.6%). The code is available at https://github.com/bo-miao/HTR.
dc.description.statementofresponsibilityBo Miao, Mohammed Bennamoun, Yongsheng Gao, Mubarak Shah, Ajmal Mian
dc.identifier.citationIEEE Transactions on Circuits and Systems for Video Technology, 2024; 34(11):11373-11385
dc.identifier.doi10.1109/TCSVT.2024.3419119
dc.identifier.issn1051-8215
dc.identifier.issn1558-2205
dc.identifier.orcidMiao, B. [0000-0002-3025-4429]
dc.identifier.urihttps://hdl.handle.net/2440/145804
dc.language.isoen
dc.publisherInstitute of Electrical and Electronics Engineers
dc.relation.granthttp://purl.org/au-research/grants/arc/IH180100002
dc.relation.granthttp://purl.org/au-research/grants/arc/FT210100268
dc.rights© 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
dc.source.urihttps://doi.org/10.1109/tcsvt.2024.3419119
dc.subjectReferring video object segmentation; temporal consistency; deep learning; feature extraction
dc.titleTemporally Consistent Referring Video Object Segmentation With Hybrid Memory
dc.typeJournal article
pubs.publication-statusPublished

Files

Collections

Research Outputs