Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation
Date
2024
Authors
Chen, Y.
Liu, Y.
Wang, H.
Liu, F.
Wang, C.
Frazer, H.
Carneiro, G.
Editors
Advisors
Journal Title
Journal ISSN
Volume Title
Type:
Conference paper
Citation
Proceedings / CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2024, pp.26487-26497
Statement of Responsibility
Yuanhong Chen, Yuyuan Liu, Hu Wang, Fengbei Liu, Chong Wang, Helen Frazer, Gustavo Carneiro
Conference Name
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (16 Jun 2024 - 22 Jun 2024 : Seattle, WA, USA)
Abstract
Audio-visual segmentation (AVS) is a challenging task that involves accurately segmenting sounding objects based on audio-visual cues. The effectiveness of audio-visual learning critically depends on achieving accurate crossmodal alignment between sound and visual objects. Successful audio-visual learning requires two essential components: 1) a challenging dataset with high-quality pixel-level multi-class annotated images associated with audio files, and 2) a model that can establish strong links between audio information and its corresponding visual object. However, these requirements are only partially addressed by current methods, with training sets containing biased audiovisual data, and models that generalise poorly beyond this biased training set. In this work, we propose a new costeffective strategy to build challenging and relatively unbiased high-quality audio-visual segmentation benchmarks. We also propose a new informative sample mining method for audio-visual supervised contrastive learning to leverage discriminative contrastive samples to enforce cross-modal understanding. We show empirical results that demonstrate the effectiveness of our benchmark. Furthermore, experiments conducted on existing AVS datasets and on our new benchmark show that our method achieves state-of-the-art (SOTA) segmentation accuracy¹
School/Discipline
Dissertation Note
Provenance
Description
Access Status
Rights
©2024 IEEE