Multi-Modal Learning With Missing Modality via Shared-Specific Feature Modelling

Simple item page
dc.contributor.authorWang, H.
dc.contributor.authorChen, Y.
dc.contributor.authorMa, C.
dc.contributor.authorAvery, J.C.
dc.contributor.authorHull, M.L.
dc.contributor.authorCarneiro, G.
dc.contributor.conferenceIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (17 Jun 2023 - 24 Jun 2023 : Vancouver, Canada)
dc.date.issued2023
dc.description.abstractThe missing modality issue is critical but non-trivial to be solved by multi-modal models. Current methods aiming to handle the missing modality problem in multi-modal tasks, either deal with missing modalities only during evaluation or train separate models to handle specific missing modality settings. In addition, these models are designed for specific tasks, so for example, classification models are not easily adapted to segmentation tasks and vice versa. In this paper, we propose the Shared-Specific Feature Modelling (ShaSpec) method that is considerably simpler and more effective than competing approaches that address the issues above. ShaSpec is designed to take advantage of all available input modalities during training and evaluation by learning shared and specific features to better represent the input data. This is achieved from a strategy that relies on auxiliary tasks based on distribution alignment and domain classification, in addition to a residual feature fusion procedure. Also, the design simplicity of ShaSpec enables its easy adaptation to multiple tasks, such as classification and segmentation. Experiments are conducted on both medical image segmentation and computer vision classification, with results indicating that ShaSpec outperforms competing methods by a large margin. For instance, on BraTS2018, ShaSpec improves the SOTA by more than 3% for enhancing tumour, 5% for tumour core and 3% for whole tumour.
dc.description.statementofresponsibilityHu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, Gustavo Carneiro
dc.identifier.citationProceedings / CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2023, vol.2023-June, pp.15878-15887
dc.identifier.doi10.1109/CVPR52729.2023.01524
dc.identifier.isbn9798350301298
dc.identifier.issn1063-6919
dc.identifier.issn2575-7075
dc.identifier.orcidChen, Y. [0000-0002-8983-2895]
dc.identifier.orcidMa, C. [0000-0002-3270-5609]
dc.identifier.orcidAvery, J.C. [0000-0002-8857-9162]
dc.identifier.orcidHull, M.L. [0000-0003-1813-3971] [0000-0003-4660-4005]
dc.identifier.orcidCarneiro, G. [0000-0002-5571-6220]
dc.identifier.urihttps://hdl.handle.net/2440/139919
dc.language.isoen
dc.publisherIEEE
dc.relation.granthttp://purl.org/au-research/grants/arc/FT190100525
dc.relation.ispartofseriesIEEE Conference on Computer Vision and Pattern Recognition
dc.rights© 2023, IEEE
dc.source.urihttps://ieeexplore.ieee.org/document/10204754
dc.subjectMulti-modal learning
dc.titleMulti-Modal Learning With Missing Modality via Shared-Specific Feature Modelling
dc.typeConference paper
pubs.publication-statusPublished

Files

Collections

Australian Institute for Machine Learning publications