Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities

Multimodal sentiment analysis (MSA) aims to understand human sentimentthrough multimodal data. Most MSA efforts are based on the assumption ofmodality completeness. However, in real-world applications, some practicalfactors cause uncertain modality missingness, which drastically degrades themodel’s performance. To this end, we propose a Correlation-decoupled KnowledgeDistillation (CorrKD) framework for the MSA task under uncertain missingmodalities. Specifically, we present a sample-level contrastive distillationmechanism that transfers comprehensive knowledge containing cross-samplecorrelations to reconstruct missing semantics. Moreover, a category-guidedprototype distillation mechanism is introduced to capture cross-categorycorrelations using category prototypes to align feature distributions andgenerate favorable joint representations. Eventually, we design aresponse-disentangled consistency distillation strategy to optimize thesentiment decision boundaries of the student network through responsedisentanglement and mutual information maximization. Comprehensive experimentson three datasets indicate that our framework can achieve favorableimprovements compared with several baselines.

Further reading#

• Access Paper in arXiv.org