Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition

The development of multimodal models has significantly advanced multimodalsentiment analysis and emotion recognition. However, in real-worldapplications, the presence of various missing modality cases often leads to adegradation in the model’s performance. In this work, we propose a novelmultimodal Transformer framework using prompt learning to address the issue ofmissing modalities. Our method introduces three types of prompts: generativeprompts, missing-signal prompts, and missing-type prompts. These prompts enablethe generation of missing modality features and facilitate the learning ofintra- and inter-modality information. Through prompt learning, we achieve asubstantial reduction in the number of trainable parameters. Our proposedmethod outperforms other methods significantly across all evaluation metrics.Extensive experiments and ablation studies are conducted to demonstrate theeffectiveness and robustness of our method, showcasing its ability toeffectively handle missing modalities.

Further reading#

• Access Paper in arXiv.org