ArticleProceedings of the conference. Association for Computational Linguistics. Meeting2020
Integrating Multimodal Information in Large Pretrained Transformers.
Article in Proceedings of the conference. Association for Computational Linguistics. Meeting, 2020. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 25 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
25 citing papers in PubMed.
- Structured Prototype Learning with Feature Fusion for Sparse and Asynchronous Audio-Visual Depression Recognition.Sensors (Basel, Switzerland) · 2026Article
- A Prompt-Guided and Quality-Aware Robust Text-Audio Intent Recognition Framework for Elderly Care.Sensors (Basel, Switzerland) · 2026Article
- A cancer-type-aware framework for robust multimodal survival prediction under missing modalities.Briefings in bioinformatics · 2026Article
- The role of multimodality in clinical disease diagnosis: advances, challenges, and opportunities.Frontiers in public health · 2026Review
- Early prediction of postoperative delirium in major surgery patients using multimodal preoperative EHR and medical notes: a retrospective cohort study in the Indiana network for patient care.BMJ public health · 2026Article
- Article
- A text guided multimodal scale path fusion network for multimodal sentiment analysis.Scientific reports · 2025Article
- DAST-GAN: An Adversarial Learning Multimodal Sentiment Analysis Model Based on Dynamic Attention and Spatio-Temporal Fusion.Sensors (Basel, Switzerland) · 2025Article
- Hierarchical Text-Guided Refinement Network for Multimodal Sentiment Analysis.Entropy (Basel, Switzerland) · 2025Article
- Hierarchical cross-modal attention and dual audio pathways for enhanced multimodal sentiment analysis.Scientific reports · 2025Article
- The future of multimodal artificial intelligence models for integrating imaging and clinical metadata: a narrative reviewDiagnostic and interventional radiology (Ankara, Turkey) · 2025Review
- Sm-Nd Isotope Data Compilation from Geoscientific Literature Using an Automated Tabular Extraction Method.Scientific data · 2025Article
- Enhanced sentiment analysis in tourism reviews via multimodal graph convolutional networks.PloS one · 2025Article
- R3DG: Retrieve, Rank, and Reconstruction with Different Granularities for Multimodal Sentiment Analysis.Research (Washington, D.C.) · 2025Article
- Semantic enhancement and cross-modal interaction fusion for sentiment analysis in social media.PloS one · 2025Article
- Affordable and real-time antimicrobial resistance prediction from multimodal electronic health records.Scientific reports · 2024Article
- Improving the Prognostic Evaluation Precision of Hospital Outcomes for Heart Failure Using Admission Notes and Clinical Tabular Data: Multimodal Deep Learning Model.Journal of medical Internet research · 2024Article
- Automated Fusion of Multimodal Electronic Health Records for Better Medical Predictions.Proceedings of the ... SIAM International Conference on Data Mining. SIAM International Conference on Data Mining · 2024Article
- Deep learning on multi-view sequential data: a survey.Artificial intelligence review · 2023Article
- Using Deep Learning to Identify Linguistic Features that Facilitate or Inhibit the Propagation of Anti- and Pro-Vaccine Content on Social Media.IEEE International Conference on Digital Health. IEEE International Conference on Digital Health · 2022Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
Abstract
Recent Transformer-based contextual word representations, including BERT and XLNet, have shown state-of-the-art performance in multiple disciplines within NLP. Fine-tuning the trained contextual models on task-specific datasets has been the key to achieving superior performance downstream. While fine-tuning these pre-trained models is straight-forward for lexical applications (applications with only language modality), it is not trivial for multimodal language (a growing area in NLP focused on modeling face-to-face communication). Pre-trained models don't have the necessary components to accept two extra modalities of vision and acoustic. In this paper, we proposed an attachment to BERT and XLNet called Multimodal Adaptation Gate (MAG). MAG allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning. It does so by generating a shift to internal representation of BERT and XLNet; a shift that is conditioned on the visual and acoustic modalities. In our experiments, we study the commonly used CMU-MOSI and CMU-MOSEI datasets for multimodal sentiment analysis. Fine-tuning MAG-BERT and MAG-XLNet significantly boosts the sentiment analysis performance over previous baselines as well as language-only fine-tuning of BERT and XLNet. On the CMU-MOSI dataset, MAG-XLNet achieves human-level multimodal sentiment analysis performance for the first time in the NLP community.
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.