MMSMI: Multilingual Multimodal Summarization for Multimodal Input

Publications

MMSMI: Multilingual Multimodal Summarization for Multimodal Input

Author : Dr Ashu Abdul

Year : 2025

Publisher : Springer Science and Business Media Deutschland GmbH

Source Title : Lecture Notes in Networks and Systems

Document Type :

Abstract

Multilingual summarization (MS) takes significant information from a source input and transforms it into a summary of a different language. It is a difficult task requiring a system to analyze, summarize, and translate all at once, closely related to machine translation (MT) and multilingual summarization. There are significantly more training resources available for MT than for MS. Therefore, MS performance would be improved by including the MT corpus. However, the current effort goes beyond a basic multitask framework to incorporate MT. The generation of multimodal data likewise proliferates on day-to-day basics due to the Internet’s rapidly expanding user base. In this paper, we suggest a system that can handle multilingual multimodal summarization for multimodal input (MMSMI), which includes an article’s audio, images, text, and videos. A multioutput (MO) summary for the multimodal input (MI) data is the primary goal of this MMSMI approach. Creating a summary in one language for a set of documents in another is known as cross-linguistic document summarizing. We train our MMSMI approach with Indian languages. We fine-tune the transformers in the proposed approach to provide a better summary. Our approach considers multimodal input data and uses fine-tuned transformers to translate it into a target summary. In our MMSMI approach, we introduce a visual attention function (VAF) to capture the semantic meaning of the images. Datasets from India Today, CNN/Daily Mail, and Indian Express were utilized in our MMSMI method. We use the ROUGE and cosine similarity scores to evaluate the MMSMI approach. Our proposed model outperformed traditional transformers due to the use of multimodal data.