Multi-hop visual question generation (VQG) seeks to create coherent, fluent, and contextually rich questions by integrating knowledge from a structured knowledge graph (KG) with information inferred from image pairs, where at least one reasoning step involves a visual relationship between the images. Traditional supervised QG methods which rely on token-level alignment with fixed gold labels struggle to capture diverse valid question formulations. We propose M3RQG (Multimodal Multi-hop Multi-decoder Retrieval-augmented Question Generation), a model-agnostic framework that integrates multimodal inputs (images, KG facts) with a multi-decoder architecture to optimize for multiple labels per sample to design multi-hop questions. M3RQG addresses these challenges: (1) generating meaningful visually-grounded questions given a pair of images, (2) generating rich questions that require multi-hop reasoning across images and KG facts, and (3) integrating diverse question labels during fine-tuning. We extend the WebQA dataset with multi-hop questions generated by GPT-4V and Gemini, resulting in two complementary silver labels per sample. Our approach integrates retrieval-augmented generation (RAG) for accessing external knowledge, a PPO objective with ROUGE-based rewards to prioritize structural correctness, and a named entity overlap loss to improve factual accuracy. Experiments across BART, Phi-3.5, and LLaVA backbones demonstrate significant improvements in fluency, reasoning depth, and relevance. We release our code and dataset to facilitate future research.

This paper introduces SAMAT, a Stereotype-Aware Multimodal Alignment Transformer for detecting and explaining implicit misogyny in memes, where harm arises from subtle visual-textual incongruity and cultural stereotypes. SAMAT integrates three components: a Stereotype Subspace Projection Module (SSPM) that structures representations; a fidelity-based retrieval mechanism aligned with a curated Rationale Bank; and an evidenceconditioned explanation generator. For evaluation, we rely on the MEE corpus with 8,000 explanations, Stereotype Alignment (SAS) and Contextual Faithfulness (CFS) scores. Experiments show that SAMAT achieves a Macro-F1 of 88.1%, surpassing MLLM baselines, while improving retrieval faithfulness (SAS: 0.78) and explanation grounding (CFS: 0.68). Ablations confirm gains stem from structured stereotype projection and evidential retrieval, not scale. SAMAT offers a transparent, culturally grounded framework for accountable content moderation, aligning with Responsible AI objectives.

Humour detection has attracted considerable attention due to its significance in interpreting dialogues across text, visual, and acoustic modalities. However, effective methods to map correlations among different modalities remain an active area of research. In this study, we go beyond traditional machine learning techniques by introducing a Variational Quantum Circuit (VQC) that capitalizes on the inherent quantum properties of superposition, entanglement, and interference. Our proposed model, Quantum Multi-Modal Data Fusion (QuMIN), is designed to better capture and reproduce the interaction across modalities, as well as the internal correlations within each modality. Our introduction of the novel VQC, which augments the DialogueRNN baseline with only an additional 4,809 parameters, signifies a substantial advancement in multi-modal humor detection with improvements of 12.34% in precision, 8.84% in recall and 10.57% in F1 score compared to the state-of-the-art methods.

Throughout the years, major advancements have been made in the field of EEG-based emotion classification. Implementing deep architectures for supervised and unsupervised learning from data has come a long way. This study aims to capitalize on these advancements to classify emotions from EEG signals accurately. It still is, however, a challenging task. The fact that the data we are reliant on changes from person to person calls for an elaborate machine-learning solution that can achieve high degrees of abstraction without sacrificing accuracy and legibility. In this study, the Xception model from Keras API was utilized, as well as wavelet transform for feature extraction, which was then used for classification using different classifiers. These features were classified into three distinct categories: NEGATIVE, POSITIVE and NEUTRAL. To examine the effectiveness of the Xception deep neural net, we compare the results of different classifiers like Support Vector Machine, Random Forest, AdaBoostM1, LogitBoost, Naïve Bayes Updateable and Non-Nested Generalization Exemplars. The random forest ensemble achieved the best results from all the classifiers implemented in this study. It had higher accuracy scores than existing models without compromising on areas like precision, F1 score, and recall value.

Sarcasm detection in unimodal or multimodal setting is a very complex task. Sarcasm, emotion, and sentiment are related to each other, and hence any multitask model could be an effective way to leverage the interdependence among these tasks. In order to better represent these clandestine associations, we avoid solely relying on traditional machine learning methods to encode the relationships between the modalities. In this article, we propose a hybrid quantum model that banks upon the low computational complexity and robust representational power of a variational quantum circuit (VQC) and the tried and tested dense neural network to tackle sentiment, emotion, and sarcasm classification simultaneously. We empirically establish that the quantum properties like superposition, entanglement, and interference will better capture and replicate not only the cross-modal interactions between text, acoustics, and visuals but also the correlations between the three responses. We consider the extended MUStARD dataset to evaluate our proposed hybrid model. The results show that our proposed hybrid quantum framework yields more promising results for the primary task of sarcasm detection with the help of the two secondary classification tasks, viz. sentiment and emotion.

Previous studies on question generation from videos have mostly focused on generating questions about common objects and attributes and hence are not entity-centric. In this work, we focus on the generation of entity-centric information-seeking questions from videos. Such a system could be useful for video-based learning, recommending “People Also Ask” questions, video-based chatbots, and fact-checking. Our work addresses three key challenges: identifying question-worthy information, linking it to entities, and effectively utilizing multimodal signals. Further, to the best of our knowledge, there does not exist a large-scale dataset for this task. Most video question generation datasets are on TV shows, movies, or human activities or lack entity-centric information-seeking questions. Hence, we contribute a diverse dataset of YouTube videos, VIDEOQUESTIONS, consisting of 411 videos with 2265 manually annotated questions. We further propose a model architecture combining Transformers, rich context signals (titles, transcripts, captions, embeddings), and a combination of cross-entropy and contrastive loss function to encourage entity-centric question generation. Our best method yields BLEU, ROUGE, CIDEr, and METEOR scores of 71.3, 78.6, 7.31, and 81.9, respectively, demonstrating practical usability. We make the code and dataset publicly available.

Over the years, memes have evolved into multifaceted narratives on platforms like Instagram, TikTok, and Reddit, blending text, images and audio to amplify humor and engagement. The objective of the task described in this article is to bridge the gap for individuals who may struggle to understand memes due to cultural, geographical, ancillary insights, or relevant exposure constraints, aiming to enhance meme comprehension across diverse audiences. The lack of large datasets for supervised learning and alternatives to resource-intensive vision language models have historically hindered the development of such technology. In this work, we have made strides to overcome these challenges. We introduce "MMD" a Multimodal Meme Dataset comprising 13,494 instances, including 3,134 with audio, rendering it the largest of its kind, with 2.1 times as many samples and 9.5 times as many words in the human annotated meme summary compared to the largest available meme captioning dataset, MemeCap. Our framework, MeSum (Meme Summariser), employs a fusion of Vision Transformer and Large Language Model technologies, providing an efficient alternative to resource-intensive Vision Language Models pioneering the integration of all three modalities, we attain a ROUGE-L score of 0.439, outperforming existing approaches such as zero-shot Gemini, GPT4 Vision, LLaVA and QwenVL which yield scores of 0.259, 0.213, 0.177 and 0.198. We have made our codes and datasets publicly available.

Electroencephalogram (EEG) is widely used in emotion recognition which is achieved by recording the electrical activity of the brain. It is a complex signal because of its high temporal resolution and thus requires sophisticated methods and expertise to be interpreted with a reasonable degree of accuracy. Over the years, significant strides have been made in the field of supervised and unsupervised feature learning from data, using deep architectures. The purpose of this study is to build on some of these improvements for better classification of emotions from EEG signals. This is a rather challenging task, and more so if the data we are reliant on is noted for being unsteady, as it changes from person to person. There is a need for an intricate deep learning algorithm that can achieve high levels of abstraction and can still dish out robust/accurate results. In this paper, we have used the Xception (Chollet in 2017 IEEE Conference on computer vision and pattern recognition (CVPR), pp 1800–1807, 2017 [1]) model from Keras API, further reinforced by fine tuning, to classify emotions into three categories namely NEGATIVE, POSITIVE and NEUTRAL. An open-source EEG dataset from Kaggle (Bird et al. in The international conference on digital image and signal processing (DISP’19). Springer, Berlin, 2019 [2]) was used in this study for the purpose of classification. Our experimental results achieved a precision score of 98.34%, a recall value of 98.33%, and an F1-score of 98.336%. This result outperforms many other popular models based upon support vector machine, k-nearest neighbor, self-organizing maps, etc., whose accuracy usually ranges from anywhere between 53 and 92%.

There has been a sudden increase in demand for algorithms or models to correctly and accurately identify human emotions. The conformity for machines has come a long way from when smart machines capable of reaching a decision on their own were all that was expected of them, to machines capable of understanding what goes on in a person’s brain. Such autonomous agents can prove to be helpful not only in developing smarter machines but also in the field of medicine. Early prediction or recognizing brainwave patterns for epilepsy, seizures, manic depression, etc. is a key to achieve faster aid responses or prevention. In our work, we are limiting our focus to the most common practices used by researchers in this field, which is to obtain the electroencephalogram or EEG data, extract features and implement a classification algorithm. However, we are also trying to capitalize upon the massive improvements made in the field of supervised and unsupervised learning. The robust depth-wise separable convolution architecture called Xception has been implemented in this study to observe its performance as a feature extractor to the notoriously mutating EEG data. The EEG dataset being used in this study is open source. It is available in Kaggle and has three classes, namely positive, negative and neutral. We are implementing wavelet transform along with the Xception architecture to extract features from the dataset which are then classified using support vector machine. We achieve stellar results as a performance score of 98% can be observed for the measures accuracy, precision, recall as well as F1 score.

Most commercial uses of LiDAR prefer high-end LiDAR systems with equally sophisticated software packages that have limited accessibility due to cost or complexity. Our purpose for developing this LiDAR point classification framework is to develop a flexible method for visualizing and processing LiDAR data that is simple and cost-effective, and yet achieves a similar degree of functionality as that of its high-end peers. To that end, we have classified data points from a LiDAR-Lite V2 (Blue Label) to represent the distance and height of obstacles, and the gaps between them. This will enable an autonomous mobile agent to determine palatable paths through the satisfactory gaps.