Personality trait identification from social media data has many applications in health monitoring, education, candidate screening, and business. Personality trait identification using text and images published on Social media (Twitter) is challenging due to unpredictable text and the background in the personality trait images. Unlike the existing studies that use multimodal (image and text) for the classification of personality traits images, this study fuses the text of emotions, text in the personality traits image, and image information. We believe that there is a strong correlation between emotion and personality traits. This observation motivated us to combine the textual features of emotional images, text in personality traits images, and image features. The above observations are captured through transformer encoders and multiple convolutional layers, followed by max-pool layers. Before feeding to transformers, our method extracts text in the personality trait images, resulting in text regions and non-text regions separately. Then, the proposed model obtains captions for the emotion and personality trait images. To integrate the strengths of the features extracted from emotion and personality text and images, the proposed work introduces a bilinear fusion approach, which fuses features and modalities. The experiments are conducted on different standard datasets of personality trait images to demonstrate the effectiveness of classification. A comparative study with state-of-the-art methods shows that our method is superior to existing methods. Experiments are also conducted on different races and genders to validate bias and fairness.

Personality traits prediction plays a significant role in several real-world applications, such as improving the education system, improving production in manufacturing, monitoring social media content, sentiment analysis of crowds for opinion mining, judging abnormalities in personal behavior, etc. The demand for personality traits prediction has increased drastically after COVID-19. Therefore, numerous methods have emerged to predict personality traits from a variety of sources, including handwriting, interviews, social media, text, images, and audio. This review focuses on methods developed between the years 2020 and 2024, categorizing them into Handwriting (Graphology), Vision (images and videos), Audio (speech and acoustic signals), Textual (status updates, descriptions), and Multimodal (combinations of the aforementioned) approaches. We critically analyze the methods proposed, the datasets, the scope of the work, the results obtained, and noteworthy remarks. Based on our critical analysis, we notice that increasingly methods tend to use deep learning over handcrafted features. Additionally, personality traits prediction methods are trending more toward multimodal methods because they consistently achieve the highest accuracy among the input modalities. Detailed discussions, tabular presentations, and figures facilitate easy comprehension and future reference. Then, we shed light on the challenges in this field. Many key applications are detailed. Additionally, we highlight significant limitations and offer insights into potential future directions.

Visual Question Answering (VQA) is one of the attractive topics in the field of multimedia, affective, and empathic computing to garner user interest. Unlike existing models which aim at addressing challenges of VQA for the scene images, this work aims at developing a new model for Personality Trait Question Answering (PQA). It uses Twitter account information, which includes shared images, profile pictures, banners, text in the images, and descriptions of the images. Motivated by the accomplishments of the transformer, for encoding visual features of the images, a new InfoGain Multi-Axial Wavelet Vision Transformer (IgMaWaViT) is explored here. For encoding textual features in the images and descriptions, a new Information Gain BERT (InfoBert) method is introduced, which can handle the variable length encoding of text by choosing the optimal discriminator. Furthermore, the model fuses encodings of images and text according to the questions on different personality traits for question answering. The model is called InfoGain Multi-Axial Wavelet Vision Transformer for Personality Traits Question Answering (IgMaWaViT-PQA). To validate the efficacy of the proposed model, a dataset has been constructed, and it is used along with standard datasets for experimentation. Comprehensive experiments show that the proposed model is better than the state-of-the-art models. The code is available at the link: https://github.com/biswaskunal29/InfoGain_MultiAxial_PQA.

Personality traits identification is challenging due to unpredictable changes in the foreground and background of images. In this work, we propose a new deep learning model for personality traits image classification by exploiting visual and textual information from dating images. We believe that there is a strong correlation between the features (visual + textual) of dating images and personality traits images. The features which draw attention are defined as impressive features of dating images and features that convey emotions are expressive features of personality traits images. This observation motivated us to combine the features of dating images to improve the performance of personality traits images classification. To affirm the above observation, we propose multiple convolutional layers followed by max pool layers which extract features from dating and personality traits images simultaneously. To integrate the strengths of impressive and expressive features, the proposed work introduces a dual fusion approach, which fuses features and modalities at different levels. The experiments are conducted on different standard datasets of personality traits images to demonstrate the effectiveness of impressive features in terms of personality trait image classification.

Visual Question Answering (VQA) for personality trait images on social media is challenging because of multiple emotions and actions with complex backgrounds in social media images. This work aims at developing a new VQA model for different personality traits (VQAPT) identification in a single image. This work considers the Big Five Factors (BFF) for personality traits namely, Openness, Conscientiousness, Extraversion, Agreeableness and Neuroticism. VQA is proposed based on the observation that multiple personality traits can be seen in a single image. We propose a model integrating text recognition and person/face recognition to derive the unique relationship between the text and the person's action in the image. Furthermore, a dynamic text-object graph for personality traits identification is constructed according to the query. For understanding a query, we explore the Contrastive Language-Image Pre-trained (CLIP) transformer encoder in this work. Since it is the first work of its kind, we have created a new dataset under this work for evaluation and the dataset is available publicly as mentioned in Section 4. The effectiveness of the proposed method is also evaluated on two benchmark datasets, namely TextVQA for VQA and PTI for personality traits identification.

Aesthetic image analysis is essential for improving the performance of multimedia image retrieval systems, especially from a repository of social media and multimedia content stored on mobile devices. This paper presents a novel method for classifying aesthetic natural scene images by studying the naturalness of image content using statistical features, and reading text in the images using semantic features. Unlike existing methods that focus only on image quality with human information, the proposed approach focuses on image features as well as text-based semantic features without human intervention to reduce the gap between subjectivity and objectivity in the classification. The aesthetic classes considered in this work are (i) Very Pleasant, (ii) Pleasant, (iii) Normal and (iv) Unpleasant. The naturalness is represented by features of focus, defocus, perceived brightness, perceived contrast, blurriness and noisiness, while semantics are represented by text recognition, description of the images and labels of images, profile pictures, and banner images. Furthermore, a deep learning model is proposed in a novel way to fuse statistical and semantic features for the classification of aesthetic natural scene images. Experiments on our own dataset and the standard datasets demonstrate that the proposed approach achieves 92.74%, 88.67% and 83.22% average classification rates on our own dataset, AVA dataset and CUHKPQ dataset, respectively. Furthermore, a comparative study of the proposed model with the existing methods shows that the proposed method is effective for the classification of aesthetic social media images.

Number of users of social media is increasing exponentially. People are getting addicted to social media, and because of such addiction, it sometimes causes psychological and mental effects on the users. Understanding user interaction with social media is essential to study personality traits of the users. This paper focuses on classification of personality traits called Big Five factors, namely (i) agreeableness, (ii) conscientiousness, (iii) neuroticism, (iv) extraversion and (v) openness by combining image and textual features through ontology and fully connected neural network (FCNN). The intuition to classify the images of the five classes is that there is a strong correlation between images, profile picture, images of status, text in the images, description of the images uploaded on social media and the person’s mind. To extract such observation, we explore an ontology-based approach, which constructs a weighted undirected graph (WUG) based on labels of the images, profile picture, banner image, text in the images and description of the images in a novel way for feature extraction. The proposed approach uses mini-batch gradient descent to convert WUG to feature vector form for each word. Furthermore, the feature vectors are fed to the FCNN for classification of personality traits-oriented images. The proposed classification method is evaluated by testing on our dataset of five classes containing 5000 images and two benchmark datasets, namely 5-class dataset which provides 33,556 images, and 10-class dataset which provides 2000 images. The results on different datasets show that the proposed approach is superior to the existing methods in terms of average classification rate.

Estimating the degree of multiple personality traits in a single image is challenging due to the presence of multiple people, occlusion, poor quality etc. Unlike existing methods which focus on the classification of a single personality using images, this work focuses on estimating different personality traits using a single image. We believe that when the image contains multiple persons and modalities, one can expect multiple emotions and expressions. This work separates given input images into different faces of people, recognized text, meta-text and background information using face segmentation, text recognition and scene detection techniques. Contrastive learning is explored to extract features from each segmented region based on clustering. The proposed work fuses textual and visual features extracted from the image for estimating the degree of multiple personality traits. Experimental results on our benchmark datasets show that the proposed model is effective and outperforms the existing methods.

In recent years, the usage of social media has been increasing exponentially because of its various real world applications in digital communication such as content sharing, entertainment, creating awareness, sending alerts, etc. One such task is to upload images/videos, write comments and post user reactions to express feedback, which can then be used to study human personality traits. Classifying images according to different personality traits, like Agreeableness, Conscientiousness, Extraversion, Neuroticism, Openness, etc., is challenging and essential because of several real-world applications mentioned above. This paper proposes a new personality-traits based method for classifying social images using Fuzzy and genetic algorithms. For each user, the proposed approach extracts profile picture, banners and descriptions to construct a set of vocabularies with the help of text detection, recognition and image annotation. For each word in the vocabulary, we employ a fuzzy logic-based method for obtaining a fuzzy co-occurrence matrix by defining the relationship between the words, which results in a fuzzy co-occurrence matrix for each input data point. We also propose a genetic algorithm based fusion method to generate a feature matrix, which is ultimately fed to the fully connected neural network for classification. The effectiveness of the proposed approach is demonstrated on our dataset with five classes containing 5000 images along with four benchmark datasets, namely, (i) five classes of Liu et al.’s dataset (33556 images) (ii) five classes of PERS dataset (28434 images), (iii) ten classes of Krishnani et al.’s dataset (2000 images), and (iv) two classes of facial emotions of FERPlus dataset (26398 images). The results show that the proposed approach outperforms the existing methods for all the datasets in terms of classification rate.

Understanding document images uploaded on social media is challenging because of multiple types like handwritten, printed and scene text images. This study presents a new model called Deep Fuzzy based MSER for classification of multiple document images (like handwritten, printed and scene text). The proposed model detects candidate components that represent dominant information irrespective of the type of document images by combining fuzzy and MSER in a novel way. For every candidate component, the proposed model extracts distance-based features which result in proximity matrix (feature matrix). Further, the deep learning model is proposed for classification by feeding input images and feature matrix as input. To evaluate the proposed model, we create our own dataset and to show effectiveness, the proposed model is tested on standard datasets. The results show that the proposed approach outperforms the existing methods in terms of average classification rate.