Hybrid Image Captioning Model

Publications

Hybrid Image Captioning Model

Year : 2023

Publisher : Institute of Electrical and Electronics Engineers Inc.

Source Title : 2022 OPJU International Technology Conference on Emerging Technologies for Sustainable Development, OTCON 2022

Document Type :

Abstract

Image captioning is implemented using Deep learning and NLP (Natural Language Processing) resulting in producing a description of an image. The proposed model generates a caption for an image using a Convolutional Neural Network (CNN) together with a Recurrent Neural Network (RNN) and area of attention. Previously, the image names were used as keys to map the images with descriptions. In order to achieve high performance, in the proposed model the image caption is based on the relationship between the areas of a picture (attention model), the words used in the caption, and the state of an RNN language model. The approach of progressive loading is employed for the loading of the image dataset. Further, for encoding the image dataset into a feature vector, VGG16 a pre-trained CNN is used. The extracted feature vector is given as input to the RNN model. These image encodings are output to a specific type of RNN model known as Long Short-Term Memory (LSTM) networks. Subsequently, the LSTM works on decoding the feature vector and predicts the sequence of words, resulting in the generation of descriptions or captions. The training performance is measured using one of the model’s quantitative analysis metrics known as BLEU.