Enhancing Real Time Communication for Hearing Impaired: A CNN-Based Audio-to-captions system

Publications

Enhancing Real Time Communication for Hearing Impaired: A CNN-Based Audio-to-captions system

Year : 2024

Publisher : Institute of Electrical and Electronics Engineers Inc.

Source Title : 2024 International Conference on Intelligent Systems and Advanced Applications, ICISAA 2024

Document Type :

Abstract

For those with hearing disabilities, real-time speech-to-text change is basic for progressing communication, particularly in energetic settings like online conferences and classes. For comprehensive and profitable communication, this transformation must be precise and speedy. Our work improves the accuracy of speech recognition and makes a software solution that converts speech to text in real-time. This research depicts a architecture based on Variable Span Output Kernel (VOSK) models and Convolutional Neural Systems (CNNs) for a real-time audio-to-captions framework. The method changes over natural sound into spectrograms so that highlights can be extracted. We implemented CNNs with numerous layers for efficient extraction and a VOSK demonstrate to adaptively handle variable speech patterns and complexities. To create accurate captions, a language model is combined with a streamlined pipeline that processes the information in real-time. As a result, those who have hearing loss can take part completely in discussions without losing vital data. The CNN and VOSK models were compared to see whether one superior fulfills the necessities of computational effectiveness and low latency whereas the CNN show reliably accomplished lower latency, making it more fitting for real-time applications where speed is critical, the VOSK model adjusts well to a assortment of speech patterns. This research is basically centered on tending to the troubles of minimizing latency and computing complexity while taking equipment restrictions into consideration, ensuring the system’s effective deployment over a range of gadgets. Through illustrating the CNN model’s prevalent latency in real-time audio-to-text conversion, this paper makes a difference to hearing challenged have better access to communication.