Faculty Dr Abir Sen

Dr Abir Sen

Assistant Professor

Department of Computer Science and Engineering

Contact Details

abir.s@srmap.edu.in

Office Location

CV Block, Level 2, Cubicle No: 2

Social Links

Education

2024
PhD
National Institute of Technology, Rourkela, Odisha
India
2017
M.Tech
National Institute of Technology, Meghalaya, Meghalaya
India
2015
B.Tech
National Institute of Technology, Durgapur, West Bengal
India

Personal Website

Experience

  • SRM University-AP, Amaravati | Present-
  • Assistant Professor in KIIT Bhubaneswar

Research Interest

  • My research interests are protocols for Delay Tolerant Networks. I am particularly interested in incorporating delay tolerance over Internet of Things (IoT), which helps to interconnect physical world smart entities is Internet of Things (IoT). Building IoT over DTN is possible in case of limited connectivity.
  • My research interest are Computer Vision, DL and biomedical image processing

Awards

Memberships

  • IEEE

Publications

  • Navigating Ad Hoc Networks: Current Challenges and Future Perspectives

    Behura A., Khilar P.M., Sen A.

    Book chapter, Studies in Big Data, 2027, DOI Link

    View abstract ⏷

    Over the past few years, ad hoc networks have developed very fast and become a leading and emerging field of research. Ad hoc networks are without fixed infrastructure and consist of many categories such as Mobile Ad Hoc Networks (MANETs), Vehicular Ad Hoc Networks (VANETs), Flying Ad Hoc Networks (FANETs), Underground Ad Hoc Networks (UNDGANs), and Sensor Ad Hoc Networks (SANETs). They are general-purpose in applications like disaster management, military missions, environment monitoring, surveillance, safety systems, security services, weather forecasting, and water quality analysis. Ad hoc networks are categorized by communication methods, signal traveling, mission goals, usage environments, mobility behavior, and node deployment. Each category has a different domain. MANETs enable communication between mobile users within terrestrial environments, with the ability to have flexibility in environments with low infrastructure. VANETs provide vehicle-to-vehicle and vehicle-to-infrastructure communication, facilitating road safety and traffic efficiency in urban, rural, and highway environments. FANETs facilitate high-speed aerial communication between UAVs (drones), appropriate for surveillance, reconnaissance, and remote sensing. UNDGANs are designed for communication in underground environments, appropriate for mining, pipeline monitoring, and tunnel surveillance. SANETs function in aquatic environments, having a key role in marine life tracking and underwater condition monitoring. Although they are increasingly significant, there has been a lack of comprehensive and concerted reviews on all these subdomains. This survey will act as a basic reference point, investigating architecture designs, real-world deployment, key features, MAC protocols, routing methods, security issues, comparative studies, current research activity, various challenges, and future directions in ad hoc networking.
  • Network Intrusion Detection Using AI: A Comparative Study and Evaluation

    Koirala B., Adhikari I., Behura A., Sen A., Khilar P.M.

    Book chapter, Studies in Big Data, 2027, DOI Link

    View abstract ⏷

    The growing complexity of cyberattacks in future-generation network infrastructures requires advanced intrusion detection systems with adaptive learning and response functions to respond to dynamic attack vectors. Various attacks can compromise the reliability of vehicular applications and services, leading to incorrect decision-making by vehicles and infrastructure, and potentially causing accidents or traffic congestion. For example, a malicious node could inject false information about traffic conditions, leading vehicles to take unnecessary detours or make unsafe maneuvers. Thus, an attacker can easily inject any leading system faults. This chapter offers a comprehensive comparative analysis of artificial intelligence(AI) techniques applied to network intrusion detection, comparing the performance of conventional machine learning techniques with deep learning models. By performing extensive experimentation across four well-known benchmark datasets—Canadian Institute for Cybersecurity Intrusion Detection System 2018 (CICIDS-2018), Canadian Institute for Cybersecurity Intrusion Detection System 2017 (CICIDS-2017), Aegean Wi-Fi Intrusion Dataset (AWID), and Network Security Laboratory Knowledge Discovery Dataset (NSL-KDD)—we comparatively evaluate Random Forest (RF), Decision Tree (DT), and Logistic Regression (LR) classifiers against Long Short-Term Memory (LSTM) networks. Empirical evaluation is conducted across various network traffic scenarios, including distributed denial-of-service (DDoS) attacks, brute-force intrusions, and wireless network vulnerabilities. The results affirm the superior performance of LSTM networks, which achieved outstanding accuracy levels of 99.56, 98.87, 99.69, and 98.77% on their respective datasets, significantly outperforming traditional machine learning models. The performance of the LSTM architecture for recognizing temporal dependencies and sequential patterns in network traffic data defines its paradigmatic superiority in identifying sophisticated cyber attacks. These findings and analysis indicate the efficiency of deep learning methods to boost the security of cyber-physical systems and deal with future security issues, specifically in TCP-based networks as well as centralized and decentralized environments.
  • HGR-FYOLO: a robust hand gesture recognition system for the normal and physically impaired person using frozen YOLOv5

    Sen A., Dombe S., Mishra T.K., Dash R.

    Article, Multimedia Tools and Applications, 2024, DOI Link

    View abstract ⏷

    Hand gesture recognition is important in human-machine interaction (HMI), enabling interaction with the systems without physically touching them. But current methodologies regarding gesture recognition face different challenges, such as poor lighting conditions, complex backgrounds, lower detection rates, slower speed, etc. The physically handicapped people with fewer fingers find it difficult to engage with desktop applications in case of complex backgrounds. To overcome these concerns, we have chosen the YOLOv5s model, one small version of YOLOv5 (You Only Look Once) object detection algorithm, to detect and classify the hand portion in a real-time scenario. We have fine-tuned the YOLOv5s architecture by freezing some convolutional layers in the backbone portion, which reduces the number of parameters, model size, and inference time. Two datasets, one public (American sign language) and one custom dataset, named ‘NITR-Hand gesture’ dataset have been utilized to evaluate the suggested work. There is always a trade-off between accuracy and detection speed. Therefore, the accuracy has been slightly reduced after freezing the backbone of YOLOv5s architecture. but the inference time and detection speed have both improved. Experimental results exhibit that our suggested frozen YOLOv5s model has achieved a mean average precision (mAP@50-95) of 92.60%, and the average speed has reached more than 55 frames per second (fps). We have conducted a comparative analysis of our used models with other state-of-the-art methods, and it is noticed that the fine-tuned YOLOv5s have outperformed other models in terms of mAP and inference speed. This prototype is very fruitful for physically impaired persons to interact with the systems in real-time.
  • Deep Learning-Based Hand Gesture Recognition System and Design of a Human–Machine Interface

    Sen A., Mishra T.K., Dash R.

    Article, Neural Processing Letters, 2023, DOI Link

    View abstract ⏷

    Hand gesture recognition plays an important role in developing effective human–machine interfaces (HMIs) that enable direct communication between humans and machines. But in real-time scenarios, it is difficult to identify the correct hand gesture to control an application while moving the hands. To address this issue, in this work, a low-cost hand gesture recognition system based human-computer interface (HCI) is presented in real-time scenarios. The system consists of six stages: (1) hand detection, (2) gesture segmentation, (3) feature extraction and gesture classification using five pre-trained convolutional neural network models (CNN) and vision transformer (ViT), (4) building an interactive human–machine interface (HMI), (5) development of a gesture-controlled virtual mouse, (6) smoothing of virtual mouse pointer using of Kalman filter. In our work, five pre-trained CNN models (VGG16, VGG19, ResNet50, ResNet101, and Inception-V1) and ViT have been employed to classify hand gesture images. Two multi-class datasets (one public and one custom) have been used to validate the models. Considering the model’s performances, it is observed that Inception-V1 has significantly shown a better classification performance compared to the other four CNN models and ViT in terms of accuracy, precision, recall, and F-score values. We have also expanded this system to control some multimedia applications (such as VLC player, audio player, playing 2D Super-Mario-Bros game, etc.) with different customized gesture commands in real-time scenarios. The average speed of this system has reached 25 fps (frames per second), which meets the requirements for the real-time scenario. Performance of the proposed gesture control system obtained the average response time in milisecond for each control which makes it suitable for real-time. This model (prototype) will benefit physically disabled people interacting with desktops.
  • A novel hand gesture detection and recognition system based on ensemble-based convolutional neural network

    Sen A., Mishra T.K., Dash R.

    Article, Multimedia Tools and Applications, 2022, DOI Link

    View abstract ⏷

    Nowadays, hand gesture recognition has become an alternative for human-machine interaction. It has covered a large area of applications like 3D game technology, sign language interpreting, VR (virtual reality) environment, and robotics. But detection of the hand portion has become a challenging task in computer vision and pattern recognition communities. Deep learning algorithm like convolutional neural network (CNN) architecture has become a very popular choice for classification tasks, but CNN architectures suffer from some problems like high variance during prediction, overfitting problem and also prediction errors. To overcome these problems, an ensemble of CNN-based approaches is presented in this paper. Firstly, the gesture portion is detected by using the background separation method based on binary thresholding. After that, the contour portion is extracted, and the hand region is segmented. Then, the images have been resized and fed into three individual CNN models to train them in parallel. In the last part, the output scores of CNN models are averaged to construct an optimal ensemble model for the final prediction. Two publicly available datasets (labeled as Dataset-1 and Dataset-2) containing infrared images and one self-constructed dataset have been used to validate the proposed system. Experimental results are compared with the existing state-of-the-art approaches, and it is observed that our proposed ensemble model outperforms other existing proposed methods.

Patents

Projects

Scholars

Interests

  • Biomedical Image Processing
  • Computer Vision
  • Deep Learning

Thought Leaderships

There are no Thought Leaderships associated with this faculty.

Top Achievements

Research Area

No research areas found for this faculty.

Computer Science and Engineering is a fast-evolving discipline and this is an exciting time to become a Computer Scientist!

Computer Science and Engineering is a fast-evolving discipline and this is an exciting time to become a Computer Scientist!

Recent Updates

No recent updates found.

Education
2015
B.Tech
National Institute of Technology, Durgapur
India
2017
M.Tech
National Institute of Technology, Meghalaya
India
2024
PhD
National Institute of Technology, Rourkela
India
Experience
  • SRM University-AP, Amaravati | Present-
  • Assistant Professor in KIIT Bhubaneswar
Research Interests
  • My research interests are protocols for Delay Tolerant Networks. I am particularly interested in incorporating delay tolerance over Internet of Things (IoT), which helps to interconnect physical world smart entities is Internet of Things (IoT). Building IoT over DTN is possible in case of limited connectivity.
  • My research interest are Computer Vision, DL and biomedical image processing
Awards & Fellowships
Memberships
  • IEEE
Publications
  • Navigating Ad Hoc Networks: Current Challenges and Future Perspectives

    Behura A., Khilar P.M., Sen A.

    Book chapter, Studies in Big Data, 2027, DOI Link

    View abstract ⏷

    Over the past few years, ad hoc networks have developed very fast and become a leading and emerging field of research. Ad hoc networks are without fixed infrastructure and consist of many categories such as Mobile Ad Hoc Networks (MANETs), Vehicular Ad Hoc Networks (VANETs), Flying Ad Hoc Networks (FANETs), Underground Ad Hoc Networks (UNDGANs), and Sensor Ad Hoc Networks (SANETs). They are general-purpose in applications like disaster management, military missions, environment monitoring, surveillance, safety systems, security services, weather forecasting, and water quality analysis. Ad hoc networks are categorized by communication methods, signal traveling, mission goals, usage environments, mobility behavior, and node deployment. Each category has a different domain. MANETs enable communication between mobile users within terrestrial environments, with the ability to have flexibility in environments with low infrastructure. VANETs provide vehicle-to-vehicle and vehicle-to-infrastructure communication, facilitating road safety and traffic efficiency in urban, rural, and highway environments. FANETs facilitate high-speed aerial communication between UAVs (drones), appropriate for surveillance, reconnaissance, and remote sensing. UNDGANs are designed for communication in underground environments, appropriate for mining, pipeline monitoring, and tunnel surveillance. SANETs function in aquatic environments, having a key role in marine life tracking and underwater condition monitoring. Although they are increasingly significant, there has been a lack of comprehensive and concerted reviews on all these subdomains. This survey will act as a basic reference point, investigating architecture designs, real-world deployment, key features, MAC protocols, routing methods, security issues, comparative studies, current research activity, various challenges, and future directions in ad hoc networking.
  • Network Intrusion Detection Using AI: A Comparative Study and Evaluation

    Koirala B., Adhikari I., Behura A., Sen A., Khilar P.M.

    Book chapter, Studies in Big Data, 2027, DOI Link

    View abstract ⏷

    The growing complexity of cyberattacks in future-generation network infrastructures requires advanced intrusion detection systems with adaptive learning and response functions to respond to dynamic attack vectors. Various attacks can compromise the reliability of vehicular applications and services, leading to incorrect decision-making by vehicles and infrastructure, and potentially causing accidents or traffic congestion. For example, a malicious node could inject false information about traffic conditions, leading vehicles to take unnecessary detours or make unsafe maneuvers. Thus, an attacker can easily inject any leading system faults. This chapter offers a comprehensive comparative analysis of artificial intelligence(AI) techniques applied to network intrusion detection, comparing the performance of conventional machine learning techniques with deep learning models. By performing extensive experimentation across four well-known benchmark datasets—Canadian Institute for Cybersecurity Intrusion Detection System 2018 (CICIDS-2018), Canadian Institute for Cybersecurity Intrusion Detection System 2017 (CICIDS-2017), Aegean Wi-Fi Intrusion Dataset (AWID), and Network Security Laboratory Knowledge Discovery Dataset (NSL-KDD)—we comparatively evaluate Random Forest (RF), Decision Tree (DT), and Logistic Regression (LR) classifiers against Long Short-Term Memory (LSTM) networks. Empirical evaluation is conducted across various network traffic scenarios, including distributed denial-of-service (DDoS) attacks, brute-force intrusions, and wireless network vulnerabilities. The results affirm the superior performance of LSTM networks, which achieved outstanding accuracy levels of 99.56, 98.87, 99.69, and 98.77% on their respective datasets, significantly outperforming traditional machine learning models. The performance of the LSTM architecture for recognizing temporal dependencies and sequential patterns in network traffic data defines its paradigmatic superiority in identifying sophisticated cyber attacks. These findings and analysis indicate the efficiency of deep learning methods to boost the security of cyber-physical systems and deal with future security issues, specifically in TCP-based networks as well as centralized and decentralized environments.
  • HGR-FYOLO: a robust hand gesture recognition system for the normal and physically impaired person using frozen YOLOv5

    Sen A., Dombe S., Mishra T.K., Dash R.

    Article, Multimedia Tools and Applications, 2024, DOI Link

    View abstract ⏷

    Hand gesture recognition is important in human-machine interaction (HMI), enabling interaction with the systems without physically touching them. But current methodologies regarding gesture recognition face different challenges, such as poor lighting conditions, complex backgrounds, lower detection rates, slower speed, etc. The physically handicapped people with fewer fingers find it difficult to engage with desktop applications in case of complex backgrounds. To overcome these concerns, we have chosen the YOLOv5s model, one small version of YOLOv5 (You Only Look Once) object detection algorithm, to detect and classify the hand portion in a real-time scenario. We have fine-tuned the YOLOv5s architecture by freezing some convolutional layers in the backbone portion, which reduces the number of parameters, model size, and inference time. Two datasets, one public (American sign language) and one custom dataset, named ‘NITR-Hand gesture’ dataset have been utilized to evaluate the suggested work. There is always a trade-off between accuracy and detection speed. Therefore, the accuracy has been slightly reduced after freezing the backbone of YOLOv5s architecture. but the inference time and detection speed have both improved. Experimental results exhibit that our suggested frozen YOLOv5s model has achieved a mean average precision (mAP@50-95) of 92.60%, and the average speed has reached more than 55 frames per second (fps). We have conducted a comparative analysis of our used models with other state-of-the-art methods, and it is noticed that the fine-tuned YOLOv5s have outperformed other models in terms of mAP and inference speed. This prototype is very fruitful for physically impaired persons to interact with the systems in real-time.
  • Deep Learning-Based Hand Gesture Recognition System and Design of a Human–Machine Interface

    Sen A., Mishra T.K., Dash R.

    Article, Neural Processing Letters, 2023, DOI Link

    View abstract ⏷

    Hand gesture recognition plays an important role in developing effective human–machine interfaces (HMIs) that enable direct communication between humans and machines. But in real-time scenarios, it is difficult to identify the correct hand gesture to control an application while moving the hands. To address this issue, in this work, a low-cost hand gesture recognition system based human-computer interface (HCI) is presented in real-time scenarios. The system consists of six stages: (1) hand detection, (2) gesture segmentation, (3) feature extraction and gesture classification using five pre-trained convolutional neural network models (CNN) and vision transformer (ViT), (4) building an interactive human–machine interface (HMI), (5) development of a gesture-controlled virtual mouse, (6) smoothing of virtual mouse pointer using of Kalman filter. In our work, five pre-trained CNN models (VGG16, VGG19, ResNet50, ResNet101, and Inception-V1) and ViT have been employed to classify hand gesture images. Two multi-class datasets (one public and one custom) have been used to validate the models. Considering the model’s performances, it is observed that Inception-V1 has significantly shown a better classification performance compared to the other four CNN models and ViT in terms of accuracy, precision, recall, and F-score values. We have also expanded this system to control some multimedia applications (such as VLC player, audio player, playing 2D Super-Mario-Bros game, etc.) with different customized gesture commands in real-time scenarios. The average speed of this system has reached 25 fps (frames per second), which meets the requirements for the real-time scenario. Performance of the proposed gesture control system obtained the average response time in milisecond for each control which makes it suitable for real-time. This model (prototype) will benefit physically disabled people interacting with desktops.
  • A novel hand gesture detection and recognition system based on ensemble-based convolutional neural network

    Sen A., Mishra T.K., Dash R.

    Article, Multimedia Tools and Applications, 2022, DOI Link

    View abstract ⏷

    Nowadays, hand gesture recognition has become an alternative for human-machine interaction. It has covered a large area of applications like 3D game technology, sign language interpreting, VR (virtual reality) environment, and robotics. But detection of the hand portion has become a challenging task in computer vision and pattern recognition communities. Deep learning algorithm like convolutional neural network (CNN) architecture has become a very popular choice for classification tasks, but CNN architectures suffer from some problems like high variance during prediction, overfitting problem and also prediction errors. To overcome these problems, an ensemble of CNN-based approaches is presented in this paper. Firstly, the gesture portion is detected by using the background separation method based on binary thresholding. After that, the contour portion is extracted, and the hand region is segmented. Then, the images have been resized and fed into three individual CNN models to train them in parallel. In the last part, the output scores of CNN models are averaged to construct an optimal ensemble model for the final prediction. Two publicly available datasets (labeled as Dataset-1 and Dataset-2) containing infrared images and one self-constructed dataset have been used to validate the proposed system. Experimental results are compared with the existing state-of-the-art approaches, and it is observed that our proposed ensemble model outperforms other existing proposed methods.
Contact Details

abir.s@srmap.edu.in

Scholars
Interests

  • Biomedical Image Processing
  • Computer Vision
  • Deep Learning

Education
2015
B.Tech
National Institute of Technology, Durgapur
India
2017
M.Tech
National Institute of Technology, Meghalaya
India
2024
PhD
National Institute of Technology, Rourkela
India
Experience
  • SRM University-AP, Amaravati | Present-
  • Assistant Professor in KIIT Bhubaneswar
Research Interests
  • My research interests are protocols for Delay Tolerant Networks. I am particularly interested in incorporating delay tolerance over Internet of Things (IoT), which helps to interconnect physical world smart entities is Internet of Things (IoT). Building IoT over DTN is possible in case of limited connectivity.
  • My research interest are Computer Vision, DL and biomedical image processing
Awards & Fellowships
Memberships
  • IEEE
Publications
  • Navigating Ad Hoc Networks: Current Challenges and Future Perspectives

    Behura A., Khilar P.M., Sen A.

    Book chapter, Studies in Big Data, 2027, DOI Link

    View abstract ⏷

    Over the past few years, ad hoc networks have developed very fast and become a leading and emerging field of research. Ad hoc networks are without fixed infrastructure and consist of many categories such as Mobile Ad Hoc Networks (MANETs), Vehicular Ad Hoc Networks (VANETs), Flying Ad Hoc Networks (FANETs), Underground Ad Hoc Networks (UNDGANs), and Sensor Ad Hoc Networks (SANETs). They are general-purpose in applications like disaster management, military missions, environment monitoring, surveillance, safety systems, security services, weather forecasting, and water quality analysis. Ad hoc networks are categorized by communication methods, signal traveling, mission goals, usage environments, mobility behavior, and node deployment. Each category has a different domain. MANETs enable communication between mobile users within terrestrial environments, with the ability to have flexibility in environments with low infrastructure. VANETs provide vehicle-to-vehicle and vehicle-to-infrastructure communication, facilitating road safety and traffic efficiency in urban, rural, and highway environments. FANETs facilitate high-speed aerial communication between UAVs (drones), appropriate for surveillance, reconnaissance, and remote sensing. UNDGANs are designed for communication in underground environments, appropriate for mining, pipeline monitoring, and tunnel surveillance. SANETs function in aquatic environments, having a key role in marine life tracking and underwater condition monitoring. Although they are increasingly significant, there has been a lack of comprehensive and concerted reviews on all these subdomains. This survey will act as a basic reference point, investigating architecture designs, real-world deployment, key features, MAC protocols, routing methods, security issues, comparative studies, current research activity, various challenges, and future directions in ad hoc networking.
  • Network Intrusion Detection Using AI: A Comparative Study and Evaluation

    Koirala B., Adhikari I., Behura A., Sen A., Khilar P.M.

    Book chapter, Studies in Big Data, 2027, DOI Link

    View abstract ⏷

    The growing complexity of cyberattacks in future-generation network infrastructures requires advanced intrusion detection systems with adaptive learning and response functions to respond to dynamic attack vectors. Various attacks can compromise the reliability of vehicular applications and services, leading to incorrect decision-making by vehicles and infrastructure, and potentially causing accidents or traffic congestion. For example, a malicious node could inject false information about traffic conditions, leading vehicles to take unnecessary detours or make unsafe maneuvers. Thus, an attacker can easily inject any leading system faults. This chapter offers a comprehensive comparative analysis of artificial intelligence(AI) techniques applied to network intrusion detection, comparing the performance of conventional machine learning techniques with deep learning models. By performing extensive experimentation across four well-known benchmark datasets—Canadian Institute for Cybersecurity Intrusion Detection System 2018 (CICIDS-2018), Canadian Institute for Cybersecurity Intrusion Detection System 2017 (CICIDS-2017), Aegean Wi-Fi Intrusion Dataset (AWID), and Network Security Laboratory Knowledge Discovery Dataset (NSL-KDD)—we comparatively evaluate Random Forest (RF), Decision Tree (DT), and Logistic Regression (LR) classifiers against Long Short-Term Memory (LSTM) networks. Empirical evaluation is conducted across various network traffic scenarios, including distributed denial-of-service (DDoS) attacks, brute-force intrusions, and wireless network vulnerabilities. The results affirm the superior performance of LSTM networks, which achieved outstanding accuracy levels of 99.56, 98.87, 99.69, and 98.77% on their respective datasets, significantly outperforming traditional machine learning models. The performance of the LSTM architecture for recognizing temporal dependencies and sequential patterns in network traffic data defines its paradigmatic superiority in identifying sophisticated cyber attacks. These findings and analysis indicate the efficiency of deep learning methods to boost the security of cyber-physical systems and deal with future security issues, specifically in TCP-based networks as well as centralized and decentralized environments.
  • HGR-FYOLO: a robust hand gesture recognition system for the normal and physically impaired person using frozen YOLOv5

    Sen A., Dombe S., Mishra T.K., Dash R.

    Article, Multimedia Tools and Applications, 2024, DOI Link

    View abstract ⏷

    Hand gesture recognition is important in human-machine interaction (HMI), enabling interaction with the systems without physically touching them. But current methodologies regarding gesture recognition face different challenges, such as poor lighting conditions, complex backgrounds, lower detection rates, slower speed, etc. The physically handicapped people with fewer fingers find it difficult to engage with desktop applications in case of complex backgrounds. To overcome these concerns, we have chosen the YOLOv5s model, one small version of YOLOv5 (You Only Look Once) object detection algorithm, to detect and classify the hand portion in a real-time scenario. We have fine-tuned the YOLOv5s architecture by freezing some convolutional layers in the backbone portion, which reduces the number of parameters, model size, and inference time. Two datasets, one public (American sign language) and one custom dataset, named ‘NITR-Hand gesture’ dataset have been utilized to evaluate the suggested work. There is always a trade-off between accuracy and detection speed. Therefore, the accuracy has been slightly reduced after freezing the backbone of YOLOv5s architecture. but the inference time and detection speed have both improved. Experimental results exhibit that our suggested frozen YOLOv5s model has achieved a mean average precision (mAP@50-95) of 92.60%, and the average speed has reached more than 55 frames per second (fps). We have conducted a comparative analysis of our used models with other state-of-the-art methods, and it is noticed that the fine-tuned YOLOv5s have outperformed other models in terms of mAP and inference speed. This prototype is very fruitful for physically impaired persons to interact with the systems in real-time.
  • Deep Learning-Based Hand Gesture Recognition System and Design of a Human–Machine Interface

    Sen A., Mishra T.K., Dash R.

    Article, Neural Processing Letters, 2023, DOI Link

    View abstract ⏷

    Hand gesture recognition plays an important role in developing effective human–machine interfaces (HMIs) that enable direct communication between humans and machines. But in real-time scenarios, it is difficult to identify the correct hand gesture to control an application while moving the hands. To address this issue, in this work, a low-cost hand gesture recognition system based human-computer interface (HCI) is presented in real-time scenarios. The system consists of six stages: (1) hand detection, (2) gesture segmentation, (3) feature extraction and gesture classification using five pre-trained convolutional neural network models (CNN) and vision transformer (ViT), (4) building an interactive human–machine interface (HMI), (5) development of a gesture-controlled virtual mouse, (6) smoothing of virtual mouse pointer using of Kalman filter. In our work, five pre-trained CNN models (VGG16, VGG19, ResNet50, ResNet101, and Inception-V1) and ViT have been employed to classify hand gesture images. Two multi-class datasets (one public and one custom) have been used to validate the models. Considering the model’s performances, it is observed that Inception-V1 has significantly shown a better classification performance compared to the other four CNN models and ViT in terms of accuracy, precision, recall, and F-score values. We have also expanded this system to control some multimedia applications (such as VLC player, audio player, playing 2D Super-Mario-Bros game, etc.) with different customized gesture commands in real-time scenarios. The average speed of this system has reached 25 fps (frames per second), which meets the requirements for the real-time scenario. Performance of the proposed gesture control system obtained the average response time in milisecond for each control which makes it suitable for real-time. This model (prototype) will benefit physically disabled people interacting with desktops.
  • A novel hand gesture detection and recognition system based on ensemble-based convolutional neural network

    Sen A., Mishra T.K., Dash R.

    Article, Multimedia Tools and Applications, 2022, DOI Link

    View abstract ⏷

    Nowadays, hand gesture recognition has become an alternative for human-machine interaction. It has covered a large area of applications like 3D game technology, sign language interpreting, VR (virtual reality) environment, and robotics. But detection of the hand portion has become a challenging task in computer vision and pattern recognition communities. Deep learning algorithm like convolutional neural network (CNN) architecture has become a very popular choice for classification tasks, but CNN architectures suffer from some problems like high variance during prediction, overfitting problem and also prediction errors. To overcome these problems, an ensemble of CNN-based approaches is presented in this paper. Firstly, the gesture portion is detected by using the background separation method based on binary thresholding. After that, the contour portion is extracted, and the hand region is segmented. Then, the images have been resized and fed into three individual CNN models to train them in parallel. In the last part, the output scores of CNN models are averaged to construct an optimal ensemble model for the final prediction. Two publicly available datasets (labeled as Dataset-1 and Dataset-2) containing infrared images and one self-constructed dataset have been used to validate the proposed system. Experimental results are compared with the existing state-of-the-art approaches, and it is observed that our proposed ensemble model outperforms other existing proposed methods.
Contact Details

abir.s@srmap.edu.in

Scholars