Real-Time Vision-Based Indian Sign Language Translation Using Deep Learning Techniques

Subham Pandey; Sumaiya Tahseen; Rohit Pathak; Hina Parveen; Maruti Maurya

doi:10.55524/ijircst.2025.13.3.6

Abstract

This work proposes a vision-based approach to real-time sign language translation for Indian Sign Language (ISL). The system uses state-of-the-art deep learning architectures such as CNN (Convolutional Neural Networks), LSTM (Long Short-Term Memory) networks, and Transformer-based encoder-decoder models for gesture recognition in both isolated and continuous forms. Data preprocessing techniques such as DTW (Dynamic Time Warping) were applied to augment and normalize gesture sequences from custom ISL and public ASL datasets. The model performance was quantitatively evaluated using precision, recall, F1-score, BLEU, ROUGE, CER(character error rate) and WER (word error rate). A Transformer-based model outperformed the achieving a BLEU score of 0.74 and a classification accuracy of 96.1%. The developed desktop application enables real-time ISL-to-English translation at 18 FPS without requiring external sensors, while ablation studies validate the benefits of multimodal fusion and pose-language alignment. This work demonstrates a robust, scalable approach to non-intrusive sign language translation, advancing accessibility for the DHH community.

Keywords

Transformer-based Encoder-Decoder, Spatiotemporal Gesture Modeling, Indian Sign Language (ISL), Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), Dynamic Time Warping (DTW), Real-time Sign Language Translation

References

H. Bay, T. Tuytelaars, and L. Van Gool, “SURF: Speeded Up Robust Features,” European Conference on Computer Vision, pp. 404–417, 2006. Available from: https://tinyurl.com/yjbdyb4c
A. Yalciner, "Bag of Visual Words (BoVW)," Available: https://medium.com/@yalcinera/bag-of-visual-words-bovw-cb90c6f3c405
C. Cortes and V. Vapnik, “Support-vector networks,” Machine Learning, vol. 20, no. 3, pp. 273–297, 1995. Available from: https://tinyurl.com/4sx9dk7u
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015. Available from: https://tinyurl.com/3wxunkdw
R. Elakkiya and B. Natarajan, “ISL-CSLTR: Indian Sign Language Dataset for Continuous Sign Language Translation and Recognition,” Mendeley Data, 2021.
R. Verma, “Indian Sign Language Alphabet Dataset,” Kaggle, 2022. Available from: https://shorturl.at/pgh6d
S. Thakar, S. Shah, B. Shah, and A. V. Nimkar, “Sign Language to Text Conversion in Real Time using Transfer Learning,” 2022. Available from: https://doi.org/10.1109/GCAT55367.2022.9971953
C. C. de Amorim and C. Zanchettin, “ASL-Skeleton3D and ASL-Phono: Two Novel Datasets for the American Sign Language,” 2022. Available from:
https://doi.org/10.48550/arXiv.2201.02065
P. Roy, S. Bhattacharya, P. P. Roy, and U. Pal, “Position and Rotation Invariant Sign Language Recognition from 3D Kinect Data with Recurrent Neural Networks,” 2020. Available from:
https://doi.org/10.48550/arXiv.2010.12669
M. Gupta et al., “CNN-LSTM Hybrid Real-Time IoT-Based Cognitive Approaches for ISLR with WebRTC: Auditory Impaired Assistive Technology,” NCBI, 2022. Available from: https://doi.org/10.1155/2022/3978627
M. R. K. et al., “Image-based Indian Sign Language Recognition: A Practical Review using Deep Neural Networks,” 2023. Available from:
https://doi.org/10.48550/arXiv.2304.14710
B. Shi, “Toward American Sign Language Processing in the Real World: Data, Tasks, and Methods,” 2023. Available from: https://doi.org/10.48550/arXiv.2308.12419
I. R. Shaffer, “A study of facial expression recognition technologies on deaf adults and their children,” 2018. Available from: https://tinyurl.com/535wehdu
H. Cate, F. Dalvi, and Z. Hussain, “Sign Language Recognition Using Temporal Classification,” 2017. Available from: https://doi.org/10.48550/arXiv.1701.01875
F. Wen et al., “AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove,” NCBI, 2021. Available from: https://tinyurl.com/4vywrujt
L. A. Khuzayem et al., “Efhamni: A Deep Learning-Based Saudi Sign Language Recognition Application,” NCBI, 2024. Available from: https://doi.org/10.3390/s24103112
S. Albanie et al., “BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues,” 2020. Available from: https://tinyurl.com/y2t8eme2
N. C. Camgoz et al., “Content4All Open Research Sign Language Translation Datasets,” 2021. Available from: https://doi.org/10.1109/FG52635.2021.9667087
A. Desai et al., “ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition,” 2023. https://tinyurl.com/yv6cwmkv
H. Walsh et al., “A Data-Driven Representation for Sign Language Production,” 2024. Available from: https://doi.org/10.1109/FG59268.2024.10581995
P. Roy et al., “American Sign Language Video to Text Translation,” 2024. Available from:
https://doi.org/10.48550/arXiv.2402.07255
B. Fang, J. Co, and M. Zhang, “DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation,” 2018. Available from: https://doi.org/10.1145/3131672.3131693
G. Halvardsson et al., “Interpretation of Swedish Sign Language using Convolutional Neural Networks and Transfer Learning,” 2020. https://tinyurl.com/a88tj2bk
N. Tran et al., “Assessment of Sign Language-Based versus Touch-Based Input for Deaf Users Interacting with Intelligent Personal Assistants,” 2024. Available from: https://doi.org/10.1145/3613904.3642094
M. Huenerfauth and H. Kacorri, “Best practices for conducting evaluations of sign language animation,” 2015. Available from: https://tinyurl.com/as8z3ykc
P. Rust et al., “Towards Privacy-Aware Sign Language Translation at Scale,” 2024. Available from:
https://doi.org/10.48550/arXiv.2402.09611
H. Zhang, S. Li, and M. Sun, “Automatic Sign Language Recognition Using Convolutional Neural Networks,” IEEE ICCV, pp. 154–162, 2019. Available from: https://tinyurl.com/2a3fhh8a
M. Papatsimouli, P. Sarigiannidis, and G. F. Fragulis, "A survey of advancements in real-time sign language translators: Integration with IoT technology," Technologies, vol. 11, no. 4, p. 83, 2023. Available from: https://doi.org/10.3390/technologies11040083
A. M. Buttar, U. Ahmad, A. H. Gumaei, A. Assiri, M. A. Akbar, and B. F. Alkhamees, "Deep learning in sign language recognition: A hybrid approach for the recognition of static and dynamic signs," Mathematics, vol. 11, no. 17, p. 3729, 2023. Available from: https://doi.org/10.3390/math11173729
J. Liu et al., “Multi-Scale Convolutional Neural Networks for Sign Language Recognition,” IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 12, pp. 5124–5137, 2020. Available from: https://doi.org/10.1109/SPACES.2018.8316344
A. S. M. Miah, M. A. M. Hasan, S. Nishimura, and J. Shin, "Sign language recognition using graph and general deep neural network based on large scale dataset," IEEE Access, 2024. Available from: https://doi.org/10.3390/technologies11040083
S. Albanie et al., “Co-Articulated Sign Language Recognition from Video Data,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 42, no. 7, pp. 1637–1650, 2020. Available from: https://tinyurl.com/y2t8eme2
A. S. M. Miah, M. A. M. Hasan, S. Nishimura, and J. Shin, "Sign language recognition using graph and general deep neural network based on large scale dataset," IEEE Access, 2024.. Available from: https://doi.org/10.1109/ACCESS.2024.3372425
A. Halder and A. Tayade, "Real-time vernacular sign language recognition using MediaPipe and machine learning," Int. J. Res. Publ. Rev., vol. 2, no. 2582-7421, 2021. Available from: https://shorturl.at/Ionu8
P. J. Wong et al., “Deep Learning-Based Sign Language Recognition Using Hand Gestures,” Int. J. Adv. Robot. Syst., vol. 17, no. 5, p. 1726, 2021. Available from: https://doi.org/10.1007/s00521-019-04691-y
R. S. Camara and S. Lima, “A Comparison of Deep Learning Architectures for Sign Language Translation,” ICONIP, pp. 357–366, 2021. Available from:
Y. Wang and Z. Li, “Sign Language Recognition via Deep Learning Techniques: A Review,” J. Electr. Eng. Technol., vol. 16, pp. 3589–3596, 2021. Available from: https://tinyurl.com/3wu4983v
D. Rempel, M. J. Camilleri, and D. L. Lee, "The design of hand gestures for human–computer interaction: Lessons from sign language interpreters," Int. J. Hum.-Comput. Stud., vol. 72, no. 10–11, pp. 728–735, 2014. Available from: https://doi.org/10.1016/j.ijhcs.2014.05.003
Y. Xu et al., “Deep Gesture Recognition for Real-Time Sign Language Translation,” IEEE Trans. Human-Mach. Syst., vol. 51, no. 5, pp. 502–511, 2021. Available from: https://doi.org/10.1109/ACCESS.2020.3032140
P. Singh, S. Verma, and A. Saxena, “A Survey on Sign Language Recognition Using Vision-Based Techniques,” IEEE Access, vol. 9, pp. 51001–51015, 2021. Available from: https://doi.org/10.1016/j.measen.2022.100385
M. S. Sahu and R. S. Meher, “Recognition of Sign Language Gestures Using CNN-LSTM Hybrid Model,” J. Electr. Eng. Technol., vol. 17, no. 1, pp. 435–444, 2022. Available from: https://doi.org/10.1007/978-981-97-3591-4_2
M. N. Saiful, A. Al Isam, H. A. Moon, R. T. Jaman, M. Das, M. R. Alam, and A. Rahman, "Real-time sign language detection using CNN," in Proc. 2022 Int. Conf. Data Analytics for Business and Industry (ICDABI), Oct. 2022, pp. 697–701. Available from: https://doi.org/10.1109/ICDABI56818.2022.10041711
W. Khan and M. Haroon, “An unsupervised deep learning ensemble model for anomaly detection in static attributed social networks,” Int. J. Cogn. Comput. Eng., vol. 3, pp. 153–160, 2022. Available from: https://doi.org/10.1016/j.ijcce.2022.08.002
Hyunin Lee, Dongwook Kim, and Yong-Lae Park. Explainable deep learning model for emg-based finger angle estimation using attention. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 30:1877–1886, 2022.https://doi.org/10.1109/TNSRE.2022.3188275

Cites this article as

S. Pandey, S. Tahseen, R. Pathak, H. Parveen, M. Maurya, "Real-Time Vision-Based Indian Sign Language Translation Using Deep Learning Techniques", International Journal of Innovative Research in Engineering & Management (ijircst), Vol-13, Issue-3, Page No-35-46, 2025. Available from: https://doi.org/10.55524/ijircst.2025.13.3.6

Corresponding Author

Subham Pandey

B.Tech Scholar, Computer Science and Engineering, Integral University, Lucknow, Integral University

Download Full Paper

Download PDF

No. of Downloads: 74 | No. of Views: 562

Cybersecurity Incident Detection (IDs) Using Machine Learning

Rehan Raja, Hiba Saleem, Shayan Ahmad, Mohd Arslaan, Nida Khan.

May 2025 - Vol 13, Issue 3
Ethical Challenges in Artificial Intelligence: A Review of Issues and Frameworks

Shivangi Srivastava.

May 2025 - Vol 13, Issue 3
Predictive Modeling for Chemotherapy Response Using Machine Learning

Abdullah Mazharuddin Khaja, Michidmaa Arikhad, Yawar Hayat, Saad Rasool.

May 2025 - Vol 13, Issue 3

Real-Time Vision-Based Indian Sign Language Translation Using Deep Learning Techniques

Citations

Download Full Paper PDF

Total View 562

Total Download 74