Enhancing Speech-to-Text and Translation Capabilities for Developing Arabic Learning Games: Integration of Whisper OpenAI Model and Google API Translate
Abstract
This study tackles language barriers in computer-mediated communication by developing an application that integrates OpenAI’s Whisper ASR model and Google Translate machine translation to enable real-time, continuous speech transcription and translation and the processing of video and audio files. The application was developed using the Experimental method, incorporating standards for testing and evaluation. The integration expanded language coverage to 133 languages and improved translation accuracy. Efficiency was enhanced through the use of greedy parameters and the Faster Whisper model. Usability evaluations, based on questionnaires, revealed that the application is efficient, effective, and user-friendly, though minor issues in user satisfaction were noted. Overall, the Speech Translate application shows potential in facilitating transcription and translation for video content, especially for language learners and individuals with disabilities. Additionally, this study introduces an Arabic learning game incorporating an Artificial Neural Network using the CNN algorithm. Focusing on the “Speaking” skill, the game applies to voice and image extraction techniques, achieving a high accuracy rate of 95.52%. This game offers an engaging and interactive method for learning Arabic, a language often considered challenging. The incorporation of Artificial Neural Network technology enhances the effectiveness of the learning game, providing users with a unique and innovative language learning experience. By combining voice and image extraction techniques, the game offers a comprehensive approach to enjoyably improving Arabic speaking skills.
Full Text:
PDFReferences
M. Rezi, J. Quintana, W. Dominic, and L. Darius, “Development of Educandy Platform as an Educational Game to Improve Arabic Language Learning Achievement,” Journal International Inspire Education Technology, 2023, doi: 10.55849/jiiet.v2i1.445.
I. S. Wekke, “Arabic Teaching and Learning: A Model from Indonesian Muslim Minority,” Procedia - Social and Behavioral Sciences, 2015, doi: 10.1016/j.sbspro.2015.04.236.
M. T. A. Ghani, M. Hamzah, W. A. A. W. Daud, and T. R. M. Romli, “The Impact of Mobile Digital Game in Learning Arabic Language at Tertiary Level,” Contemporary Educational Technology, 2022, doi: 10.30935/cedtech/11480.
H. Liang Ni, E. Fadzrin, and A. Shaubari, “Development of Think & Go Road Safety Mobile Game using Gamification Approach,” Applied Information Technology And Computer Science, 2021.
K. Chemnad and A. Othman, “Advancements in Arabic Text-to-Speech Systems: A 22-Year Literature Review,” IEEE Access, 2023, doi: 10.1109/ACCESS.2023.3260844.
K. F. Shaalan, “Arabic GramCheck: A grammar checker for Arabic,” Software - Practice and Experience, 2005, doi: 10.1002/spe.653.
D. Khairani, M. Iqbal, D. Rosyada, Z. Zulkifli, and F. Mintarsih, “Penerimaan Sistem Pembelajaran Bahasa Arab Dengan E-Learning dan Gim di Masa Pandemi COVID-19,” EDUKASI: Jurnal Penelitian Pendidikan Agama dan Keagamaan, 2021, doi: 10.32729/edukasi.v19i3.958.
Y. Peng et al., “Reproducing Whisper-Style Training Using An Open-Source Toolkit And Publicly Available Data,” 2023, doi: 10.1109/ASRU57964.2023.10389676.
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision,” 2023.
A. Waheed, B. Talafha, P. Sullivan, A. R. Elmadany, and M. Abdul-Mageed, “A Robust Dialect-Aware Arabic Speech Recognition System,” 2023.
M. M. Duisenova and A. N. Zhorabekova, “The effectiveness of gamification and artificial intelligence in increasing the motivation and effectiveness of students in learning English in elementary school,” Eurasia Journal of Mathematics, Science and Technology Education, 2023, doi: 10.29333/ejmste/13670.
R. Ma, M. Qian, M. J. F. Gales, and K. M. Knill, “Adapting an Unadaptable ASR System,” 2023, doi: 10.21437/Interspeech.2023-1899.
V. R. and I. A. Funcke, “aiLangu - Real-time Transcription and Translation to Reduce Language Barriers,” KTH Royal Institute of Technology, 2023.
A. Ahmed, Y. Hifny, K. Shaalan, and S. Toral, “End-to-End Lexicon Free Arabic Speech Recognition Using Recurrent Neural Networks,” in Computational Linguistics, Speech and Image Processing for Arabic Language, 2018.
D. Wang, X. Wang, and S. Lv, “An overview of end-to-end automatic speech recognition,” Symmetry. 2019, doi: 10.3390/sym11081018.
A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” 2020.
A. Paszke et al., “PyTorch: An imperative style, high-performance deep learning library,” 2019.
S. Rangineni, “An Analysis of Data Quality Requirements for Machine Learning Development Pipelines Frameworks,” International Journal of Computer Trends and Technology, 2023, doi: 10.14445/22312803/ijctt-v71i8p103.
R. Yakubovskyi and Y. Morozov, “Speech Models Training Technologies Comparison Using Word Error Rate,” Advances in Cyber-Physical Systems, 2023, doi: 10.23939/acps2023.01.074.
R. Putri Fajriati, D. Khairani, N. Faizah Rozy, N. Husin, L. Wiyartanti, and T. Rosyadi, “Towards the Implementation of Arabic Language Mobile Apps Learning: Designed by User Insight,” 2020, doi: 10.1109/CITSM50537.2020.9268901.
D. Macháček, R. Dabre, and O. Bojar, “Turning Whisper into Real-Time Transcription System,” 2024, doi: 10.18653/v1/2023.ijcnlp-demo.3.
DOI: https://doi.org/10.15408/jti.v17i2.41240 Abstract - 0 PDF - 0
Refbacks
- There are currently no refbacks.
Copyright (c) 2024 Dewi Khairani, Tabah Rosyadi, Arini, Imam Luthfi Rahmatullah, Fauzan Farhan Antoro
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
3rd Floor, Dept. of Informatics, Faculty of Science and Technology, UIN Syarif Hidayatullah Jakarta
Jl. Ir. H. Juanda No.95, Cempaka Putih, Ciputat Timur.
Kota Tangerang Selatan, Banten 15412
Tlp/Fax: +62 21 74019 25/ +62 749 3315
Handphone: +62 8128947537
E-mail: jurnal-ti@apps.uinjkt.ac.id
Jurnal Teknik Informatika by Prodi Teknik Informatika Universitas Islam Negeri Syarif Hidayatullah Jakarta is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Based on a work at http://journal.uinjkt.ac.id/index.php/ti.
JTI Visitor Counter: View JTI Stats