Exploring AI Capabilities in Arabic Grammar: Comparative Analysis of ChatGPT and Gemini

Amanda Resti Maulidiya; Maman Abdurrahman; Nalahuddin Saleh

doi:10.15408/a.v11i2.42671

Exploring AI Capabilities in Arabic Grammar: Comparative Analysis of ChatGPT and Gemini

Amanda Resti Maulidiya, Maman Abdurrahman, Nalahuddin Saleh

Abstract

This study aims to compare the performance of ChatGPT and Gemini in analyzing i'rāb Marfû’ât al-Asmâ’', a critical aspect of Arabic grammar for determining the grammatical roles of words in sentences. The analysis uses 11 examples from the book Mulakhash Qawâ'id al-Lugah al-'Arabiyyah by Fuad Ni'mah, focusing on identifying grammatical components such as mubtada', khabar, isim ashbah, isim kāda, khabar inna, fā'il, nā'ib al-fā'il, na'at, 'atf, tawkid, and badal. The study employs the Mann-Whitney test to assess statistical significance and the Cosine Similarity Index (CSI) to measure semantic similarity. Results show that ChatGPT outperforms Gemini with a significant value of 0.019, while the CSI score of 0.800 indicates high similarity between the models' outputs. ChatGPT excels in providing detailed and accurate analyses, while Gemini is more suitable for concise answers but may lack precision. These findings highlight the unique strengths of each model and underscore the necessity of manual correction to ensure the accuracy and relevance of results, particularly in technology-based Arabic grammar learning.

Keywords

ChatGPT; Gemini, Generative AI; I'râb analyzing

Full Text:

PDF

References

Aladeemy, A. A., Alzahrani, A., Algarni, M. H., Alsubari, S. N., Aldhyani, T. H. H., Deshmukh, S. N., Khalaf, O. I., Wong, W.-K., & Aqburi, S. (n.d.). Advancements and Challenges in Arabic Sentiment Analysis: A Decade of Methodologies, Applications, and Resource Development. Heliyon.

Ali, M. A. (2016). Artificial intelligence and natural language processing: The Arabic corpora in online translation software. International Journal of Advanced and Applied Sciences, 3(9).

Aljanabi, M. (2024). Assessing the Arabic Parsing Capabilities of ChatGPT and Cloude: An Expert-Based Comparative Study. Mesopotamian Journal of Arabic Language Studies, 2024.

Anwar, M. R., & Ahyarudin, H. A. (2023). AI-Powered Arabic Language Education in the Era of Society 5.0. IAIC Transactions on Sustainable Digital Innovation (ITSDI), 5(1).

Berliani, D., Azhari, I., & Taufiqurrochman, R. (2024). Exploring The Potential Of Chatgpt As A Technology-Based Tool For Arabic Language Learning: Benefits And Risks. Kitaba, 2(2).

Elkhatat, A. M., Elsaid, K., & Almeer, S. (2023). Evaluating the efficacy of AI content detection tools in differentiating between human and AI-generated text. International Journal for Educational Integrity, 19(1).

Gomez-Cabello, C. A., Borna, S., Pressman, S. M., Haider, S. A., & Forte, A. J. (2024). Large Language Models for Intraoperative Decision Support in Plastic Surgery: A Comparison between ChatGPT-4 and Gemini. Medicina, 60(6).

Khalatia, M., & Al-Romanyb, T. A. H. (2020). Artificial intelligence development and challenges (Arabic language as a model). Artificial Intelligence, 13(5).

Khidhir, A. M. (2024). An AI model for Parsing the Text of Holy Quran Sentences. Mesopotamian Journal of Quran Studies, 2024.

Kishan, A. U., & Reiter, R. E. (2021). Caught Between a Rock and a Hard Place. International Journal of Radiation Oncology, Biology, Physics, 109(4).

Lahitani, A. R., Permanasari, A. E., & Setiawan, N. A. (2016). Cosine similarity to determine similarity measure: Study case in online essay assessment. 2016 4th International Conference on Cyber and IT Service Management.

Mannaa, Z. M., Azmi, A. M., & Aboalsamh, H. A. (2022). Computer-assisted i ‘raab of Arabic sentences for teaching grammar to students. Journal of King Saud University-Computer and Information Sciences, 34(10).

Marsiah, M., Mubarak, M. R., & Audina, N. A. (2021). The students’ perspective towards youtube as the replacement of lecturer in nahwu learning. Jurnal Al Bayan: Jurnal Jurusan Pendidikan Bahasa Arab, 13(2).

Maulida, M., & Marsiah, M. (2023). Mapping Kesulitan Dalam Membaca dan Mengharokati Teks Bahasa Arab Bagi Mahasiswa Pendidikan Bahasa Arab. Journal Of Foreign Language Learning and Teaching, 3(2).

Milenović, Ž. (2011). Application Of Mann-Whitney U Test In Research Of Professional Training Of Primary School Teachers. Metodički Obzori/Methodological Horizons, 6(1).

Mualif, A. (2019). Metodologi pembelajaran ilmu nahwu dalam pendidikan bahasa Arab. AL-HIKMAH (Jurnal Pendidikan Dan Pendidikan Agama Islam), 1(1).

Muawanah, R., & Rifa’i, A. F. (2018). Analisis Kedudukan I’rab Kalimah Bahasa Arab Pada Kitab Al-Imrithi Menggunakan Algoritma Breadth First Search (BFS). JISKA (Jurnal Informatika Sunan Kalijaga), 3(1).

Musthafa, F. A. D. (2024). Penggunaan Artificial Intelligence (AI) dalam Pembelajaran: Fenomena Transformasi Otoritas Pengetahuan di Kalangan Mahasiswa. Journal of Contemporary Islamic Education, 4(1).

Nurmala, M., Sopian, A., Saleh, N., & Syihabuddin, S. (2022). Why is Masdar Important? An Investigating of Masdar and its Translation. Alsuniyat: Jurnal Penelitian Bahasa, Sastra, Dan Budaya Arab, 5(1).

Oti, E. U., Olusola, M. O., & Esemokumo, P. A. (2021). Statistical Analysis of the Median Test and the Mann-Whitney U Test. 7(9).

Pressman, S. M., Borna, S., Gomez-Cabello, C. A., Haider, S. A., & Forte, A. J. (2024). AI in Hand Surgery: Assessing Large Language Models in the Classification and Management of Hand Injuries. Journal of Clinical Medicine, 13(10).

Qadir, J. (2023). Engineering education in the era of ChatGPT: Promise and pitfalls of generative AI for education.

Rahmat, A. S., & Abdurrahman, M. (2017). Analysis of Nahwu Learning Progress in Indonesian Universities Based on the Character of Value Concerns. Tawarikh, 8(2).

Rahutomo, F., Kitasuka, T., & Aritsugi, M. (n.d.). Semantic Cosine Similarity.

Ramli, M. (2023). Mengeksplorasi tantangan etika dalam penggunaan Chat GPT sebagai alat bantu penulisan ilmiah: Pendekatan terhadap integritas akademik. TA’DIBAN: Journal of Islamic Education, 4(1).

Rane, N., Choudhary, S., & Rane, J. (2024). Gemini or ChatGPT? Capability, Performance, and Selection of Cutting-edge Generative Artificial Intelligence (AI) in Business Management. Capability, Performance, and Selection of Cutting-Edge Generative Artificial Intelligence (AI) in Business Management (February 19, 2024).

Siregar, F. H., Hasmayni, B., & Lubis, A. H. (2023). The analysis of chat gpt usage impact on learning motivation among scout students. International Journal of Research and Review, 10(7).

Suryono, M. N. R. N., Bhagaskara, R. E., Pratama, M. A., & Pratama, A. (2023). Analisis Pengaruh ChatGPT Terhadap Produktivitas Mahasiswa. 3(1).

Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19(1).

Wijonarko, P., Wagiman, W., Khana, R., Tundo, T., Salam, A., & James, B. (2023). Penerapan dan Kontribusi Kecerdasan Buatan ChatGPT Untuk Menafsir Teks Hukum (Studi Kasus Penafsiran Pasal 10, Pasal 13, Permenkes No. 889 Tahun 2011). Jurnal Kajian Teknik Elektro, 8(2).

Yatri, D., Anugerahwati, M., & Setyowati, L. (2023). Artificial Intelligence (Ai) In Language Learning (English And Arabic Class): Students’and Teachers’experience And Perceptions. Transformational Language, Literature, And Technology Overview In Learning, 3(1).

DOI: https://doi.org/10.15408/a.v11i2.42671 Abstract - 0 PDF - 0