Evaluating Machine Translation of Cultural Terms: Readability Comparison Between Google and Yandex
Abstract
Purpose
This study aimed to analyze the readability of Google Translate (GT) and Yandex Translate (YT) translation results on dialogue texts containing cultural terms from the book Antologi Cerita Anak Indonesia (ACAI). This study evaluated the effectiveness of the Neural Machine Translation (NMT) approach in GT and the Hybrid Machine Translation (HMT) approach in YT in conveying text meanings clearly and comprehensibly to readers.
Method
This research employed a cloze test involving 28 participants aged 18-24 years, along with a questionnaire to assess user preferences regarding GT and YT translation results. Text readability was analyzed using the Flesch-Kincaid Grade Level and Gunning Fog Index to measure the linguistic complexity of the translations.
Results/Findings
The study results show that GT's readability reaches 81.1%, while YT's readability is 74.5%, both categorized as the independent level according to Rankin & Culhane's (1969) criteria. Additionally, 80% of the 20 questionnaire respondents stated that GT's translations were clearer than those of YT. Analysis using the Flesch-Kincaid Grade Level and Gunning Fog Index shows that the readability level of GT and YT translations is classified as advanced suitable for readers with a minimum education level equivalent to a bachelor's degree.
Conclusion
This study showed that GT has a higher readability level than YT, which might be because of its use of NMT, producing more natural sentence structures. Meanwhile, YT, which also relied on SMT, translates based on statistical patterns, making its translations more rigid. Although both systems could produce comprehensible translations, they still struggled with accurately translating cultural terms without additional context. Therefore, human involvement remained essential to improving accuracy and contextual appropriateness in machine translation.
Keywords
References
Ajhuri, K. F. (2019). Psikologi Perkembangan: Pendekatan Sepanjang Rentang Kehidupan. In Lukman (Ed.), Psikologi Perkembangan Pendekatan Sepanjang Rentang Kehidupan. Penebar Media Pustaka.
Aleshko-Ozhevskaya, S. S., & Ternovskaya, I. L. (2023). Titles of Acts: Translation from English into Russian. NSU Vestnik, 21(4), 144. https://doi.org/10.25205/1818-7935-2023-21-4-143-155
Alvionita, S. (2023). Cultural Words Translation Analysis From English Into Indonesian in Burnt Movie. Univeristas Nasional.
Andriani, R., Eriyanti, R. W., & Huda, A. M. (2023). Problem dalam Menggunakan Mesin Terjemahan: Error dalam Menterjemahkan Teks Bahasa Inggris ke dalam Bahasa Indonesia. INNOVATIVE: Journal Of Social Science Research Volume, 3.
Bilal, D. (2013). Comparing Google’s Readability of Search Results to the Flesch Readability Formulae : A Preliminary Analysis on Children’s Search Queries. Proceedings of the 76th ASIS&T Annual Meeting, 1–9.
Dewi, R. P. (2014). Tingkat Keterbacaan Buku Teks cakap Berbahasa Indonesia SMP Kelas VII pada SMP Budya Wacana dan SMP Don Bosco Yogyakarta. Widya Dharma: Jurnal Kependidikan, 26(2), 201–221.
Du, J., & Way, A. (2017). Neural Pre-Translation for Hybrid Machine Translation. In: MT Summit XVI - 16th Machine Translation Summit, 27–37.
Dugonik, J., Maučec, M. S., Verber, D., & Brest, J. (2023). Reduction of Neural Machine Translation Failures by Incorporating Statistical Machine Translation. Mathematics, 11(11), 2484. https://doi.org/10.3390/math11112484
Dwivedi, R. K., Nand, P., & Pal, O. (2023). Evolution of Machine Translation for Indian Regional Languages using Artificial Intelligence. 2023 International Conference on Disruptive Technologies (ICDT), 764–768. https://doi.org/10.1109/ICDT57929.2023.10150776
Fajarini, A., & Suharto, A. W. B. (2022). Early Childhood Learning Based on Local Wisdom of The Banyumas Region in Early Childhood Education Institutions in RA Masyithoh 13 Sokaraja Lor. International Proceedings of Nusantara Raya, 1(1), 186–193. https://doi.org/10.24090/nuraicon.v1i1.126
Flesch, R. (1949). The Art of Readable Writing. Harper & Row.
Gosselin, A. M., Maux, J. Le, & Smaili, N. (2021). Readability of Accounting Disclosures: A Comprehensive Review and Research Agenda. Accounting Perspectives, 20(4), 543–581.
Grundkiewicz, R., & Junczys-Dowmunt, M. (2018). Near Human-Level Performance in Grammatical Error Correction with Hybrid Machine Translation Roman.
Gunning, R. (1971). The Technique of Clear Writing (Revised). McGraw-Hill.
Hardjasujana, A., Suriamiharja, A., Kuswari, U., Koswara, D., & Nurjanah, N. (1999). Evaluasi Keterbacaan Buku Teks Bahasa Sunda untuk Sekolah Dasar di Jawa Barat. Pusat Pembinaan dan Pengembangan Bahasa.
Imanjaya, E. (2024). Wayang Kulit Show, Layar Tancap (Traveling Cinema Show), and History of Pre-Cinema. Journal of Art, Film, Television, Animation, Games and Technology, 3(2), 7–19.
Janiesch, C., Zschech, P., & Heinrich, K. (2021). Machine learning and deep learning. Electron Markets, 31, 685–695. https://doi.org/10.1007/s12525-021-00475-2
Khoshafah, S. A. N. A. K. (2023). Effectiveness of Machine Translation in Rendering Yemeni Culture- Specific Items into English: Sana’ani Dialect as a Case-in-Point. Sana’a University Journal of Human Sciences, 5(1), 637–651.
Krippendorff, K. (2019). Content Analysis An Introduction to Its Methodology (4th ed.). SAGE Publication, Inc. https://doi.org/https://doi.org/10.4135/9781071878781
Lewerissa, Y. A., Ayal, F. W., & Letsoin, Y. N. (2023). Efisiensi Kinerja Sasi Teripang Pasir (Holothuria scabra) Desa Tungu Kepulauan Aru. PAPALELE: Jurnal Penelitian Sosial Ekonomi Perikanan Dan Kelautan, 7(1), 67–76. https://doi.org/https://doi.org/10.30598/papalele.2023.7.1.67
Lusiana, Y., Laksono, P. M., & Hariri, T. (2020). Self-Styling, Popular Culture, and the Construction of Global-Local Identity among Japanese Food Lovers in Purwokerto. I-Pop: International Journal of Indonesian Popular Culture and Communication, 1(1), 21–40. https://doi.org/10.36782/i-pop.v1i1.33
Manalu, H. B. (2024). Itak Gurgur and its Meaning in the Batak Cultural Community. Lincah: Jurnah Inovasi Dan Tren, 2(1), 11–18. https://doi.org/https://doi.org/10.35870/ljit.v2i1.2207
Mentari, D., Al Farisi, M. Z., & Maulani, H. (2024). Machine Translation Shifts on The Meaning Equivalence of Cullture Sentence and Illocuttionary Speech Act: Back-Translation. Journal of Culture, Arts, Literature, and Linguistics, 10(1). https://doi.org/10.30872/calls.v10i1.15168
Morony, S., Flynn, M., McCaffery, K. J., Jansen, J., & Webster, A. C. (2015). Readability of Written Materials for CKD Patients: A Systematic Review. American Journal of Kidney Diseases, 65(6), 842–850. https://doi.org/https://doi.org/10.1053/j.ajkd.2014.11.025
Muin, Abdul; Rakuasa, H. (2023). Sasi Laut as a Culture of Natural Resources Conservation to Overcome the Tragedy of the Commons in Maluku Province. International Journal of Multidisciplinary Approach Research and Science, 1(3), 277–287. https://doi.org/10.59653/ijmars.v1i03.139
Nida, E. A., & Taber, C. R. (1969). The Theory and Practice of Translation. E.J. Brill.
Nonghuloo, M. S., & Rao, N. (2020). Analyses and Modeling of Neural Machine Translation for English to Khasi. International Journal of Recent Technology and Engineering (IJRTE), 9(2), 115–118. https://doi.org/10.35940/ijrte.B3175.079220
Ojha, P. K., Ismail, A., & Srinivasan, K. K. (2021). Perusal of readability with focus on web content understandability. Journal of King Saud University - Computer and Information Sciences, 33(1), 1–10. https://doi.org/https://doi.org/10.1016/j.jksuci.2018.03.007
Olusola, F. J. (2013). Comparative analysis of readability level of basic six pupils in private and public schools in Ibadan Land. Journal of Educational Development and Practice, 4(1), 105–117. https://doi.org/https://doi.org/10.47963/jedp.v4i.960
Petersen, D. (2014). The Essential Oils of Indonesia. American College of Healthcare Sciences.
Prasetiyo, A. (2022). The Phenomenon of Gotong Royong in Java Community: A Case Study Nyumbang. Indonesian Journal of Multidisciplinary Science, 1(7), 792–801. https://doi.org/10.55324/ijoms.v1i7.145
Qudah, K., Muflih, M. K., & Sardiah, S. (2020). The level of readability of the computer sciences textbook among the eleventh grade students in Jordan. Educational Research and Reviews, 15(2), 72–80. https://doi.org/https://doi.org/10.5897/ERR2019.3887
Rankin, E., & Culhane, J. (1969). Comparable Cloze and Multiple-Choice Comprehension Test Scores. Journal of Reading, 13(3), 193–198.
Readable. (2020). Flesch Reading Ease and the Flesch Kincaid Grade Level. https://readable.com/readability/flesch-reading-ease-flesch-kincaid-grade-level/
Sall, S. B., Sharma, R., & Shedamkar. (2013). Example Based Machine Translation Using Natural Language Processing. International Journal of Scientific & Engineering Research, 4(8).
Sari, I. J., Syafira, R., Zakkiya, Y. H., Ambarsari, R., & Saputra, O. (2024). Ethnophysics of Klepon: Exploring Physics Concepts in Traditional Pasuruan Snack. International Journal of Research and Community Empowerment, 2(2), 48–55. https://doi.org/https://doi.org/10.58706/ijorce.v2n2.p48-55
Shaye, S. Al. (2021). Readability Level of Arabic Language Textbook of the Sixth Grade in the State of Kuwait. Journal of Educational and Social Research, 11(4), 197–212.
Sibeko, J. (2024). Harnessing Google Translations to Develop a Readability Corpus for Sesotho: An Exploratory Study. Journal of the Digital Humanities Association of Southern Africa (DHASA), 5(1). https://doi.org/10.55492/dhasa.v5i1.5010
Sormin, E., Harefa, N., Purba, L. S. L., Sumiyati, & Nadeak, B. (2021). Benzoic Acid Isolation from Frankincense. Proceedings of the 2nd Annual Conference on Blended Learning, Educational Technology and Innovation (ACBLETI 2020), 217–221.
Sumarsono, A., & Wasa, C. (2019). Traditional Sasi wisdom in Papua-based nature conservation. IOP Conference Series Earth and Environmental Science, 1–6. https://doi.org/10.1088/1755-1315/235/1/012092
Sutami, H., & Kabul, A. R. (2024). Culinary for The Spirits : The Reality of Cultural Acculturation of The Chinese Peranakan of Palembang. MANDARINABLE: Journal of Chinese Studies Language, Literature, Culture, and Journalism, 03(01), 88–103. https://doi.org/https://doi.org/10.20961/mandarinable.v3i1.1091
Taylor, W. L. (1953). “Cloze Procedure”: A New Tool For Measuring Readability. Journalism Quarterly : Devoted to Research Studies in The Field of Mass Communications. https://doi.org/10.1177/107769905303000401
Torgesen, J. K., Houston, D. D., Rissman, L. M., Decker, S. M., Roberts, G., Vaughn, S., Wexler, J., Francis, D. J., Rivera, M. O., & Lesaux, N. (2017). Academic Literacy Instruction for Adolescents: A Guidance Document from the Center on Instruction. Center on Instruction (RMC Research Corporation) Description:
Winarno, F. G., Ahnan-Winarno, A. D., & Ahnan, W. W. (2017). Tempe: Kumpulan Fakta Menarik Berdasarkan Penelitian (Tempeh: A Collection of Interesting Facts based on Research). Gramedia Pustaka Utama.
Yu, Y., & Gutt, D. (2024). Review Helpfulness Scores vs. Review Unhelpfulness Scores: Two Sides of the Same Coin or Different Coins? IEEE Transactions on Engineering Management., 71, 8031–8044. https://doi.org/10.1109/TEM.2024.3384960
DOI: 10.15408/bat.v31i1.44469
Refbacks
- Evaluating Machine Translation of Cultural Terms: Readability Comparison Between Google Translate and Yandex Translate
- Evaluating Machine Translation of Cultural Terms: Readability Comparison Between Google Translate and Yandex Translate
Copyright (c) 2025 Diana Mentari
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
