Integrating Linguistic Features with IndoBERT for Detecting Underage-Associated Comments in Indonesian YouTube
DOI:
https://doi.org/10.15408/aism.v9i2.53149Abstract
Indonesia has issued Government Regulation No. 17 of 2025 on Electronic System Governance for Child Protection (PP TUNAS), which prohibits independent access to digital platforms for individuals under 16. However, age verification remains difficult since users may provide fake birth dates during registration. This study aims to develop a hybrid text classification framework that integrates IndoBERT contextual embeddings with eight handcrafted linguistic features (teen_score, adult_score, emotion_score, grammar_score, word_count, exclamation_count, question_count, and teen_adult_ratio) to identify underage-associated comments on Indonesian YouTube. This study collected 37,450 comments from 271 videos using the YouTube Data API v3 and validated annotation consistency through manual labeling of 500 stratified samples, achieving Cohen's Kappa of 0.853 (almost perfect agreement). To address severe class imbalance, in which under-16 samples accounted for only 1.64%, then applied stratified 5-fold cross-validation with weighted sampling. The hybrid model achieved F1-scores of 0.7777 ± 0.0680 (cross-validation) and 0.9609 (test set). Ablation experiments confirmed explicit age disclosures are essential for the current annotation strategy. Although no significant performance difference was observed versus IndoBERT-only (p = 0.5743), the hybrid model provides explicit linguistic evidence through SHAP and LIME analyses, substantially improving interpretability. This makes the proposed framework suitable for transparent policy-oriented comment screening under PP TUNAS.
Downloads
References
[1] Kementerian Komunikasi dan Digital Republik Indonesia, “Peraturan Menteri Komunikasi dan Digital Nomor 9 Tahun 2026 tentang Peraturan Pelaksanaan Peraturan Pemerintah Nomor 17 Tahun 2025 tentang Tata Kelola Penyelenggaraan Sistem Elektronik dalam Pelindungan Anak,” Jakarta, Indonesia, 2026.
[2] A. Iqbal, M. Younas, S. Iftikhar, F. Fatima, and R. Saleem, “Spam detection using hybrid model on fusion of spammer behavior and linguistics features,” Egypt. Informatics J., vol. 29, p. 100605, 2025, doi: 10.1016/j.eij.2024.100605.
[3] Z. F. Hamida, A. Refoufi, A. Drif, and S. Giordano, “Hybrid-MELAu: A Hybrid Mixing Engineered Linguistic Features Framework Based on Autoencoder for Social Bot Detection,” Informatica, vol. 46, no. 6, pp. 143–159, 2022, doi: 10.31449/inf.v46i6.4081.
[4] Asosiasi Penyelenggara Jasa Internet Indonesia (APJII), “Profil Internet Indonesia 2026,” Jakarta, Indonesia, 2026. [Online]. Available: https://survei.apjii.or.id/survei/group/12
[5] M. Ibriza, M. R. Handayani, W. D. Yuniarti, and K. Umam, “Modeling Political Discourse in Indonesia ’ s 2024 Election Using Unsupervised Machine Learning,” SISFOKOM, vol. 14, no. 2, pp. 174–182, 2025, doi: https://doi.org/10.32736/sisfokom.v14i2.2359.
[6] N. D. P. Nivia, “Pengguna internet di indonesia : tren , tantangan , dan peluang di era digital,” APJII Survei Penetrasi Internet Indones., vol. 3, pp. 604–608, 2025, [Online]. Available: http://urj.uin-malang.ac.id/index.php/mij/index
[7] L. Jansen, A. Sinclair, M. J. van der Goot, R. Fernández, and S. Pezzelle, “Detecting age-related linguistic patterns in dialogue: Toward adaptive conversational systems,” in Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021, Milan, Italy: Accademia University Press, 2022. doi: 10.4000/books.aaccademia.10678.
[8] T. K. Koch, P. Romero, and C. Stachl, “Age and Gender in Language, Emoji, and Emoticon Usage in Instant Messages,” Comput. Human Behav., vol. 126, no. November 2020, p. 106990, 2022, doi: 10.1016/j.chb.2021.106990.
[9] R. Bagestra, A. Misbullah, Z. Zulfan, R. Rasudin, L. Farsiah, and S. A. Nazhifah, “Performance Assessment of Machine Learning and Transformer Models for Indonesian Multi-Label Hate Speech Detection,” Infolitika J. Data Sci., vol. 2, no. 2, pp. 62–71, 2024, doi: 10.60084/ijds.v2i2.235.
[10] T. Dehdarirad, “Evaluating Explainability In Language Classification Models: A Unified Framework Incorporating Feature Attribution Methods and Key Factors Affecting Faithfulness,” Data Inf. Manag., vol. 9, no. 4, p. 100101, 2025, doi: 10.1016/j.dim.2025.100101.
[11] S. M. Lundberg and S. Lee, “A Unified Approach to Interpreting Model Predictions,” in 31st Conference on Neural Information Processing Systems, Long Beach, California USA, 2017, pp. 1–10. doi: https://doi.org/10.48550/arXiv.1705.07874.
[12] M. T. Ribeiro, S. Singh, and C. Guestrin, “‘ Why Should I Trust You ?’ Explaining the Predictions of Any Classifier,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, California, USA, 2016. doi: https://doi.org/10.48550/arXiv.1602.04938.
[13] S. Tam, R. Ben Said, and Ö. Ö. Tanriöver, “A ConvBiLSTM Deep Learning Model-Based Approach for Twitter Sentiment Classification,” IEEE Access, vol. 9, pp. 41283–41293, 2021, doi: 10.1109/ACCESS.2021.3064830.
[14] G. Z. Nabiilah, I. N. Alam, E. S. Purwanto, and M. F. Hidayat, “Indonesian multilabel classification using IndoBERT embedding and MBERT classification,” Int. J. Electr. Comput. Eng., vol. 14, no. 1, pp. 1071–1078, 2024, doi: 10.11591/ijece.v14i1.pp1071-1078.
[15] I. Budiman et al., “Classification Performance Comparison of BERT and IndoBERT on Self-Report of COVID-19 Status on Social Media,” J. Comput. Sci. Inst., vol. 30, pp. 61–67, 2024, doi: 10.35784/jcsi.5564.
[16] M. I. Alfiansyah and A. Muzakir, “Enhanced Detection of Indonesian Online Gambling Advertisements Using Multimodal Ensemble Deep Learning,” Int. J. Artif. Intell. Sci., vol. 2, no. 2, pp. 159–175, 2025, doi: 10.63158/ijais.v2i2.49.
[17] A. Nurkholis, S. Alim, H. Saputra, and A. Ferriyan, “Sentiment Analysis of COVID-19 Booster Vaccines on Twitter Using Multi-Class Support Vector Machine,” Appl. Inf. Syst. Manag., vol. 8, no. 1, pp. 29–36, 2025, doi: 10.15408/aism.v8i1.42911.
[18] S. A. Devi and S. Sivakumar, “A Hybrid Ensemble Word Embedding based Classification Model for Multi-document Summarization Process on Large Multi-domain Document Sets,” Int. J. Adv. Comput. Sci. Appl., vol. 12, no. 9, pp. 141–152, 2021, doi: 10.14569/IJACSA.2021.0120918.
[19] T. Bao, N. Ren, R. Luo, B. Wang, G. Shen, and T. Guo, “A BERT-Based Hybrid Short Text Classification Model Incorporating CNN and Attention-Based BiGRU,” J. Organ. End User Comput., vol. 33, no. 6, pp. 1–21, 2021, doi: 10.4018/joeuc.294580.
[20] T. R. Masuzzahra, K. Umam, H. Mustofa, and M. R. Handayani, “HANA : An AI Chatbot for Islamic Jurisprudence on Menstruation using SBERT and TF-IDF,” J. Appl. Informatics Comput., vol. 9, no. 3, pp. 1013–1024, 2025, doi: https://doi.org/10.30871/jaic.v9i3.9449.
[21] S. Gao et al., “Limitations of Transformers on Clinical Text Classification,” IEEE J. Biomed. Heal. INFORMATICS, vol. 25, no. 9, pp. 3596–3607, 2021, doi: 10.1109/JBHI.2021.3062322.
[22] H. Ahmadian, T. F. Abidin, H. Riza, and K. Muchtar, “Hybrid Models for Emotion Classification and Sentiment Analysis in Indonesian Language,” Appl. Comput. Intell. Soft Comput., vol. 2024, 2024, doi: 10.1155/2024/2826773.
[23] A. A. Hafiza and E. B. Setiawan, “Enhancing Cyberbullying Detection on Platform ‘X’ Using IndoBERT and Hybrid CNN-LSTM Model,” J. Tek. Inform., vol. 6, no. 2, pp. 655–672, 2025, doi: 10.52436/1.jutif.2025.6.2.4321.
[24] V. U. Maheswari and R. Priya, “Novel Approach to Offensive Language Detection on Social Media: Tree CNN Integration with Adversarial Bi-LSTM,” Int. J. Eng. Trends Technol., vol. 72, no. 7, pp. 187–197, 2024, doi: 10.14445/22315381/IJETT-V72I7P120.
[25] O. A. Lantang, R. E. C. Sendow, and F. D. Kambey, “Implementation of Bidirectional Long Short-Term Memory and Convolutional Neural Network in Detecting Hoax Content on Social Media,” Appl. Inf. Syst. Manag., vol. 8, no. 1, pp. 151–158, 2025, doi: 10.15408/aism.v8i1.45222.
[26] Sarmini, Sunardi, and A. Fadlil, “Evaluating The Effectiveness of Augmentation and Classifier Algorithms for Fraud Detection: Comparing CGAN and SMOTE with Random Forest and XGBoost,” Appl. Inf. Syst. Manag., vol. 8, no. 2, pp. 221–230, 2025, doi: 10.15408/aism.v8i2.46308.
[27] M. J. Smith, C. Buckton, C. Patterson, and S. Hilton, “User-generated content and influencer marketing involving e-cigarettes on social media: a scoping review and content analysis of YouTube and Instagram,” BMC Public Health, vol. 23, no. 1, pp. 1–10, 2023, doi: 10.1186/s12889-023-15389-1.
[28] N. Alami, M. El Mallahi, H. Amakdouf, and H. Qjidaa, “Hybrid method for text summarization based on statistical and semantic treatment,” Multimed. Tools Appl., vol. 80, no. 13, pp. 19567–19600, 2021, doi: 10.1007/s11042-021-10613-9.
[29] S. Mustafa and M. Hama Saeed, “Empowering text classification with NLP and explainable AI for enhanced interpretability,” J. Electr. Syst. Inf. Technol., vol. 12, 2025, doi: 10.1186/s43067-025-00273-2.
[30] A. Masih et al., “Classifying human vs. AI text with machine learning and explainable transformer models,” Sci. Rep., vol. 15, no. 1, pp. 1–21, 2025, doi: 10.1038/s41598-025-27377-z.
[31] K. Adib, M. R. Handayani, W. D. Yuniarti, and K. Umam, “Opini Publik Pasca-Pemilihan Presiden : Eksplorasi Analisis Sentimen Media Sosial X Menggunakan SVM,” SINTECH (Science Inf. Technol. J., vol. 7, no. 2, pp. 80–91, 2024, doi: https://doi.org/10.31598/sintechjournal.v7i2.1581.
[32] Q. Aini, E. Rahajeng, M. Tiohandra, H. A. Pratama, and J. Hammad, “Predictive Modeling of Student Dropout Using Academic Data and Machine Learning Techniques,” Appl. Inf. Syst. Manag., vol. 8, no. 2, pp. 203–212, 2025, doi: 10.15408/aism.v8i2.46659.
[33] N. K. Nissa and E. Yulianti, “Multi-label text classification of Indonesian customer reviews using bidirectional encoder representations from transformers language model,” Int. J. Electr. Comput. Eng., vol. 13, no. 5, pp. 5641–5652, 2023, doi: 10.11591/ijece.v13i5.pp5641-5652.
[34] R. Qasim, W. H. Bangyal, M. A. Alqarni, and A. A. Almazroi, “A Fine-Tuned BERT-Based Transfer Learning Approach for Text Classification,” J. Healthc. Eng., no. 1, 2022, doi: 10.1155/2022/3498123.
[35] M. K. Jain, D. Gopalani, and Y. K. Meena, “Hybrid CNN-BiLSTM model with HHO feature selection for enhanced fake news detection,” Soc. Netw. Anal. Min., vol. 15, 2025, doi: 10.1007/s13278-025-01455-6.
[36] L. Afuan, N. Hidayat, H. Hamdani, H. Ismanto, B. C. Purnama, and D. I. Ramdhani, “Optimizing BERT Models with Fine-Tuning for Indonesian Twitter Sentiment Analysis,” J. Wirel. Mob. Networks, Ubiquitous Comput. Dependable Appl., vol. 16, no. 2, pp. 248–267, 2025, doi: 10.58346/JOWUA.2025.I2.016.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Inez Nurdika Irzani, Maya Rini Handayani, Dwi Pebrianti, Khothibul Umam
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
