Evaluating User Satisfaction in The Halodoc Application Using a Hybrid CNN-BiLTSM Model for Sentiment Analysis

Dian Kurniasari; Arif Su'admaji; Favorisen Rosyking Lumbanraja; Warsono

doi:10.15408/jti.v18i2.42762

Authors

Dian Kurniasari Department of Mathematics, Faculty of Mathematics and Natural Science, Lampung University, indonesia https://orcid.org/0000-0001-8488-8112
Arif Su'admaji Department of Mathematics, Faculty of Mathematics and Natural Science, Lampung University, Indonesia https://orcid.org/0009-0002-8946-011X
Favorisen Rosyking Lumbanraja Department of Computer Science, Faculty of Mathematics and Natural Science, Lampung University, Indonesia https://orcid.org/0000-0002-1790-831X
Warsono Department of Mathematics, Faculty of Mathematics and Natural Science, Lampung University, Indonesia https://orcid.org/0000-0002-8129-5706

DOI:

https://doi.org/10.15408/jti.v18i2.42762

Keywords:

Hybrid CNN-BiLSTM, Deep Learning, Text Mining, Halodoc

Abstract

The growing demand for digital healthcare services in Indonesia has driven the adoption of Online Healthcare Applications (OHApps) such as Halodoc. Despite over 65 million users, maintaining user satisfaction remains a challenge. This study employs sentiment analysis using a hybrid Convolutional Neural Network (CNN) and Bidirectional Long Short-Term Memory (BiLSTM) model to classify user review ratings. A dataset of 10,000 Google Play Store reviews was divided into COVID-19 and post-pandemic segments. The methodology includes data collection, pre-processing, and dataset segmentation for training, validation, and testing. Results indicate that the CNN-BiLSTM model surpasses traditional machine learning by combining CNN’s feature extraction with BiLSTM’s long-term dependency capture, achieving 98.71% accuracy on COVID-19 data and 98.16% post-pandemic. Additionally, the model demonstrates strong performance across other key evaluation metrics, with precision, recall, and F1-score. Misclassification analysis highlights minor errors, particularly in ratings 4 and 5. These findings help healthcare providers enhance digital services by identifying user concerns, improving platform features, and optimizing customer engagement. Beyond healthcare, this approach has real-world applications in e-commerce and financial services, where sentiment analysis informs user experience improvements.

References

[1] R. V. Silalahi, N. Hartono, and M. A. Tumpak, “Profile and preferences users of doctors consultation application in Indonesia,” in IOP Conference Series: Earth and Environmental Science, 2018. doi: 10.1088/1755-1315/195/1/012069.

[2] N. Hartono, L. Laurence, and T. O. Tedja, “Development initial model of intention to use Halodoc application using PLS-SEM,” Int. Conf. Informatics, Technol. Eng. 2019, no. August, pp. 63–70, 2019.

[3] O. Thinnukool, P. Khuwuthyakorn, P. Wientong, and T. Panityakul, “Non-prescription medicine mobile healthcare application: Smartphone-based software design and development review,” 2017. doi: 10.3991/ijim.v11i5.7123.

[4] A. J. Barton, “The regulation of mobile health applications,” 2012. doi: 10.1186/1741-7015-10-46.

[5] B. Martínez-Pérez, I. De La Torre-Díez, and M. López-Coronado, “Mobile health applications for the most prevalent conditions by the world health organization: Review and analysis,” 2013. doi: 10.2196/jmir.2600.

[6] H. Abaza and M. Marschollek, “mHealth application areas and technology combinations: A comparison of literature from high and low/middle income countries,” 2017. doi: 10.3414/ME17-05-0003.

[7] M. A. Kushendriawan, H. B. Santoso, P. O. H. Putra, and M. Schrepp, “Evaluating User Experience of a Mobile Health Application ‘Halodoc’ using User Experience Questionnaire and Usability Testing,” J. Sist. Inf., vol. 17, no. 1, pp. 58–71, 2021, doi: 10.21609/jsi.v17i1.1063.

[8] M. Christian, E. Retno Indriyarti, S. Sunarno, and S. Wibowo, “Determinants of Satisfaction Using Healthcare Application: A Study on Young Halodoc Users in Jakarta During the COVID-19 Pandemic,” vol. 2, no. 1, pp. 36–48, Aug. 2022, doi: 10.31098/quant.947.

[9] S. Chatterjee, “Explaining customer ratings and recommendations by combining qualitative and quantitative user generated contents,” Decis. Support Syst., vol. 119, pp. 14–22, 2019, doi: 10.1016/j.dss.2019.02.008.

[10] Q. A. Xu, V. Chang, and C. Jayne, “A systematic review of social media-based sentiment analysis: Emerging trends and challenges,” Decis. Anal. J., vol. 3, p. 100073, 2022, doi: 10.1016/j.dajour.2022.100073.

[11] J. Hirschberg and C. D. Manning, “Advances in natural language processing,” 2015. doi: 10.1126/science.aaa8685.

[12] J. Berger, A. Humphreys, S. Ludwig, W. W. Moe, O. Netzer, and D. A. Schweidel, “Uniting the Tribes: Using Text for Marketing Insight,” J. Mark., vol. 84, no. 1, pp. 1–25, 2020, doi: 10.1177/0022242919873106.

[13] J. Wang et al., “Global evidence of expressed sentiment alterations during the COVID-19 pandemic,” Nat. Hum. Behav., vol. 6, no. 3, pp. 349–358, 2022, doi: 10.1038/s41562-022-01312-y.

[14] J. Hartmann, M. Heitmann, C. Siebert, and C. Schamp, “More than a Feeling: Accuracy and Application of Sentiment Analysis,” Int. J. Res. Mark., vol. 40, no. 1, pp. 75–87, 2023, doi: 10.1016/j.ijresmar.2022.05.005.

[15] L. Alzubaidi et al., “Review of deep learning: concepts, CNN architectures, challenges, applications, future directions,” J. Big Data, vol. 8, no. 1, 2021, doi: 10.1186/s40537-021-00444-8.

[16] A. Ghosh, A. Sufian, F. Sultana, A. Chakrabarti, and D. De, “Fundamental concepts of convolutional neural network,” in Intelligent Systems Reference Library, vol. 172, 2019, pp. 519–567. doi: 10.1007/978-3-030-32644-9_36.

[17] M. Vineethmohan, P. Hemanth, M. Mounica, and P. Lakshmi Prasanna, “Image classification using deep learning,” J. Adv. Res. Dyn. Control Syst., vol. 12, no. 6, pp. 1–10, Mar. 2020, doi: 10.5373/JARDCS/V12I6/S20201001.

[18] H. Fan, M. Jiang, L. Xu, H. Zhu, J. Cheng, and J. Jiang, “Comparison of long short term memory networks and the hydrological model in runoff simulation,” Water (Switzerland), vol. 12, no. 1, 2020, doi: 10.3390/w12010175.

[19] S. M. Mousavi, M. Ghasemi, M. D. Manshadi, and A. Mosavi, “Deep learning for wave energy converter modeling using long short-term memory,” Mathematics, vol. 9, no. 8, 2021, doi: 10.3390/math9080871.

[20] L. Shan, Y. Liu, M. Tang, M. Yang, and X. Bai, “CNN-BiLSTM hybrid neural networks with attention mechanism for well log prediction,” J. Pet. Sci. Eng., vol. 205, no. December 2020, p. 108838, 2021, doi: 10.1016/j.petrol.2021.108838.

[21] N. M. Ali, M. M. A. El Hamid, and A. Youssif, “Sentiment Analysis for Movies Reviews Dataset Using Deep Learning Models,” Int. J. Data Min. Knowl. Manag. Process, vol. 09, no. 03, pp. 19–27, 2019, doi: 10.5121/ijdkp.2019.9302.

[22] P. K. Jain, V. Saravanan, and R. Pamula, “A Hybrid CNN-LSTM: A Deep Learning Approach for Consumer Sentiment Analysis Using Qualitative User-Generated Contents,” ACM Trans. Asian Low-Resource Lang. Inf. Process., vol. 20, no. 5, 2021, doi: 10.1145/3457206.

[23] M. Rhanoui, M. Mikram, S. Yousfi, and S. Barzali, “A CNN-BiLSTM Model for Document-Level Sentiment Analysis,” Mach. Learn. Knowl. Extr., vol. 1, no. 3, pp. 832–847, 2019, doi: 10.3390/make1030048.

[24] L. Xiaoyan, R. C. Raga, and S. Xuemei, “GloVe-CNN-BiLSTM Model for Sentiment Analysis on Text Reviews,” J. Sensors, vol. 2022, 2022, doi: 10.1155/2022/7212366.

[25] M. Ashraf et al., “A Hybrid CNN and RNN Variant Model for Music Classification,” Appl. Sci., vol. 13, no. 3, 2023, doi: 10.3390/app13031476.

[26] N. Singh, U. C. Jaiswal, and R. Singh, “Detecting Sarcasm Text in Sentiment Analysis Using Hybrid Machine Learning Approach,” Int. J. Intell. Syst. Appl., vol. 16, no. 4, pp. 72–85, 2024, doi: 10.5815/ijisa.2024.04.05.

[27] Y. A. Ramaziyah and E. B. Setiawan, “Hybrid Deep Learning CNN and BiLSTM with FastText as Feature Expansion for Sentiment Analysis in President Election 2024,” in COMNETSAT 2024 - IEEE International Conference on Communication, Networks and Satellite, 2024, pp. 176–183. doi: 10.1109/COMNETSAT63286.2024.10862946.

[28] S. Susandri, S. Defit, and M. Tajuddin, “Enhancing Text Sentiment Classification with Hybrid CNN-BiLSTM Model on WhatsApp Group,” J. Adv. Inf. Technol., vol. 15, no. 3, pp. 355–363, 2024, doi: 10.12720/jait.15.3.355-363.

[29] D. Jatnika, M. A. Bijaksana, and A. A. Suryani, “Word2vec model analysis for semantic similarities in English words,” in Procedia Computer Science, 2019, pp. 160–167. doi: 10.1016/j.procs.2019.08.153.

[30] I. Surjandari, C. Megawati, A. Dhini, and I. B. N. Sanditya Hardaya, “Application of text mining for classification of textual reports: A study of Indonesia’s national complaint handling system,” in Proceedings of the International Conference on Industrial Engineering and Operations Management, 2016, pp. 1147–1156.

[31] I. H. Sarker, “Machine Learning: Algorithms, Real-World Applications and Research Directions,” 2021. doi: 10.1007/s42979-021-00592-x.

[32] R. Mohammed, J. Rawashdeh, and M. Abdullah, “Machine Learning with Oversampling and Undersampling Techniques: Overview Study and Experimental Results,” in 2020 11th International Conference on Information and Communication Systems, ICICS 2020, 2020, pp. 243–248. doi: 10.1109/ICICS49469.2020.239556.

[33] S. Sharma, S. Sharma, and A. Athaiya, “Activation Functions in Neural Networks,” Int. J. Eng. Appl. Sci. Technol., vol. 04, no. 12, pp. 310–316, 2020, doi: 10.33564/ijeast.2020.v04i12.054.

[34] E. Elgeldawi, A. Sayed, A. R. Galal, and A. M. Zaki, “Hyperparameter tuning for machine learning algorithms used for arabic sentiment analysis,” Informatics, vol. 8, no. 4, 2021, doi: 10.3390/informatics8040079.

[35] J. Ha, M. Kambe, and J. Pe, Data Mining: Concepts and Techniques, 4th ed., no. August. 2011. doi: 10.1016/C2009-0-61819-5.

[36] D. Gao, X. Liu, Z. Zhu, and Q. Yang, “A hybrid CNN-BiLSTM approach for remaining useful life prediction of EVs lithium-Ion battery,” Meas. Control (United Kingdom), 2022, doi: 10.1177/00202940221103622.

[37] H. Yu, L. T. Yang, Q. Zhang, D. Armstrong, and M. J. Deen, “Convolutional neural networks for medical image analysis: State-of-the-art, comparisons, improvement and perspectives,” Neurocomputing, vol. 444, pp. 92–110, 2021, doi: 10.1016/j.neucom.2020.04.157.

[38] A. F. Hidayatullah, S. Cahyaningtyas, and R. D. Pamungkas, “Attention-based CNN-BiLSTM for Dialect Identification on Javanese Text,” Kinet. Game Technol. Inf. Syst. Comput. Network, Comput. Electron. Control, pp. 317–324, 2020, doi: 10.22219/kinetik.v5i4.1121.

[39] R. Bharal and O. V. V. Krishna, “Social Media Sentiment Analysis Using CNN-BiLSTM,” Int. J. Sci. Res., vol. 10, no. 9, pp. 656–661, 2020, doi: 10.21275/SR21913110537.

[40] D. H. N. Aini, D. Kurniasari, A. Nuryaman, and M. Usman, “Implementation of Artificial Neural Network With Backpropagation Algorithm for Rating Classification on Sales of Blackmores in Tokopedia,” J. Tek. Inform., vol. 4, no. 2, pp. 365–372, 2023, doi: 10.52436/1.jutif.2023.4.2.539.

[41] M. Hossin and M. . Sulaiman, “A Review on Evaluation Metrics for Data Classification Evaluations,” Int. J. Data Min. Knowl. Manag. Process, vol. 5, no. 2, pp. 01–11, 2015, doi: 10.5121/ijdkp.2015.5201.

[42] B. Bohara, R. I. Fernandez, V. Gollapudi, and X. Li, “Short-Term Aggregated Residential Load Forecasting using BiLSTM and CNN-BiLSTM,” 2022 Int. Conf. Innov. Intell. Informatics, Comput. Technol. 3ICT 2022, pp. 37–43, 2022, doi: 10.1109/3ICT56508.2022.9990696.