MangoVQA: A Multi-Answer Visual Question Answering System for Grading SNI-Aligned Mango Quality using Deep Learning and Text Processing
DOI:
https://doi.org/10.15408/aism.v9i2.53416Keywords:
Visual question answering, multi-answer, deep learning, rule-based reasoning, text processing, object detection, YOLOv11n, mango quality grading, SNI 3164:2024, interpretable machine learningAbstract
Visual Question Answering (VQA), a multimodal AI task combining computer vision and language understanding, answers text-based questions about visual content. Although VQA has developed rapidly, most existing approaches assume a single correct answer, whereas real-world images often require several valid answers simultaneously. This study proposes MangoVQA, to our knowledge the first multi-answer VQA framework for mango quality grading aligned with SNI 3164:2024. MangoVQA separates perception from reasoning, combining YOLOv11n-based neural perception with deterministic, rule-based question interpretation, unlike end-to-end learned-fusion VQA models. Each answer is grounded in detected mango instances, enabling traceable multi-answer responses rather than opaque predictions. The YOLOv11n detector localizes fruits and identifies four quality grades (Extra Class, Class I, Class II, and Reject), while the Perspective Engine extracts color and defect attributes through calibrated HSV analysis. These facts populate a structured answer representation, queried via predefined intents and slots. From 468 annotated images, the pipeline yielded 17,764 question-answer pairs across 22 question types, two reasoning levels, and two languages. 22.85% of questions required multi-answer responses, scored with set-based precision, recall, F1-score, and subset exact match. Evaluation reaches 0.919 accuracy on global open-ended questions, 0.937 on local open-ended questions, and 0.846 micro-F1 on global multi-answer questions, exceeding the 62.91% F1 reported by EMQA, the only prior multi-answer VQA baseline, on a different dataset. Practically, MangoVQA offers auditable decision support for smallholders, cooperatives, exporters needing SNI-compliant evidence, and mango customers; theoretically, decoupling visual perception from text interpretation and reasoning offers a viable, interpretable alternative to end-to-end multimodal fusion.
Downloads
References
[1] H. M. R. Iqbal and A. Hakim, “Classification and grading of harvested mangoes using convolutional neural network,” Int. J. Fruit Sci., vol. 22, no. 1, pp. 95–109, 2022, doi: 10.1080/15538362.2021.2023069.
[2] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2016-Decem, pp. 779–788, 2016, doi: 10.1109/CVPR.2016.91.
[3] R. Khanam and M. Hussain, “YOLOV11: An overview of the key architectural enhancements,” arXiv (Cornell University), Oct. 2024, doi: 10.48550/arxiv.2410.17725.
[4] N. Jegham, C. Y. Koh, M. Abdelatti, and A. Hendawi, “YOLO evolution: A comprehensive benchmark and architectural review of YOLOV12, YOLO11, and their previous versions,” arXiv.org, Oct. 31, 2024. https://arxiv.org/abs/2411.00201
[5] National Standardization Agency, “SNI 3164-2024: Mango (in Indonesian),” Jakarta, Indonesia, 2024. Accessed: May 18, 2026. [Online]. Available: https://akses-sni.bsn.go.id
[6] N. Hasanati, T. Djatna, I. S. Sitanggnag, and A. I. Suroso, “Evaluating Rare-Class Modeling in YOLOv11n for Imbalanced Agricultural Object Detection: A Mango Case Study,” 2026. https://github.com/nhasanati/Mango-YOLO
[7] J. S. Ignacio, K. N. A. Eisma, and M. V. C. Caya, “A YOLOv5-based deep learning model for in-situ detection and maturity grading of mango,” in 2022 6th International Conference on Communication and Information Systems (ICCIS), 2022, pp. 141–147. doi: 10.1109/ICCIS56375.2022.9998163.
[8] S. Arwatchananukul, S. Wongsawat, S. Chaiwong, M. Chen, and R. Saengrayap, “Automated physical feature extraction of namdokmai sithong mangoes using YOLOV8 and image processing techniques,” AgriEngineering, vol. 7, no. 9, Art. no. 312, Sep. 2025, doi: 10.3390/agriengineering7090312.
[9] A. Sopian, “System detection ripeness of fresh fruits bunch palm oil with YOLOv4,” Proc. Int. Conf. Ind. Eng. Oper. Manag. Istanbul, Turkey, no. Usda 2021, pp. 5307–5318, 2022, [Online]. Available: https://index.ieomsociety.org/index.cfm/item/41183
[10] J. W. Lai, H. R. Ramli, L. I. Ismail, W. Zuha, and W. Hasan, “Real-time detection of ripe oil palm fresh fruit bunch based on YOLOv4,” IEEE Access, vol. 10, no. September, pp. 95763–95770, 2022, doi: 10.1109/ACCESS.2022.3204762.
[11] H. Wibowo, I. S. Sitanggang, M. Mushthofa, and H. A. Adrianto, “Large-scale oil palm trees detection from high-resolution remote sensing images using deep learning,” Big Data Cogn. Comput., vol. 6, no. 3, Art. no. 89, 2022, doi: 10.3390/bdcc6030089.
[12] Q. Wu, D. Teney, P. Wang, C. Shen, A. Dick, and A. van den Hengel, “Visual question answering: A survey of methods and datasets,” Comput. Vis. Image Underst., vol. 163, pp. 21–40, 2017, doi: 10.1016/j.cviu.2017.05.001.
[13] S. Antol et al., “VQA: Visual question answering,” Proceedings of the IEEE International Conference on Computer Vision, vol. 2015 Inter. pp. 2425–2433, 2015. doi: 10.1109/ICCV.2015.279.
[14] Y. Goyal, T. Khot, A. Agrawal, D. Summers-Stay, D. Batra, and D. Parikh, “Making the V in VQA matter: Elevating the role of image understanding in visual question answering,” Int. J. Comput. Vis., vol. 127, no. 4, pp. 398–414, 2019, doi: 10.1007/s11263-018-1116-0.
[15] Q. Wu, P. Wang, and X. Wang, Visual Question Answering From Theory to Application. Springer, 2022.
[16] S. Barra, C. Bisogni, M. De Marsico, and S. Ricciardi, “Visual question answering: Which investigated applications?,” Pattern Recognit. Lett., vol. 151, pp. 325–331, 2021, doi: 10.1016/j.patrec.2021.09.008.
[17] S. H. Hosseinabad, M. Safayani, and A. Mirzaei, “Multiple answers to a question: a new approach for visual question answering,” Vis. Comput., vol. 37, no. 1, pp. 119–131, 2021, doi: 10.1007/s00371-019-01786-4.
[18] M. Luo et al., “‘Just because you are right, doesn’t mean I am wrong’: Overcoming a bottleneck in the development and evaluation of open-ended visual question answering (VQA) tasks,” EACL 2021 - 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. pp. 2766–2771, 2021. doi: 10.18653/v1/2021.eacl-main.240.
[19] N. Hasanati, T. Djatna, I. S. Sitanggang, and A. I. Suroso, “Exploring multi-answer visual question answering with object detection: a systematic review,” Int. J. Informatics Commun. Technol., vol. 15, no. 3, pp. 1097–1114, 2026, doi: 10.11591/ijict.v15i3.pp1097-1114.
[20] S. Harnad, “The Symbol Grounding Problem,” Phys. D Nonlinear Phenom., vol. 42, pp. 335–346, 1990, doi: 10.1016/0167-2789(90)90087-6.
[21] J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu, “The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision,” 7th Int. Conf. Learn. Represent. ICLR 2019, pp. 1–28, 2019, [Online]. Available: https://api.elsevier.com/content/abstract/scopus_id/85083954234
[22] A. d’Avila Garcez and L. C. Lamb, “Neurosymbolic AI: The 3rd wave,” Artif. Intell. Rev., vol. 56, no. 11, pp. 12387–12406, 2023, doi: 10.1007/s10462-023-10448-w.
[23] K. Yi, A. Torralba, J. Wu, P. Kohli, C. Gan, and J. B. Tenenbaum, “Neural-symbolic VQA: Disentangling reasoning from vision and language understanding,” Advances in Neural Information Processing Systems, vol. 2018-Decem. pp. 1031–1042, 2018. [Online]. Available: https://api.elsevier.com/content/abstract/scopus_id/85064824191
[24] A. M. Kiloes and D. Joyce, “Exploring the challenges and opportunities of mango export from Indonesia: Insights from stakeholder interviews,” Qual. Report-NSUWorks, vol. 29, no. 3, pp. 811–830, 2024, doi: 10.46743/2160-3715/2024.6343 This.
[25] National Standardization Agency, “SNI 3164:2009: Mango (in Indonesian),” Jakarta, Indonesia, 2009. [Online]. Available: https://akses-sni.bsn.go.id
[26] ASEAN, “ASEAN STANDARD FOR MANGO (ASEAN Stan 2:2006, Rev.1-2012),” 2012. [Online]. Available: https://www.scribd.com/document/421547432/MANGO-Asean-Standard
[27] FAO/WHO Codex Alimentarius Commission, “Codex Standard for Mangoes (CODEX STAN 184-1993, Amended 2005),” FAO/WHO, Rome, Italy, 1993. [Online]. Available: https://www.fao.org/input/download/standards/315/315/CXS_184e.pdf
[28] K. Kafle and C. Kanan, “Visual question answering: Datasets, algorithms, and future challenges,” Comput. Vis. Image Underst., vol. 163, pp. 3–20, 2017, doi: 10.1016/j.cviu.2017.06.005.
[29] K. Bayoudh, R. Knani, F. Hamdaoui, and A. Mtibaa, “A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets,” Vis. Comput., vol. 38, no. 8, pp. 2939–2970, 2022, doi: 10.1007/s00371-021-02166-7.
[30] M. T. Desta, L. Chen, and T. Kornuta, “Object-based reasoning in VQA,” Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018, vol. 2018-Janua. pp. 1814–1823, 2018. doi: 10.1109/WACV.2018.00201.
[31] M. Malinowski and M. Fritz, “A multi-world approach to question answering about real-world scenes based on uncertain input,” Adv. Neural Inf. Process. Syst., vol. 2, no. January, pp. 1682–1690, 2014.
[32] S. Upadhyay and S. S. Tripathy, “Enhancing VQA with emphasis-based soft attention,” Signal, Image Video Process., vol. 19.0, no. 13, 2025, doi: 10.1007/s11760-025-04710-7.
[33] Y. Miao, W. Cheng, S. He, and H. Jiang, “Research on visual question answering based on GAT relational reasoning,” Neural Process. Lett., vol. 54, no. 2, pp. 1435–1448, 2022, doi: 10.1007/s11063-021-10689-2.
[34] H. Liu et al., “Multi-granularity feature interaction and multi-region selection based triplet visual question answering,” IEEE Trans. Big Data, 2024, doi: 10.1109/TBDATA.2024.3453750.
[35] H. Liu, B. Wang, X. Li, Y. Sun, Y. Hu, and B. Yin, “ICQ-TransE: LLM-enhanced image-caption-question translating embeddings for knowledge-based visual question answering,” IEEE Trans. Artif. Intell., 2025, doi: 10.1109/TAI.2025.3575553.
[36] Z. Liu, K. Yang, Y. Weng, Z. He, X. Liu, and H. Gao, “SCAG: Semantic co-occurring attention guided alignment for knowledge-based visual question answering,” ACM Trans. Multimed. Comput. Commun. Appl., vol. 21.0, no. 7, 2025, doi: 10.1145/3734220.
[37] Y. Yuan, J. Zeng, and S. Shiguang, “Exp-VQA: Fine-grained facial expression analysis via visual question answering,” Pattern Recognit., vol. 168.0, 2025, doi: 10.1016/j.patcog.2025.111783.
[38] J. Zhang, B. Li, and S. Zhou, “Hierarchical modeling for medical visual question answering with cross-attention fusion,” Appl. Sci., vol. 15, no. 9, 2025, doi: 10.3390/app15094712.
[39] Z. Liu, D. Zhao, B. Yuan, and Z. Jiang, “RescueADI: Adaptive disaster interpretation in remote sensing images with autonomous agents,” IEEE Trans. Geosci. Remote Sens., vol. 63, 2025, doi: 10.1109/TGRS.2025.3532594.
[40] B. Liu, L. M. Zhan, L. Xu, L. Ma, Y. Yang, and X. M. Wu, “Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering,” Proceedings - International Symposium on Biomedical Imaging, vol. 2021-April. pp. 1650–1654, 2021. doi: 10.1109/ISBI48211.2021.9434010.
[41] H. Demirhan and W. Zadrozny, “Survey of multimodal medical question answering,” BioMedInformatics, vol. 4, no. 1, pp. 50–74, 2024, doi: 10.3390/biomedinformatics4010004.
[42] Z. Li et al., “HortiVQA-PP: Multitask framework for pest segmentation and visual question answering in horticulture,” Horticulturae, vol. 11, no. 9, Art. no. 1009, Aug. 2025, doi: 10.3390/horticulturae11091009.
[43] P. Anderson et al., “Bottom-up and top-down attention for image captioning and visual question answering,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 6077–6086, 2018. doi: 10.1109/CVPR.2018.00636.
[44] S. Shahane, “Mango Varieties Classification and Grading,” 2021. https://www.kaggle.com/datasets/saurabhshahane/mango-varieties-classification
[45] K. Ratprakhon, W. Neubauer, K. Riehn, J. Fritsche, and S. Rohn, “Developing an automatic color determination (mangifera indica) using a CCD camera and color standards,” Foods, vol. 9, no. 11, Art, no. 1709, 2020, doi: 10.3390/foods9111709.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Nidaul Hasanati, Taufik Djatna, Imas Sukaesih Sitanggang , Arif Imam Suroso
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
