Interpretable Machine Learning for Narrative Framing Classification: A TF-IDF-Based Study of Frankenstein

Luthfiana Dwi Rakhmawati; Rusdi Aulia  Romadhon; Kiki Adnan  Muzaki; Anna  Nurawalia; Dewi Aprilia Ningrum

doi:10.15408/jti.v19i2.54477

Authors

Luthfiana Dwi Rakhmawati English Literature, Faculty of Adab and Humanities, UIN Syarif Hidayatullah Jakarta, Indonesia
Rusdi Aulia Romadhon Informatics Engineering, Faculty of Science and Technology, UIN Syarif Hidayatullah Jakarta, Indonesia
Kiki Adnan Muzaki Religious Studies Department, University of California, Riverside, USA
Anna Nurawalia English Language Department, Politeknik Bau-Bau, Indonesia
Dewi Aprilia Ningrum Centre for Planners' Development, Education, and Training (Pusbindiklatren), Ministry of National Development Planning/Bappenas

DOI:

https://doi.org/10.15408/jti.v19i2.54477

Keywords:

Frankenstein, text classification, TF-IDF, narrative framing, computational literary studies, machine learning

Abstract

This study evaluates interpretable supervised machine learning for three-class narrative framing classification of literary text, using Mary Shelley's Frankenstein (1818) as a case study; narrative framing is the evaluative position a passage assigns to the Creature: Threat, Victim, or Ambivalent. The methodology comprised five stages: (1) extraction of passages narrated by Victor Frankenstein, the Creature, and Robert Walton from the 1818 Project Gutenberg text; (2) annotation of 195 passages, with a blinded second annotator re-labeling 68 passages (Cohen's κ = 0.6926); (3) sublinear TF-IDF unigram and bigram representation; (4) classification with Multinomial Naive Bayes, Linear SVM, and Logistic Regression against Majority and VADER baselines under stratified 5-fold cross-validation; and (5) a designation-masking ablation and coefficient analysis. The results show that Logistic Regression obtained the highest mean scores (66.67% accuracy; 47.75% Macro-F1), although its margin over Linear SVM (46.18%) and VADER (44.85%) lies within fold-level variability. Pooled F1 reached 69.0% for Threat and 74.3% for Victim, whereas none of the 29 Ambivalent passages was correctly identified. Masking explicit designations reduced Macro-F1 only to 45.20%, indicating partial robustness to this procedure rather than discourse-level modeling. It is concluded that interpretable classical classifiers can separate Threat from Victim framing using transparent lexical evidence but cannot yet detect ambiguous framing; given the single-novel, small, and imbalanced dataset, the findings are a preliminary proof of concept. Future work should enlarge and balance the corpus, apply group-based and repeated cross-validation, and compare context-aware models, including large language models, on the Ambivalent class.

References

[1] M. W. Shelley, Frankenstein; or, The Modern Prometheus. London, U.K.: Lackington, Hughes, Harding, Mavor, & Jones, 1818.

[2] A. K. Mellor, Mary Shelley: Her Life, Her Fiction, Her Monsters. New York, NY, USA: Routledge, 1988.

[3] A. Piper, R. J. So, and D. Bamman, "Narrative theory for computational narrative understanding," in Proc. 2021 Conf. Empirical Methods in Natural Language Processing (EMNLP), Punta Cana, Dominican Republic, 2021, pp. 298–311, doi: 10.18653/v1/2021.emnlp-main.26.

[4] H. O. Hatzel, H. Stiemer, C. Biemann, and E. Gius, "Machine learning in computational literary studies," it - Information Technology, vol. 65, no. 4–5, pp. 200–217, 2023.

[5] A. M. Weimer, F. Barth, T. Dönicke, L. Gödeke, H. Varachkina, A. Holler, C. Sporleder, and B. Gittel, "The (in-)consistency of literary concepts. Operationalising, annotating and detecting literary comment," Journal of Computational Literary Studies, vol. 1, no. 1, 2022, doi: 10.48694/jcls.90.

[6] F. Brahman, M. Huang, O. Tafjord, C. Zhao, M. Sachan, and S. Chaturvedi, "'Let your characters tell their story': A dataset for character-centric narrative understanding," in Findings of the Association for Computational Linguistics: EMNLP 2021, Punta Cana, Dominican Republic, 2021, pp. 1734–1752.

[7] B. Yang, H. Xu, J. Du, Z. Li, Y. He, and C. Lin, "EvolvTrip: Enhancing literary character understanding with temporal theory-of-mind graphs," arXiv preprint arXiv:2506.13641, 2025.

[8] A. Brei, K. Henry, A. Sharma, S. Srivastava, and S. Chaturvedi, "Classifying unreliable narrators with large language models," in Proc. 63rd Annual Meeting of the Association for Computational Linguistics (ACL), Vienna, Austria, 2025, pp. 20766–20791, doi: 10.18653/v1/2025.acl-long.1013.

[9] P. Toro Isaza, G. Xu, T. Oloko, Y. Hou, N. Peng, and D. Wang, "Are fairy tales fair? Analyzing gender bias in temporal narrative event chains of children's fairy tales," in Proc. 61st Annual Meeting of the Association for Computational Linguistics (ACL), Toronto, Canada, 2023, pp. 6509–6531, doi: 10.18653/v1/2023.acl-long.359.

[10] S. Rebora, "Sentiment analysis in literary studies. A critical survey," Digital Humanities Quarterly, vol. 17, no. 2, 2023.

[11] E. Kim and R. Klinger, "A survey on sentiment and emotion analysis for computational literary studies," Zeitschrift für digitale Geisteswissenschaften, vol. 4, 2019, doi: 10.17175/2019_008_v2.

[12] Z. C. Lipton, "The mythos of model interpretability," Communications of the ACM, vol. 61, no. 10, pp. 36–43, 2018, doi: 10.1145/3233231.

[13] L. Frermann, J. Li, S. Khanehzar, and G. Mikolajczak, "Conflicts, villains, resolutions: Towards models of narrative media framing," in Proc. 61st Annual Meeting of the Association for Computational Linguistics (ACL), Toronto, Canada, 2023, pp. 8712–8732.

[14] C. Molnar, Interpretable Machine Learning: A Guide for Making Black Box Models Explainable, 2nd ed., 2022. [Online]. Available: https://christophm.github.io/interpretable-ml-book/

[15] R. Artstein and M. Poesio, "Inter-coder agreement for computational linguistics," Computational Linguistics, vol. 34, no. 4, pp. 555–596, 2008, doi: 10.1162/coli.07-034-R2.

[16] F. W. S. Shakti, E. Purwanto, and A. I. Pradana, "Klasifikasi surat dinas berbasis text mining menggunakan TF-IDF dan algoritma Naive Bayes Classifier," Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), vol. 7, no. 5, pp. 1152–1158, 2023.

[17] D. Y. Saraswati, M. R. Handayani, K. Umam, and M. I. Mustofa, "Sentiment classification of MyPertamina reviews using TF-IDF and support vector machine algorithm," Jurnal Teknik Informatika (JUTIF), vol. 5, no. 1, pp. 177–184, 2024.

[18] R. B. Adinata, Supriyono, and D. L. Fithri, "Sentiment classification of MyTelkomsel reviews using SVM and logistic regression," Jurnal Infotel, vol. 15, no. 4, pp. 312–320, 2023.

[19] A. S. Fitrani and N. Hidayat, "Klasifikasi teks opini publik terhadap kebijakan menggunakan TF-IDF dan multinomial Naive Bayes," Jurnal SISFO (Sistem Informasi), vol. 12, no. 2, pp. 85–94, 2023.

[20] M. A. Fauzi and A. P. Kharisma, "Penerapan feature selection dan SVM pada analisis teks naratif Bahasa Indonesia," Jurnal Teknologi Informasi dan Ilmu Komputer (JTIIK), vol. 10, no. 4, pp. 789–796, 2023.

[21] J. Cohen, "A coefficient of agreement for nominal scales," Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, 1960.

[22] J. R. Landis and G. G. Koch, "The measurement of observer agreement for categorical data," Biometrics, vol. 33, no. 1, pp. 159–174, 1977.

[23] G. Salton and C. Buckley, "Term-weighting approaches in automatic text retrieval," Information Processing & Management, vol. 24, no. 5, pp. 513–523, 1988.

[24] C. J. Hutto and E. Gilbert, "VADER: A parsimonious rule-based model for sentiment analysis of social media text," in Proc. 8th International AAAI Conf. Weblogs and Social Media (ICWSM), Ann Arbor, MI, USA, 2014, pp. 216–225.

[25] F. Pedregosa et al., "Scikit-learn: Machine learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.

[26] D. Jurafsky and J. H. Martin, Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 3rd ed. draft, 2024. [Online]. Available: https://web.stanford.edu/~jurafsky/slp3/

[27] M. T. Ribeiro, S. Singh, and C. Guestrin, "'Why should I trust you?': Explaining the predictions of any classifier," in Proc. 22nd ACM SIGKDD International Conf. Knowledge Discovery and Data Mining (KDD), San Francisco, CA, USA, 2016, pp. 1135–1144.

[28] S. M. Mohammad, "Sentiment analysis: Automatically detecting valence, emotions, and other affectual states from text," in Emotion Measurement, 2nd ed., H. L. Meiselman, Ed. Elsevier, 2021, pp. 323–379, doi: 10.1016/B978-0-12-821124-3.00011-9.

[29] F. Sebastiani, "Machine learning in automated text categorization," ACM Computing Surveys, vol. 34, no. 1, pp. 1–47, 2002, doi: 10.1145/505282.505283.

[30] D. M. W. Powers, "Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation," Journal of Machine Learning Technologies, vol. 2, no. 1, pp. 37–63, 2011.

[31] E. Gius and J. Jacke, "The hermeneutic profit of annotation: On preventing and fostering disagreement in literary analysis," International Journal of Humanities and Arts Computing, vol. 11, no. 2, pp. 233–254, 2017, doi: 10.3366/ijhac.2017.0194.

[32] K. Spärck Jones, "A statistical interpretation of term specificity and its application in retrieval," Journal of Documentation, vol. 60, no. 5, pp. 493–506, 2004.

[33] G. Genette, Narrative Discourse: An Essay in Method, J. E. Lewin, Trans. Ithaca, NY, USA: Cornell Univ. Press, 1980.

[34] P. Brooks, "What is a monster? (According to Frankenstein)," in Body Work: Objects of Desire in Modern Narrative. Cambridge, MA, USA: Harvard Univ. Press, 1993, pp. 199–220.

[35] C. D. Manning, P. Raghavan, and H. Schütze, Introduction to Information Retrieval. Cambridge, U.K.: Cambridge Univ. Press, 2008.

[36] C. Nadeau and Y. Bengio, "Inference for the generalization error," Machine Learning, vol. 52, no. 3, pp. 239–281, 2003.

[37] T. Underwood, Distant Horizons: Digital Evidence and Literary Change. Chicago, IL, USA: Univ. of Chicago Press, 2019.