Validation of a High School Physics Scientific Literacy Test Using Classical Test Theoryand Item Response Theory
DOI:
https://doi.org/10.15408/jp3i.v15i2.45164Keywords:
classical test theory (CTT), high school physics (HSP), item response theory (IRT), scientific literacy test (SLT), validityAbstract
Physics scientific literacy instruments at the senior high school level are commonly validated using only Classical Test Theory
(CTT), resulting in limited psychometric evidence. This study aimed to validate the high school physics scientific literacy test
(HSP-SLT) using CTT and the Item Response Theory (IRT) framework. The study involved 513 Grade XI and XII students
from West Sulawesi Province, Indonesia. The initial instrument consisted of 25 multiple-choice items developed based on the
three PISA scientific literacy competencies. CTT analysis was conducted to evaluate item difficulty, discrimination, and internal
consistency reliability. Subsequently, IRT analysis using the two-parameter logistic (2-PL) model was performed to estimate
item discrimination and difficulty parameters and to examine the test information function to obtain a final instrument with
more optimal measurement characteristics. The internal consistency reliability yielded a Cronbach’s alpha of 0.50 for the total
scale. Item difficulty indices ranged from 0.138 to 0.390, and most items had discrimination indices ≥ 0.20. The initial IRT
analysis identified five items with unstable parameter estimates or negative discrimination, which were removed. The final model
included 20 items with positive discrimination parameters (0.113-1.027) and more stable difficulty estimates (0.611-6.580). The
model fit indices indicated an acceptable fit (RMSEA = 0.019, SRMR = 0.045, CFI = 0.944, TLI = 0.944). The test information
function showed that measurement precision was highest at ability levels between θ ≈1 and 2. Integrating CTT and IRT
produced a more stable and precise measurement model, particularly for students with moderate to high levels of scientific
literacy. The HSP-SLT therefore demonstrates potential as a validated instrument for assessing physics scientific literacy at the
senior high school level in Indonesia.
References
Adams, W. K., & Wieman, C. E. (2011). Development and validation of instruments to measure learning
of expert-like thinking. International Journal of Science Education, 33(9), 1289–1312.
https://doi.org/10.1080/09500693.2010.512369
Afnan, R., Munasir, M., Budiyanto, M., & Aulia, M. I. R. (2023). The Role of Scientific Literacy
Instruments For Measuring Science Problem Solving Ability. IJORER : International Journal of Recent
Educational Research, 4(1), 45–58. https://doi.org/10.46245/ijorer.v4i1.271
Ahorsu, D. K., Lin, C.-Y., Imani, V., Saffari, M., Griffiths, M. D., & Pakpour, A. H. (2022). The Fear
of COVID-19 Scale: Development and Initial Validation. International Journal of Mental Health and
Addiction, 20(3), 1537–1545. https://doi.org/10.1007/s11469-020-00270-8
Algiranto, A. (2022). Pengembang Instrumen Tes Berbasis Literasi Sains Materi Fluida Statis Kelas XI
SMA. Silampari Jurnal Pendidikan Ilmu Fisika, 4(1), 47–58.
Arohman, M., Saefudin, & Priyandoko, D. (2016). Kemampuan Literasi Sains Pada Pembelajaran
Ekosistem. Jurnal FMIPA Universitas Pendidikan Indonesia ISSN : 2528-5742, 13(1), 90–92.
Ayanwale, M. A., Amusa, J. O., Oladejo, A. I., & Ayedun, F. (2024). Multidimensional item Response
Theory Calibration of Dichotomous Response Structure Using R Language for Statistical
Computing. Interchange, 55(1), 137–157. https://doi.org/10.1007/s10780-024-09517-y
Baker, F. B. (2001). The Basics of Item Response Theory (C. Boston & L. Rudner (eds.); Second Edi). ERIC
Clearinghouse on Assessment and Evaluation.
Bichi, A. A., & Talib, R. (2018). Item Response Theory: An Introduction to Latent Trait Models to Testand Item Development. International Journal of Evaluation and Research in Education (IJERE), 7(2),
142. https://doi.org/10.11591/ijere.v7i2.12900
Bichi, A., Talib, R., Atan, A., Ibrahim, H., & Yusof, S. (2019). Validation of a developed university
placement test using classical test theory and Rasch measurement approach. International Journal of
ADVANCED AND APPLIED SCIENCES, 6, 22–29. https://doi.org/10.21833/ijaas.2019.06.004
Bijlsma, H., van der Lans, R., Mainhard, T., & den Brok, P. (2021). Psychometric Perspectives: CCT,
IRT. Student Feedback on Teaching in Schools: Using Student Perceptions for the Development of Teaching
and Teachers.
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best
Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A
Primer. Frontiers in Public Health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149
Bock, R. D., & Gibbons, R. D. (2021). Item Response Theory. In The SAGE Handbook of Quantitative
Methods in Psychology. John Wiley & Sons, Inc. https://doi.org/10.4135/9780857020994.n7
Cappelleri, J. C., Jason Lundy, J., & Hays, R. D. (2014). Overview of classical test theory and item
response theory for the quantitative assessment of items in developing patient-reported outcomes
measures. Clinical Therapeutics, 36(5), 648–662. https://doi.org/10.1016/j.clinthera.2014.04.006
Chalmers, R. P. (2012). Mirt: A multidimensional item response theory package for the R environment.
Journal of Statistical Software, 48(6). https://doi.org/10.18637/jss.v048.i06
Christensen, K. B., Makransky, G., & Horton, M. (2017). Critical Values for Yen’s Q(3): Identification
of Local Dependence in the Rasch Model Using Residual Correlations. Applied Psychological
Measurement, 41(3), 178–194. https://doi.org/10.1177/0146621616677520
Coppi, M., Fialho, I., & Cid, M. (2023). Developing a Scientific Literacy Assessment Instrument for
Portuguese 3rd Cycle Students. Education Sciences, 13(9). https://doi.org/10.3390/educsci13090941
DeMars, C. (2010). Item Response Theory. Oxford University Press.
https://doi.org/10.1093/acprof:oso/9780195377033.001.0001
DeMars, C. E. (2018). Classical Test Theory and Item Response Theory. In The Wiley Handbook of
Psychometric Testing (pp. 49–73). https://doi.org/https://doi.org/10.1002/9781118489772.ch2
Denieffe, S. (2020). Commentary: Purposive sampling: complex or simple? Research case examples.
Journal of Research in Nursing : JRN, 25(8), 662–663. https://doi.org/10.1177/1744987120928156
Desjardins, C. D., & Bulut, O. (2018). Handbook of educational measurement and psychometrics using R. CRC
Press.
Development, O. for E. C. and. (2023). PISA 2022 assessment and analytical framework. OECD publishing.
Donhauser, T., Apfelbacher, C., Kann, G., Masur, C., Kamudoni, P., Salek, S., Abels, C., & Gabes, M.
(2023). Hyperhidrosis quality of life index (HidroQoL): further validation by applying classical test
theory and item response theory using data from a phase III clinical trial. Journal of Patient-Reported
Outcomes, 7(1), 55. https://doi.org/10.1186/s41687-023-00596-6
Doran, R. L. (1980). Basic Measurement and Evaluation of Science Instruction. National Science Teachers
Association.
Eaton, P., Johnson, K., Frank, B., & Willoughby, S. (2019). Classical test theory and item response
theory comparison of the brief electricity and magnetism assessment and the conceptual survey of
electricity and magnetism. Physical Review Physics Education Research, 15(1), 10102.
https://doi.org/10.1103/PhysRevPhysEducRes.15.010102
El-Hamamsy, Laila, Zapata-Cáceres, María, Barroso, Estefanía Martín, Mondada, Francesco,
Zufferey, Jessica Dehler, & Bruno, Barbara. (2022). The Competent Computational Thinking Test: Development and Validation of an Unplugged Computational Thinking Test for Upper Primary
School. Journal of Educational Computing Research, 60(7), 1818–1866.
https://doi.org/10.1177/07356331221081753
Embretson, S. E., & Reise, S. P. (2013). Item response theory for psychologists. Psychology Press.
Finch, W. H., & Jeffers, H. (2016). A Q(3)-Based Permutation Test for Assessing Local Independence.
Applied Psychological Measurement, 40(2), 157–160. https://doi.org/10.1177/0146621615622635
Fives, H., Huebner, W., Birnbaum, A. S., & Nicolich, M. (2014). Developing a Measure of Scientific
Literacy for Middle School Students. Science Education, 98(4), 549–580.
https://doi.org/10.1002/sce.21115
Goldmann, P., Schäfer, B., & Altendorfer, C. (2025). Development and validation of the digital mindset
scale. Journal of Business Research, 199, 115554.
https://doi.org/https://doi.org/10.1016/j.jbusres.2025.115554
Gormally, C., Brickman, P., & Lut, M. (2012). Developing a test of scientific literacy skills (TOSLS):
Measuring undergraduates’ evaluation of scientific information and arguments. CBE Life Sciences
Education, 11(4), 364–377. https://doi.org/10.1187/cbe.12-03-0026
Hartono, W., Hadi, S., & Rosnawati, R. (2024). Optimizing Item Construction in Diagnostic
Mathematics Test. TEM Journal, 1106–1115. https://doi.org/10.18421/TEM132-25
Hays, R. D., Morales, L. S., & Reise, S. P. (2000). Item response theory and health outcomes
measurement in the 21st century. Medical Care, 38(9 Suppl), II28-42.
https://doi.org/10.1097/00005650-200009002-00007
Huang, F., Ye Han, X., Chen, S.-L., Guo, Y. F., Wang, A., & Zhang, Q. (2020). Psychometric Testing
of the Chinese Simple Version of the Simulation Learning Effectiveness Inventory: Classical Theory
Test and Item Response Theory. Frontiers in Psychology, Volume 11-2020.
https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2020.00032
Indrawati, M. D., & Sunarti, T. (2018). Pengembangan Instrumen Penilaian Literasi Sains Fisika Peserta
Didik pada Bahasan Gelombang Bunyi di SMA Negeri 1 Gedangan Sidoarjo. Jurnal Inovasi
Pendidikan Fisika (JIPF), 07(01), 14–20.
Istiyono, E. (2020). Pengembangan instrumen penilaian dan analisis hasil belajar fisika dengan teori tes
klasik dan modern. Yogyakarta: UNY Press. L, I.(2019). Evaluasi Dalam Proses Pembelajaran. Jurnal
Manajemen Pendidikan Islam, 9, 478–492.
Istyadji, M., & Sauqina. (2023). Conception of scientific literacy in the development of scientific literacy
assessment tools: a systematic theoretical review. Journal of Turkish Science Education, 20(2), 281–
308. https://doi.org/10.36681/tused.2023.016
Kalkbrenner, M. T. (2021). A Practical Guide to Instrument Development and Score Validation in the
Social Sciences: The MEASURE Approach. Practical Assessment, Research & Evaluation, 26, 1.
Karimah, K., Jufri, A. W., & Ramdani, A. (2021). Improving Students’ Scientific Literacy: Learning
Studies Using Science Teaching Materials on Circulatory System Materials. Prisma Sains : Jurnal
Pengkajian Ilmu Dan Pembelajaran Matematika Dan IPA IKIP Mataram, 9(1 SE-Research Articles), 57–
63. https://doi.org/10.33394/j-ps.v9i1.3878
Kong, S.-C., & Lai, M. (2022). Validating a computational thinking concepts test for primary education
using item response theory: An analysis of students’ responses. Computers & Education, 187, 104562.
https://doi.org/https://doi.org/10.1016/j.compedu.2022.104562
Ly, C. Y., Rakkapao, S., Nualtong, K., & Sumathakulawat, W. (2021). Classical test theory and rasch
analysis of test of understanding of vectors (TUV). Journal of Physics: Conference Series, 1719(1). https://doi.org/10.1088/1742-6596/1719/1/012089
McNeish, D. (2017). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412–
433. https://doi.org/10.1037/met0000144
Noor, M. S. A. M. (2021). Assessing Secondary Students’ Scientific Literacy: A Comparative Study of
Suburban Schools in England and Malaysia. Science Education International, 32(4), 343–352.
https://doi.org/10.33828/sei.v32.i4.9
Nurjanah, S., Iqbal, M., Zafrullah, Z., Mahmud, M. N., Seran, D. S. F., Suardi, I. K., & Arriza, L.
(2024). Psychometric quality of multiple-choice tests under classical test theory (CTT): AnBuso,
Iteman, and R. Jurnal Penelitian Dan Evaluasi Pendidikan, 28(2 SE-Articles), 161–172.
https://doi.org/10.21831/pep.v28i2.71542
Nursyamsudin, N., & Jaelani, M. N. G. (2021). Penguatan literasi dan numerasi di SMA. Direktorat Sekolah
Menengah Atas.
OECD. (2015). PISA 2015 Draft Science Framework. In Oecd (Issue March 2015).
https://doi.org/10.1177/0022146512469014
OECD. (2019). PISA 2018 Assessment and Analytical Framework, PISA. In OECD Publishing.
OECD. (2023). PISA 2022 Results (Volume I): The State of Learning and Equity in Education. In OECD
Publishing (Vol. 46, Issue 183). https://doi.org/10.1787/53f23881-en
Oktalia, Y., Susila, A. B., & Iswanto, B. H. (2021). Virtual Test Instruments to Measure Scientific
Literacy of High School Students on Work and Energy. Journal of Physics: Conference Series, 2019(1),
12009. https://doi.org/10.1088/1742-6596/2019/1/012009
Petrillo, J., Cano, S. J., McLeod, L. D., & Coon, C. D. (2015). Using classical test theory, item response
theory, and Rasch measurement theory to evaluate patient-reported outcome measures: a
comparison of worked examples. Value in Health : The Journal of the International Society for
Pharmacoeconomics and Outcomes Research, 18(1), 25–34. https://doi.org/10.1016/j.jval.2014.10.005
Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what’s being
reported? critique and recommendations. Research in Nursing & Health, 29(5), 489–497.
https://doi.org/https://doi.org/10.1002/nur.20147
Qian, Z., Yang, Y., Tan, J., Li, Y., Zhou, J., Huang, J., Assanangkornchai, S., & Chen, J. (2025).
Psychometric Evaluation of the Chinese Version of the Mental Health System Responsiveness
Questionnaire for Psychiatric Outpatients: Classical Test Theory and Item Response Theory
Approaches. Patient Preference and Adherence, 19, 729–740. https://doi.org/10.2147/PPA.S503016
Raykov, T., Dimitrov, D. M., Marcoulides, G. A., & Harrison, M. (2019). On the Connections Between
Item Response Theory and Classical Test Theory: A Note on True Score Evaluation for Polytomous
Items via Item Response Modeling. Educational and Psychological Measurement, 79(6), 1198–1209.
https://doi.org/10.1177/0013164417745949
Revelle, W. (2025). Package ‘ psych ’: Procedures for Psychological, Psychometric, and Personality Research
Description (2.5.3). https://personality-project.org/r/psych/
Rusilowati, A., Kurniawati, L., Nugroho, S. E., & Widiyatmoko, A. (2016). Developing an instrument
of scientific literacy asessment on the cycle theme. International Journal of Environmental and Science
Education, 11(12), 5718–5727.
Santoso, P. H., Istiyono, E., Haryanto, & Retnawati, H. (2024). Validating light phenomena conceptual
assessment through the lens of classical test theory and item response theory frameworks. Physics
Education, 59(2), 1–19. https://doi.org/10.1088/1361-6552/ad183b
Sartika, D., Lutfin, N. A., Musdar, M., & Nurlina, N. (2024). Kemampuan Calon Guru Fisika Dalam Menyelesaikan Soal Literasi Sains Berbasis Kearifan Lokal. Jurnal Penelitian Pembelajaran Fisika,
15(1), 49–54. https://doi.org/10.26877/jp2f.v15i1.16683
Sartika, D., Nurlina, Mutmainna, Aris, N. A., & Musliana. (2023). The scientific literacy profile of senior
high school student based on science competence dimension. AIP Conference Proceedings, 2619.
https://doi.org/10.1063/5.0122563
Sartika, D., Nurlina, N., & Mutmainna, M. (2025). Scientific Literacy Assessment for Physics Instruction
by IntegrationLocal Wisdom of Mandar Tribe in West Sulawesi, Indonesia. JIPF (Jurnal Ilmu
Pendidikan Fisika), 10(2), 263–272. https://doi.org/10.26737/jipf.v10i2.6839
Şentürk, C., & Sari, H. (2018). Investigation of the contribution of differentiated instruction into science
literacy. Qualitative Research in Education, 7(2), 197–237. https://doi.org/10.17583/qre.2018.3383
Setiawati, F. A., Amelia, R. N., Sumintono, B., & Purwanta, E. (2023). Study Item Parameters of
Classical and Modern Theory of Differential Aptitude Test: Is it Comparable? European Journal of
Educational Research, 12(2), 1097–1107. https://doi.org/10.12973/eu-jer.12.2.1097
Shahzadi, I., & Nasreen, A. (2020). Assessing Scientific Literacy Levels among Secondary School
Science Students of District Lahore. Bulletin of Education and Research, 42(3), 1–21.
Sharkness, J., & DeAngelo, L. (2011). Measuring Student Involvement: A Comparison of Classical Test
Theory and Item Response Theory in the Construction of Scales from Student Surveys. Research in
Higher Education, 52(5), 480–507. https://doi.org/10.1007/s11162-010-9202-3
Sheng, Y. (2019). CTT Package in R. Measurement, 17(4), 211–219.
https://doi.org/10.1080/15366367.2019.1600839
Shi, H., Ren, Y., Xian, J., Ding, H., Liu, Y., & Wan, C. (2024). Item analysis on the quality of life scale
for anxiety disorders QLICD-AD(V2.0) based on classical test theory and item response theory.
Annals of General Psychiatry, 23(1), 1–12. https://doi.org/10.1186/s12991-024-00504-2
Shohib, M., Rahayu, Y., Wasis, W., & Hariyono, E. (2021). Scientific Literacy Ability of Junior High
School Students on Static Electricity and Electricity in Living Things. IJORER : International Journal
of Recent Educational Research, 2, 700–708. https://doi.org/10.46245/ijorer.v2i6.170
Syamsuddin, S. (2023). Implementasi Classic Test dan Item Respon Theory Pada Penilaian Tes
Pembelajaran Matematika. EDUSCOPE: Jurnal Pendidikan, Pembelajaran, Dan Teknologi, 8(2 SEArticles), 28–43. https://doi.org/10.32764/eduscope.v8i2.3488
Taber, K. S. (2018). The Use of Cronbach’s Alpha When Developing and Reporting Research
Instruments in Science Education. Research in Science Education, 48(6), 1273–1296.
https://doi.org/10.1007/s11165-016-9602-2
Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical
Education, 2, 53–55. https://doi.org/10.5116/ijme.4dfb.8dfd
Thissen, D. (2015). Psychometrics: Item Response Theory. In International Encyclopedia of the Social &
Behavioral Sciences: Second Edition (Second Edi, Vol. 19). Elsevier. https://doi.org/10.1016/B978-0-
08-097086-8.42071-4
Tong, L. K., Au, M. L., Zheng, M. R., Liu, Y. B., Fu, G. L., & Li, Y. Y. (2024). Health behavior
inventory – short form: a Chinese validation study using classical test theory and item response
theory. BMC Public Health, 24(1), 2281. https://doi.org/10.1186/s12889-024-19774-2
Uyigue, A. V., & Orheruata, M. U. (2019). Test Length and Sample Size for Item-Difficulty Parameter
Estimation in Item Response Theory. Journal of Education and Practice, 10(30), 72–75.
https://doi.org/10.7176/jep/10-30-08
Vilca, L. W., Chávez, B. V., Fernández, Y. S., & Caycho-Rodríguez, T. (2022). Spanish Version of the Revised Mental Health Inventory-5 (R-MHI-5): New Psychometric Evidence from the Classical
Test Theory (CTT) and the Item Response Theory Perspective (IRT). Trends in Psychology, 30(1),
111–128. https://doi.org/10.1007/s43076-021-00107-w
Wickham, H., Chang, W., Henry, L., Pedersen, T. L., Takahashi, K., Wilke, C., Woo, K., Yutani, H.,
Dunnington, D., & Brand, T. van den. (2025). Package ‘ggplot2’: Create Elegant Data Visualisations
Using the Grammar of Graphics (3.5.2). https://ggplot2.tidyverse.org
Wikcham, H., François, R., Henry, L., Müller, K., & Vaughan, D. (2023). Package “dplyr”: A Grammar
of Data Manipulation (1.1.4).
Yang, F. M., & Kao, S. T. (2014). Item response theory for measurement validity. Shanghai Archives of
Psychiatry, 26(3), 171–177. https://doi.org/10.3969/j.issn.1002-0829.2014.03.010
Yang, F., Zhao, F., Zheng, Y., & Li, G. (2020). Modification and verification of the Infant–Toddler
Meaningful Auditory Integration Scale: a psychometric analysis combining item response theory
with classical test theory. Health and Quality of Life Outcomes, 18(1), 367.
https://doi.org/10.1186/s12955-020-01620-9
Zhang, C., Zhang, H., Zhao, M., Liu, D., Zhao, Y., & Yao, Y. (2020). Assessment of Geriatric
Depression Scale’s Applicability in Longevous Persons based on Classical Test and Item Response
Theory. Journal of Affective Disorders, 274, 610–616.
https://doi.org/https://doi.org/10.1016/j.jad.2020.05.090
Zhang, D., Wang, C., Yuan, T., Li, X., Yang, L., Huang, A., Li, J., Liu, M., Lei, Y., Sun, L., Zhang, J.,
& Zhang, L. (2023). Psychometric properties of the Coronavirus Anxiety Scale based on Classical
Test Theory (CTT) and Item Response Theory (IRT) models among Chinese front-line healthcare
workers. BMC Psychology, 11(1). https://doi.org/10.1186/s40359-023-01251-x
Zhang, L., Liu, X., & Feng, H. (2023). Development and validation of an instrument for assessing
scientific literacy from junior to senior high school. Disciplinary and Interdisciplinary Science Education
Research, 5(1). https://doi.org/10.1186/s43031-023-00093-2
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Nurlina Nurlina, Mutmainna Mutmainna, Nursakinah Annisa Lutfin, Kaharuddin Arafah, Helmi Abdullah
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
