Comparison of KMO Results, Eigen Value, Reliability, and Standard Error of Measurement: Original & Rescaling Through Summated Rating Scaling
Abstract
This study aims to compare the results of KMO MSA analysis, Eigen Value, reliability, and Standard Error Measurement (SEm) between raw scores (original) and standardized scores (rescaling) through the summated rating scaling method on critical reasoning attitudes of vocational students in Yogyakarta City. This study used a quantitative descriptive approach involving 204 private vocational students as subjects. The instrument used to measure critical reasoning attitudes has gone through thorough validity and reliability testing before the research was carried out. The analysis process was carried out by calculating the KMO MSA, Eigen Value, reliability, and SEm values on both raw and standardized scores. The results of the two types of scores were then compared to identify any differences. Based on the results of the analysis, it was found that the raw score had a KMO MSA of 0.87, reliability of 0.823, and SEm of 0.337. After rescaling, the KMO MSA value decreased slightly to 0.86, the reliability also decreased slightly to 0.821, while the SEm increased to 0.406. Eigenvalue analysis showed that both the raw and standardized scores yielded seven factors with Eigenvalues greater than 1. The differences found between these two types of scores, namely 0.01 for KMO MSA, 0.002 for reliability, and -0.069 for SEm, indicate small but significant changes, especially in terms of the increase in SEm after rescaling, which impacts the level of measurement accuracy.
Keywords
References
Abduqodirovich, M. A. (2023). “Teaching foreign languages in the context of sustainable development: best practices, problems and opportunities” picularities and importance of critical thinking. International Scientific and Current Research Conferences, 1(1), 355–358. https://doi.org/10.55640/iscrc-3214
Alderson, J. Charles & Wall, D. (1992). Summated Rating Scale Construction: An Introduction. Series: Quantitative Applications in the Social Sciences. In Sage Publications, Inc. https://doi.org/10.20595/jjbf.19.0_3
Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Brooks/Cole.
Alotaibi, K. (2017). The Relationship Between Self-Regulated Learning and Academic Achievement for a Sample of Community College Students at King Saud University. Education Journal, 6(1), 28. https://doi.org/10.11648/j.edu.20170601.14
Amin, S., Kartowagiran, B., & Inang, P. (2018). A factor analysis of an instrument for measuring physical abuse experience of students at school. REID (Research and Evaluation in Education), 4(1), 22–34. https://doi.org/10.21831/reid.v4i1.18831
Azwar, Saifuddin. (2014). Reliabilitas dan validitas. Pustaka Pelajar.
Brown, T. C., & Daniel, T. C. (1990). Scaling of ratings: Concepts and methods. https://www.fs.usda.gov/rm/pubs_rm/rm_rp293.pdf
Carifio, J., & Perla, R. (2008). Resolving the 50-year debate around using and misusing Likert scales. In Medical Education (Vol. 42, Issue 12, pp. 1150–1152). https://doi.org/10.1111/j.1365-2923.2008.03172.x
Christmann, A., & Aelst, S. Van. (2006). Robust estimation of Cronbach’s alpha. 97, 1660–1674. https://doi.org/10.1016/j.jmva.2005.05.012
Clark, T., Foster, L., & Bryman, A. (2019). Analysing quantitative data. How to do your social research project or dissertation. Oxford University Press. https://doi.org/10.1093/hepl/9780198811060.003.0013
Cromwell, L. S. (1992). Assessing critical thinking. New Directions for Community Colleges, 1992. Wiley.
Crossley, J. (2016). Assessing the non-cognitive domains. Sage Publications.
Crossley, J. (2017). Assessing the non-cognitive domains: measuring what matters well. Routledge.
Effendi, H., Soenarto, S., & Sofyan, H. (2015). The effectiveness of web-based interactive blended learning model in electrical engineering courses. REID (Research and Evaluation in Education), 1(2), 175–185. https://doi.org/10.21831/reid.v1i2.7140
Fajri, F. R., & Amir, M. F. (2022). Math self-regulated learning assisted by metacognitive support by reviewing sex differences in mathematics anxiety. REID (Research and Evaluation in Education), 8(2), 100–113. https://doi.org/10.21831/reid.v8i2.49157
Faralina, A. A. N., Kadri, A. S., & Yap, A. N. S. (2016). Validity and reliability of pre-class reading tasks for waves and optics (PRT-WO). REID (Research and Evaluation in Education), 2(1), 42–52. https://doi.org/10.21831/reid.v2i1.8466
Flinton, D., & Malamateniou, C. (2020). Quantitative methods and analysis. Pearson.
Golden, B. (2023). Enabling critical thinking development in higher education through the use of a structured planning tool. Irish Educational Studies, 42(4), 949–969. https://doi.org/10.1080/03323315.2023.2258497
Hajaroh, M., Rukiyati, Purwastuti, L. A., & Nurhayati, R. (2021). Development of the evaluation instrument of the child-friendly school policy in elementary schools. International Journal of Instruction, 14(3), 327–340. https://doi.org/10.29333/iji.2021.14319a
Hamdi, S., Suganda, I. A., & Hayati, N. (2018). Developing higher-order thinking skill (HOTS) test instrument using Lombok local cultures as contexts for junior secondary school mathematics. REID (Research and Evaluation in Education), 4(2), 126–135. https://doi.org/10.21831/reid.v4i2.22089
Hasanatin, M., & Rohaeti, E. (2021). How are the characteristics of integrated assessment instruments to measure critical thinking skills and scientific attitudes of high school students? Jurnal Penelitian Pendidikan IPA, 7(SpecialIssue), 359–364. https://doi.org/10.29303/jppipa.v7ispecialissue.1082
Hudha, S. A., & Mardapi, D. (2018). Developing an instrument for measuring the spiritual attitude of high school students. REID (Research and Evaluation in Education), 4(1), 35–44. https://doi.org/10.21831/reid.v4i1.20304
Jamieson, S. (2004). Likert scales: How to (ab)use them. In Medical Education (Vol. 38, Issue 12, pp. 1217–1218). https://doi.org/10.1111/j.1365-2929.2004.02012.x
Jian-hua, C. (2005). The application of SPSS (software) in the sports measurement & survey. Sports Sciences Researches.
Kalkbrenner, M. T. (2021). A practical guide to instrument development and score validation in the social sciences: The measure approach. Practical Assessment, Research and Evaluation, 26(1), 1–18. https://doi.org/https://doi.org/10.7275/svg4-e671
Kampen, J. K. (2019). Reflections on and test of the metrological properties of summated rating, Likert, and other scales based on sums of ordinal variables. Measurement: Journal of the International Measurement Confederation, 137, 428–434. https://doi.org/10.1016/j.measurement.2019.01.083
Kim, M., & Pegg, J. (2019). Case analysis of children’s reasoning in problem-solving process. International Journal of Science Education, 41(6), 739–758. https://doi.org/10.1080/09500693.2019.1579391
Kumolohadi, R., Mangunsong, F., & Suleeman, J. (2021). Development and validation of ethical awareness scale among university students. Jurnal Pengukuran Psikologi dan Pendidikan Indonesia, 10(1), 20–31. https://doi.org/10.15408/jp3i.v10i1.17216
Kurniawan, N. I. A., & Munadi, S. (2019). Analysis of the quality of test instrument and students’ accounting learning competencies at vocational school. Jurnal Penelitian dan Evaluasi Pendidikan, 23(1), 68–75. https://doi.org/10.21831/pep.v23i1.22484
Lei, Y. (2006). Application of SPSS technique in demonstration study data for statistics and analysis. Computer Engineering and Applications.
Liu, J., & Cohen, J. (2021). Measuring teaching practices at scale: A novel application of text-as-data methods. Educational Evaluation and Policy Analysis, 43(4), 587–614. https://doi.org/10.3102/01623737211009267
Lorenzo-Seva, U., & Ferrando, P. J. (2021). MSA: The Forgotten Index for Identifying Inappropriate Items Before Computing Exploratory Item Factor Analysis. Methodology, 17(4), 296–306. https://doi.org/10.5964/meth.7185
MacInnes, J. (2017). An Introduction to Secondary Data Analysis with IBM SPSS Statistics. Sage Publications.
Meyer, E. P., Kaiser, H. F., Cerny, B. A., & Green, B. F. (1977). MSA for a special spearman matrix. Psychometrika, 42 (1), 153-156.
Nkansah, B. K. (2018). On the Kaiser-Meier-Olkin’s Measure of Sampling Adequacy. Mathematical Theory and Modeling, 8(7), 52–76. https://iiste.org/Journals/index.php/MTM/article/view/44386/45790
Nolan, M. M., Beran, T., & Hecker, K. G. (2012). Surveys assessing students’ attitudes toward statistics: A systematic review of validity and reliability. Statistics Education Research Journal, 11(2), 103–123. https://doi.org/10.52041/serj.v11i2.333
Okoye, K., Rodriguez-Tort, J. A., Escamilla, J., & Hosseini, S. (2021). Technology-mediated teaching and learning process: A conceptual study of educators’ response amidst the Covid-19 pandemic. Education and Information Technologies, 26(6), 7225–7257. https://doi.org/10.1007/s10639-021-10527-x
Prasasti, I. H., & Istiyono, E. (2018). Developing an instrument of national examination of equivalency education Package C of mathematics subject. REID (Research and Evaluation in Education), 4(1), 58–69. https://doi.org/10.21831/reid.v4i1.15556
Proches, S. (2016). Descriptive statistics in research and teaching: are we losing the middle ground? Quality & Quantity. Springer Netherlands. 10.1007/s11135-015-0256-3
Purnamasari, N., Siswanto, S., & Malik, S. (2020). E-module as an emergency-innovated learning source during the Covid-19 outbreak. Psychology, Evaluation, and Technology in Educational Research, 3(1), 1–8. https://doi.org/10.33292/petier.v3i1.53
Puspitasari, E. D., Susilo, M. J., & Febrianti, N. (2019). Developing psychomotor evaluation instrument of biochemistry practicum for university students of biology education. REID (Research and Evaluation in Education), 5(1), 1–9. https://doi.org/10.21831/reid.v5i1.22126
Pramudita, K., Rosnawati, R., & Mam, S. (2019). Methods used by mathematics teachers in developing parallel multiple-choice test items in school. REID (Research and Evaluation in Education), 5(1), 10–20. https://doi.org/10.21831/reid.v5i1.22219
Rachmantika, A. R., & Wardono. (2019). The role of students’ critical thinking skills in learning mathematics with problem-solving. Prosiding Seminar Nasional Matematika, 2(1), 441.
Rasool, S., Cerchione, R., Salo, J., Ferraris, A., & Abbate, S. (2021). Measurement of consumer awareness of food waste: construct development with a confirmatory factor analysis. British Food Journal, 123(13), 337–361. https://doi.org/10.1108/BFJ-02-2021-0160
Retnawati, H. (2016). Analisis kuantitatif instrumen penelitian (panduan peneliti, mahasiswa, dan psikometrian). Parama Publishing.
Rosana, D., Widodo, E., Setianingsih, W., & Setyawarno, D. (2020). Developing Assessment Instruments of PISA Model to Measure Students’ Problem-Solving Skills and Scientific Literacy in Junior High Schools. Jurnal Pendidikan Sains Indonesia, 8(2), 292–305. https://doi.org/10.24815/jpsi.v8i2.17468
Sabon, Y. O. S., Istiyono, E., & Widihastuti, W. (2022). Developing “Pancasila Student Profile” instrument for self-assessment. Jurnal Penelitian dan Evaluasi Pendidikan, 26(1), 37–46. https://doi.org/10.21831/pep.v26i1.45144
Santoso, P. H., Istiyono, E., & Haryanto. (2022). Principal Component Analysis and Exploratory Factor Analysis of the Mechanical Waves Conceptual Survey. Jurnal Pengukuran Psikologi Dan Pendidikan Indonesia, 11(2), 208–224. https://doi.org/10.15408/jp3i.v11i2.27488
Setiawan, R., Wagiran, W., & Alsamiri, Y. (2024). Construction of an instrument for evaluating the teaching process in higher education: Content and construct validity. REID (Research and Evaluation in Education), 10(1), 50–63. https://doi.org/10.21831/reid.v10i1.63483
Setiawati, F. A. (2013). Penskalaan tipe likert dan thurstone dengan teori klasik dan modern: stidu pada instrumen multiple intelligences. Yogyakarta, Universitas Negeri Penelitian, Lembaga Pengabdian, dan Masyarakat, Kepada Masyarakat 2013, 42.
Setiawati, F. A. (2014). Perbandingan berbagai metode penskalaan yang dikembangkanThurstone. Jurnal Penelitian Ilmu Pendidikan, 7(1), 86–93.
Setiawati, F. A., Mardapi, D., & Azwar, S. (2013). Penskalaan Teori Klasik Instrumen Multiple Intelligences Tipe Thurstone Dan Likert. Jurnal Penelitian Dan Evaluasi Pendidikan, 17(2), 259–274. https://doi.org/10.21831/pep.v17i2.1699
Sharma, B. (2016). A focus on reliability in developmental research through Cronbach’s Alpha among medical, dental and paramedical professionals. Asian Pacific Journal of Health Sciences, 3(4), 271–278. https://doi.org/10.21276/apjhs.2016.3.4.43
Shevlin, M., Miles, J. N. V., & Bunting, B. P. (1997). Summated rating scales: a monte carlo investigation of the effects of reliability and collinearity in regression models. Elsevier Science, 23(4), 665–676. https://doi.org/10.1016/S0191-8869(97)00088-3
Spector, P. E. (1976). Choosing Response Categories for Summated Rating Scales. Journal of Applied Psychology, 61(3), 374–375. https://doi.org/10.1037/0021-9010.61.3.374
Spector, P. E. (1992). Summated rating scale construction: an introduction. Sage Publications.
Steenbergen-Hu, S., Olszewski-Kubilius, P., & Calvert, E. (2020). The Effectiveness of Current Interventions to Reverse the Underachievement of Gifted Students: Findings of a Meta-Analysis and Systematic Review. Gifted Child Quarterly, 64(2), 132–165. https://doi.org/10.1177/0016986220908601
Stępień, K., Janecki, D., & Adamczak, S. (2021). Statistical study of the method of radius-change sphericity measurements. Sensors, 21(13), 1–19. https://doi.org/10.3390/s21134563
Sugiyono, S., Mardapi, D., & Suryadarma, I. G. P. (2017). Developing assessment instruments for the sensory acceptability of food products. REID (Research and Evaluation in Education), 3(1), 92–105. https://doi.org/10.21831/reid.v3i1.13940
Sumin, Sukmawati, F., Setiawati, F. A., & Asmawi, S. (2022). The Impact of Z-Score Transformation Scaling on the Validity, Reliability, and Measurement Error of Instrument SATS-36. Jurnal Pengukuran Psikologi Dan Pendidikan Indonesia, 11(2), 166–180. https://doi.org/10.15408/jp3i.v11i2.26591
Sumintono, B. (2015). Aplikasi Pemodelan Rasch pada Assessment Pendidikan. www.trimkomunikata.com
Suryabrata, S. (2002). Pengembangan alat ukur psikologis. Andi Offset.
Syahputra, W., Yundianto, D., Baydhowi, & Paramita, N. D. (2023). The Indonesian Democratic Citizenship Attitude Scale: Translation, Cross-Cultural Adaptation, and Psychometric Properties. Jurnal Pengukuran Psikologi Dan Pendidikan Indonesia, 12(2), 191–205. https://doi.org/10.15408/jp3i.v12i2.28414
Taber, K. S. (2018). The Use of Cronbach’ Alpha When Developing and Reporting Research Instruments in Science Education. Research in Science Education, 48(6), 1273–1296. https://doi.org/10.1007/s11165-016-9602-2
Teng, Y., & Wang, X. (2021). The effect of two educational technology tools on student engagement in Chinese EFL courses. International Journal of Educational Technology in Higher Education, 18(1), 0–15. https://doi.org/10.1186/s41239-021-00263-0
Tyler, F., & Michael, W. B. (1958). An Empirical Study of the Comparability of Factor Structure when Unities and Communality Estimates are Used. Educational and Psychological Measurement, 18(2), 347–354. https://doi.org/10.1177/00131644580180021
Vijayendra, K., & Fantone, A. (2023). Recent Trends of Quantitative Approaches in Different Sectors: A Concise Review. Asian Journal of Education and Social Studies, 41(2), 22–34. https://doi.org/10.9734/ajess/2023/v41i2891
Wardani, R. E. A., Supriyoko, S., & Prihatni, Y. (2018). Developing assessment model for bandel attitudes based on the teachings of Ki Hadjar Dewantara. REID (Research and Evaluation in Education), 4(2), 117–125. https://doi.org/10.21831/reid.v4i2.21942
Wiyono, M., Solihin, F., & Putro, S. S. (2017). Aplikasi Penilaian Kuliah Kerja Nyata Universitas Trunojoyo Madura Menggunakan Metode Rating Scale. Jurnal Imliah Rekayasa, 10(1), 23–33. https://doi.org/10.21107/rekayasa.v10i1.3601
DOI: 10.15408/jp3i.v13i2.36684
Refbacks
- There are currently no refbacks.
Copyright (c) 2024 Nuryani Dwi Astuti, Farida Agus Setiawati, Heri Retnawati, Yuli Prihatni, Ari Setiawan
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
