Ensuring Fairness in Educational Book Evaluation: A Many-Facet Rasch Analysis ofRater Severity and Item Functioning

Suprananto Suprananto; Bambang  Suryadi; Jenny Lewis; Valendra Granitha Shandika Puri

doi:10.15408/jp3i.v15i2.51647

Authors

Suprananto Suprananto Universitas Singaperbangsa Karawang
Bambang Suryadi Faculty of Psychology, UIN Syarif Hidayatullah Jakarta, Indonesia
Jenny Lewis School of Education and Creative Arts, University of Southern Queensland, Australia
Valendra Granitha Shandika Puri Faculty of Psychology, UIN Syarif Hidayatullah Jakarta, Indonesia

DOI:

https://doi.org/10.15408/jp3i.v15i2.51647

Keywords:

assessment validity, book evaluation, many-facet rasch model, rater severity

Abstract

Maintaining the quality of educational textbooks requires valid, reliable, and transparent evaluation.
However, multi-rater assessments may be affected by differences in judgment and rater severity. This
study examined a textbook evaluation system using the Many-Facet Rasch Model (MFRM). Six
evaluators assessed three books using a 12-item instrument. The findings revealed substantial variation
in rater severity, including one unusually severe rater, indicating a need to improve consistency in
evaluators’ judgments. Most items functioned adequately, but two showed misfit and required revision
or removal. The analysis also distinguished differences in the estimated quality of the three books while
accounting for rater effects. These findings demonstrate the practical value of MFRM for strengthening
the fairness and credibility of textbook evaluation. Institutions responsible for textbook assessment should
regularly review evaluation instruments, clarify rating criteria, and provide training and calibration to
promote consistent judgments. They should also investigate unusually severe or inconsistent rating
patterns and revise or remove items that do not function as intended. Incorporating MFRM into
evaluation procedures can help institutions identify weaknesses in instruments and rating practices,
support evidence-based textbook selection, and improve quality assurance. These steps can help
policymakers establish more transparent and rigorous processes for evaluating educational materials.

References

Adie, L., Klenowski, V., & Wyatt-Smith, C. (2021). Moderation and assessment in higher education: A review of the literature. Assessment & Evaluation in Higher Education, 46(2), 185–197. https://doi.org/10.1080/02602938.2020.1729424

Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge.

Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch analysis in the human sciences. Springer. https://doi.org/10.1007/978-94-007-6857-4

Boone, W. J., Staver, J. R., & Yale, M. S. (2024). Rasch analysis in the human sciences (2nd ed.). Springer. https://doi.org/10.1007/978-981-99-9096-6

Brookhart, S. M. (2013). How to create and use rubrics for formative assessment and grading. ASCD.

de Klerk, S., Bosker, R. J., & van de Grift, W. (2023). Rater effects and reliability issues in performance assessment: Recent developments in many-facet Rasch measurement. Educational Measurement: Issues and Practice, 42(2), 45–57. https://doi.org/10.1111/emip.12532

Eckes, T. (2015). Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments (2nd ed.). Peter Lang.

Eckes, T. (2019). Many-facet Rasch measurement in language assessment: Principles and applications. Language Testing, 36(3), 321–342. https://doi.org/10.1177/0265532219841043

Engelhard, G. (2013). Invariant measurement: Using Rasch models in the social, behavioral, and health sciences. Routledge.

Jin, K. Y., & Chiu, M. M. (2021). Exploring rater effects in performance assessments using many-facet Rasch measurement. Educational Psychology, 41(7), 879–897. https://doi.org/10.1080/01443410.2021.1882050

Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000

Linacre, J. M. (1989). Many-facet Rasch measurement. MESA Press.

Linacre, J. M. (1994). Many-facet Rasch measurement. In M. Wilson (2nd Ed.), Objective measurement: Theory into practice (Vol. 2, pp. 3–32). Ablex.

Linacre J.M. (1999) Investigating rating scale category utility. Journal of Outcome Measurement 3:2, 103-122.

Linacre, J. M. (2002). What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions, 16(2), 878.

Liu, Y., & Jin, Y. (2022). Applications of many-facet Rasch measurement in educational assessment: A systematic review. Frontiers in Psychology, 13, 845327. https://doi.org/10.3389/fpsyg.2022.845327

Myford, C. M., & Wolfe, E. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386–422.

Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research.

Republik Indonesia. (2017). Undang-Undang Republik Indonesia Nomor 3 Tahun 2017 tentang Sistem Perbukuan. Pemerintah Republik Indonesia.

Republik Indonesia. (2019). Peraturan Pemerintah Republik Indonesia Nomor 75 Tahun 2019 tentang Peraturan Pelaksanaan Undang-Undang Nomor 3 Tahun 2017 tentang Sistem Perbukuan. Pemerintah Republik Indonesia.

Tai, J., Ajjawi, R., Boud, D., Dawson, P., & Panadero, E. (2022). Developing evaluative judgement: Enabling students to make decisions about the quality of work. Higher Education, 83(2), 433–448. https://doi.org/10.1007/s10734-021-00648-4

Trace, J., Meier, S., & Janssen, G. (2020). Rater cognition and scoring validity in educational assessment. Assessing Writing, 45, 100469. https://doi.org/10.1016/j.asw.2020.100469

Wind, S. A. (2023). Advances in many-facet Rasch measurement for educational assessment. Educational Measurement: Issues and Practice, 42(1), 5–17. https://doi.org/10.1111/emip.12530

Wind, S. A. (2023). Advances in rater-mediated assessment and many-facet Rasch measurement. Frontiers in Education, 8, 1187452. https://doi.org/10.3389/feduc.2023.1187452

Wind, S. A., & Guo, F. (2019). Exploring the application of many-facet Rasch measurement in educational assessment. Educational Research Review, 26, 1–15.

https://doi.org/10.1016/j.edurev.2019.100279

Wind, S. A., & Guo, F. (2021). Examining rater effects using many-facet Rasch measurement in educational assessment contexts. Journal of Educational Measurement, 58(3), 344–366. https://doi.org/10.1111/jedm.12298

Wind, S. A., & Jones, E. (2019). The many-facet Rasch model. In V. Zeigler-Hill & T. K. Shackelford (Eds.), Encyclopedia of personality and individual differences (pp. 1–4). Springer. https://doi.org/10.1007/978-3-319-28099-8_1348-1

Wind, S. A., & Jones, E. (2022). Psychometric issues in rater-mediated assessment systems. Frontiers in Education, 7, 873214. https://doi.org/10.3389/feduc.2022.873214

Wolfe, E. W., & McVay, A. (2012). Application of latent trait models to identifying rater effects. Educational Measurement: Issues and Practice, 31(2), 30–39. https://doi.org/10.1111/j.1745-3992.2012.00232.x

Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370.

Wright, B. D., & Masters, G. N. (1982). Rating scale analysis. MESA Press.