Evaluation of Different Person-Fit Measures in Cognitive Tests with Different Test Lengths

Sukaesi Marianti; Herdin Natalius Manao; Arij Faiha

Evaluation of Different Person-Fit Measures in Cognitive Tests with Different Test Lengths

Sukaesi Marianti, Herdin Natalius Manao, Arij Faiha

Abstract

Test takers’ characteristic is an exciting topic to discuss in psychometric research. In this study, person-fit is a part of the person characteristics applied in the context of cognitive tests. Given the importance of accurately estimating item and person parameters, person-fit is a statistical technique that can detect aberrant responses. Aberrance adversely affects the estimation process at the level of items and persons. The purpose of this study was to introduce and apply two popular person-fit statistics called and . These two statistics were applied in two studies, in study 1 using N = 317 and item = 16, and in study 2 using N = 331 and item = 49. The results showed that in studies 1 and 2, detected more aberrant responses compared to . Significant differences in estimated results from both techniques were also shown in Study 2. The outcomes of this study are valuable for researchers and practitioners in the field of psychometrics who rely on , as a foundation for identifying aberrant responses.

Keywords

aberrant response pattern; l_z ; l_z^*; person-fit.

References

De La Torre, J., & Deng, W. (2008). Improving person‐fit assessment by correcting the ability estimate and its reference distribution. Journal of Educational Measurement, 45(2), 159–177.

Desjardins, C. D., & Bulut, O. (2018). Handbook of Educational Measurement and Psychometrics Using R (1st ed.). Chapman and Hall. https://doi.org/10.1201/b20498

Drasgow, F., Levine, M. V., & Williams, E. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38(1), 67–86. https://doi.org/https://doi.org/10.1111/j.2044-8317.1985.tb00817.x

Karabatsos, G. (2003). Comparing the Aberrant Response Detection Performance of Thirty-Six Person-Fit Statistics. Applied Measurement in Education, 16(4), 277–298. https://doi.org/10.1207/S15324818AME1604_2

Levine, M. V, & Rubin, D. B. (1979). Measuring the appropriateness of multiple-choice test scores. Journal of Educational Statistics, 4(4), 269–290.

Magis, D., Raîche, G., & Béland, S. (2012). A didactic presentation of snijders’s l z* index of person fit with emphasis on response model selection and ability estimation. Journal of Educational and Behavioral Statistics, 37(1), 57–81. https://doi.org/10.3102/1076998610396894

Marianti, S., Fox, J.-P., Avetisyan, M., Veldkamp, B. P., & Tijmstra, J. (2014). Testing for aberrant behavior in response time modeling. Journal of Educational and Behavioral Statistics, 39(6), 426–451.

Meijer, R. R. (1996). Person-fit research: An introduction. Applied Measurement in Education1, 9(1), 3–8. https://doi.org/https://doi.org/10.1207/s15324818ame0901_2

Meijer, R. R., & Sijtsma, K. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25(2), 107–135. https://doi.org/10.1177/01466210122031957

Molenaar, I. W., & Hoijtink, H. (1990). The many null distributions of person fit indices. Psychometrika, 55(1), 75–106.

Mousavi, A., & Cui, Y. (2020). The Effect of Person Misfit on Item Parameter Estimation and Classification Accuracy: A Simulation Study. Education Sciences, 10(11), 324.

Nurcahyo, F. A. (2017). Aplikasi IRT dalam Analisis Aitem Tes Kognitif. Buletin Psikologi, 24(2), 64–75. https://doi.org/10.22146/buletinpsikologi.25218

Olejnik, Stephen; Li, J. S. S. (1997). Multiple Testing and Statistical Power With Modified Bonferroni Procedures. Journal of Educational and Behavioral Statistics, 22(4), 389–406. https://doi.org/https://doi.org/10.3102/10769986022004389

Reise, S. P., & Due, A. M. (1991). Test characteristics and their influence on the detection of aberrant response patterns. Applied Psychological Measurement, 15(1), 217–226.

Rizopoulos, D. (2006). ltm : An R Package for Latent Variable Modelling and Item Response Theory Analyses. Journal of Statistical Software, 17(5), 1–25. www.jstarsoft.org/v17/i05

Snijders, T. A. B. (2001). Asymptotic null distribution of person fit statistics with estimated person parameter. Psychometrika, 66(3), 331–342. https://doi.org/10.1007/BF02294437

Tendeiro, J. N., & Meijer, R. R. (2014). Detection of invalid test scores: The usefulness of simple nonparametric statistics. Journal of Educational Measurement, 51(3), 239–259. https://doi.org/10.1111/jedm.12046

Tendeiro, J. N., Meijer, R. R., & Niessen, A. S. M. (2016). PerFit: An R package for person-fit analysis in IRT. Journal of Statistical Software, 74(5). https://doi.org/10.18637/jss.v074.i05

van der Linden, W. J., & Guo, F. (2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73(3), 365–384.

Full Text: PDF

DOI: 10.15408/jp3i.v13i2.34601