Development and Validation of Situational Judgement Test to Measure Continuous Learning Competency

Yunita Faela Nisa; Jahja Umar; Puspita Dian Arista; Helena Widaningrum; Bahrul Hayat

Development and Validation of Situational Judgement Test to Measure Continuous Learning Competency

Yunita Faela Nisa, Jahja Umar, Puspita Dian Arista, Helena Widaningrum, Bahrul Hayat

Abstract

Measuring continuous learning (CL) competency is beneficial to achieving success for individuals and organizations. For this reason, the availability of a brief but valid instrument is crucial. This study aims to develop a valid instrument to measure the CL using a situational judgment test (SJT). The instrument was developed and validated using respondents of 502 employees at the supervisory level in the electrical industry (with age M=30.13, SD=6.17). For content validity, we first held focus group discussions with supervisors and managers to identify and develop some essential characteristics of the CL. Subject matter experts were involved in writing and reviewing the items. A confirmatory factor analysis was conducted to test the construct validity, and it was found that 12 of 13 items fitted a unidimensional model. All the factor loadings were statistically significant (p<.05). A further test for parallel assumption was done to check the tau-equivalence. This test is vital because raw scores are mostly used in daily practice rather than scaled scores. The results showed that 12 items met the tau equivalent requirement. For further research, scoring for multiple-choice SJTs using the nominal response model (NRM) could be considered since there was a tendency for some of the questions to elicit ambiguous choices.

Keywords

continuous learning; confirmatory factor analysis; IRT; situational judgment test

References

Akour, M., & Al-Omari, H. (2013). Empirical Investigation of the Stability of IRT Item-Parameters Estimation. International Online Journal of Educational Sciences, 5(2), 291–301. www.iojes.net

Comrey, A. L. (1988). Factor-analytic methods of scale development in personality and clinical psychology. Journal of Consulting and Clinical Psychology, 56(5), 754-761.

Converse, J.M. & Presser, S. (1986). Survey questions: Handcrafting the standardized questionnaire. Sage University Paper.

Eddy, R.E., Tannenbaum, S.I., Lorenzet, S.J. & Smith-Jentsch, K.A. (2005). The influence of a continuous learning environment on peer mentoring behaviors. Journal of Managerial Issues, 17(3), 383-395.

Gibbons, R. D., & Hedeker, D. R. (1992). Full-information item bifactor analysis. Psychometrika, 57, 423–436.

Graham, J. M. (2006). Congeneric and (essentially) tau-equivalent estimates of score reliability. Educational and Psychological Measurement, 66(6), 930–944. https://doi.org/10.1177/ 0013164406288165

Hall, D. T., & Mirvis, P. H. (1995). The new career contract: Developing the whole person at midlife and beyond. Journal of Vocational Behavior, 47, 269–289.

Ilgen, D. R., & Pulakos, E. D. (1999). The changing nature of performance: Implications for staffing, motivation, and development. San Francisco: Jossey-Bass.

Jain, S., & Martindale, T. (2012). Facilitating continuous learning: Review of research on individual learning capabilities and organizational learning environments. Paper presented at the annual meeting of the AECT International Convention, Louisville, KY.

Johnston, R., & Hawke, G. (2002). Case studies of organizations with established learning cultures. Australian National Training Authority.

Joreskog, K. G. (1971). Statistical analysis of sets of congeneric tests. Psychometrika, 36, 109 - 133.

Kluge, A., & Schilling, J. (2003). Organizational learning and learning organizations. The Psychologist Manager Journal, 6, 31-50.

Leana CR, Feldman DC. (1988). Individual responses to job loss: Perceptions, reactions, and coping behaviors. Journal of Management. 14, 375–89 .

London, M., & Sessa, V. I. (2006). Group feedback for continuous learning. Human Resource Development Review, 5(3), 1–27.

London, M., & Smither, J. W. (1999). Career-related continuous learning: Defining the construct and mapping the process. In G. R. Ferris (Ed.), Research in human resources management (Vol. 17, pp. 81– 121). Stamford, CT: JAI Press.

Lord, F. M., & Novick, R. (1968). Statistical theories of mental test scores reading, MA: Addison Wesley.

Maurer, T. J. (2001). Career-relevant learning and development, worker age, and beliefs about self-efficacy for development. Journal of Management, 27, 123-140.

Maurer, T. J., & Weiss, E. M. (2010). Continuous learning skill demands: Associations with managerial job content, age, and experience. Journal of Business and Psychology, 25, 1–13. http://dx.doi.org/10.1007/s10869- 009-9126-0

McDaniel, M. A., Morgeson, F. P., Finnegan E. B., Campion, M. A., & Braverman, E. P. (2001). Predicting job performance using situational judgment tests: A clarification of the literature. Journal of Applied Psychology. 86, 730-740.

McDaniel, M. A., & Nguyen, N. T. (2001). Situational judgment tests: A review of practice and constructs assessed. International Journal of Selection and Assessment, 9, 103-113.

Mickelson, S. K. (2001). Development of workplace competencies sufficient to measure ABET outcomes. Paper presented at the 2001 ASEE Annual Conference and Exposition, Albu- querque, NM.

Mickelson, S. K. (2002). Validation of workplace competencies sufficient to measure ABET outcomes. Paper presented at the 2002 ASEE Annual Conference and Exposition, Montreal, Quebec, Canada.

Molloy, J.C., & Noe, R.A. (2009). Learning a living: Continuous learning for survival in today’s talent market. Routledge. https://www.taylorfrancis.com/chapters/edit/10.4324/9780203878385-19/learning-living-continuous-learning-survival-today-talent-market-janice-molloy-raymond-noe

Moss, F.A. (1926). Do you know how to get along with people? Scientific American, July 1926

Motowidlo, S. J., Hanson, M. A., & Crafts, J. L. (1997). Low-fidelity simulations. In D. L. Whetzel, G. R. Wheaton (Eds.), Applied measurement methods in industrial psychology. Palo Alto, CA: Consulting Psychologists Press.

PLN (2017). Turunan Indikator Perilaku Direktori Kompetensi PT PLN (Persero) 2017: Confidential. PT. PLN (Persero).

Schmidt, F. L., Hunter, J. E., & Outerbridge, A. N. (1986). Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of job. 71, 432-439.

Tannenbaum, S.I. (1998). Enhancing continuous learning: Diagnostic findings from multiple companies. Human Resources Management, 36, Issue 4, p.437-452.

Vinokur, A.D., Schul, Y., Vuori, J., Price, R. H. (2000). Two years after a job loss: long-term impact of the JOBS program on reemployment and mental health. Journal of Occupational Health Psychology, 5, 32–47.

Vuori J, Silvonen J. (2005). The benefits of a preventive job search program on re-employment and mental health at 2-year follow-up. Journal of Occupational and Organization Psychology, 78,43–52.

Whetzel, D. L., Sullivan, T.S., & McCloy, R.A. (2020) ‘Situational judgment tests: An overview of development practices and psychometric characteristics. Personnel Assessment and Decisions, 6 (1). Article 1. https://doi.org/10.25035/pad.2020.01.001

Wanberg, C. R., Hough, L. M., Song, Z. (2002). Predictive validity of a multidisciplinary model of reemployment success. Journal of Applied Psychology, 87, 1100–20.

Wanberg, C. R., Kanfer, R., Banas, J. T. (2000). Predictors and outcomes of networking intensity among unemployed job seekers. Journal of Applied Psychology, 85, 491–503.

Zu and Kyllonen (2020). Nominal response model is useful for scoring multiple-choice situational judgment tests. Organizational Research Methods, 23, 342-366. http://doi:10.1177/1094428118812669.

Full Text: PDF

DOI: 10.15408/jp3i.v11i1.25302