The Development of The Situational Judgement Test (SJT): Workforce Agility

Avin Fadilla Helmi; Ramadhan Dwi Marvianto; Arif Budiman Al Fariz; Indahinsani Purnasari Anggoro; Wahyu Jati Anggoro

The Development of The Situational Judgement Test (SJT): Workforce Agility

Avin Fadilla Helmi, Ramadhan Dwi Marvianto, Arif Budiman Al Fariz, Indahinsani Purnasari Anggoro, Wahyu Jati Anggoro

Abstract

This research aimed to develop a workforce agility measuring tool with the situational judgment test (SJT) format and evaluate its psychometric properties. This research included 886 respondents with higher education. The data were analyzed using confirmatory factor analysis (CFA) and multi-group confirmatory factor analysis (MGCFA). The CFA results showed that 18 items have satisfactory psychometric properties, thus forming a fit factor structure and providing validity evidence based on internal structure. Besides, the MGCFA results showed that this measuring instrument has measurement invariance at the strict invariance level to be used as validity evidence based on test consequences. The implication is that this measuring instrument has at least three of the five sources of validity to allow solid interpretations of the measuring results.

Keywords

Workforce agility, situational judgment test, multigroup confirmatory factor analysis, validity, psychometrics

References

Aamodt, M. G. (2010). Industrial organizational psychology: an apploed approach (6th ed.). Wadsworth Cengage Leaning.

Alavi, S. (2016). The influence of workforce agility on external manufacturing flexibility of Iranian SMEs. International Journal of Technological Learning, Innovation and Development, 8(1), 111. https://doi.org/https://doi.org/10.1504/IJTLID.2016.075185

Alavi, S., Wahab, D. A., Muhamad, N., & Shirani, B. A. (2014). Organic structure and organisational learning as the main antecedents of workforce agility. International Journal of Production Research, 52(21), 6273–6295. https://doi.org/10.1080/00207543.2014.919420

Aldrup, K., Carstensen, B., Köller, M. M., & Klusmann, U. (2020). Measuring Teachers’ Social-Emotional Competence: Development and Validation of a Situational Judgment Test . In Frontiers in Psychology (Vol. 11).

Al-Faouri, A. H., Al-Nsour, M. M., & Al-Kasasbeh, M. M. (2014). The impact of workforce agility on organizational memory. Knowledge Management Research and Practice, 12(4), 432–442. https://doi.org/10.1057/kmrp.2013.19

Alshurideh, M. T., Hassanien, A. E., & Masa’deh, R. (Eds.). (2021). The effect of coronavirus disease (COVID-19) on business intelligence: studies in system, decision and control. Springer International Publishing. https://doi.org/http://dx.doi.org/10.1007/978-3-030-67151-8

American Educational Research Association (AERA), American Psychologucal Association (APA), & National Council on Measurement in Education (NCME). (2014). Standards for educational and psychological testing. American Educational Research Association.

Andrea, M., K., J. B., & Allison, G. (2020). Development of Global Engineering Competency Scale: Exploratory and Confirmatory Factor Analysis. Journal of Civil Engineering Education, 146(2), 4019003. https://doi.org/10.1061/(ASCE)EI.2643-9115.0000006

Barron, L. G., Aguilar, I. D., Rose, M. R., & Carretta, T. R. (2021). Development of a situational judgment test to supplement current US air force measures of officership. Military Psychology. https://doi.org/10.1080/08995605.2021.1997500

Bean, G. J., & Bowen, N. K. (2021). Item Response Theory and Confirmatory Factor Analysis: Complementary Approaches for Scale Development. Journal of Evidence-Based Social Work, 18(6), 597–618. https://doi.org/10.1080/26408066.2021.1906813

Braun, T. J., Hayes, B. C., DeMuth, R. L. F., & Taran, O. A. (2017). The Development, Validation, and Practical Application of an Employee Agility and Resilience Measure to Facilitate Organizational Change. Industrial and Organizational Psychology, 10(4), 703–723. https://doi.org/10.1017/iop.2017.79

Breu, K., Hemingway, C. J., Strathern, M., & Bridger, D. (2001). Workforce agility: the new employee strategy for the knowledge economy Workforce agility: the new employee strategy for the knowledge economy. Journal of Information Technology, 17(1), 21–31. https://doi.org/10.1080/02683960110132070

Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). The Guilford Press.

Cai, Z., Huang, Q., Liu, H., & Wang, X. (2018). International Journal of Information Management Improving the agility of employees through enterprise social media : The mediating role of psychological conditions. International Journal of Information Management, 38(1), 52–63. https://doi.org/10.1016/j.ijinfomgt.2017.09.001

Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504. https://doi.org/10.1080/10705510701301834

Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233–255. https://doi.org/10.1207/S15328007SEM0902_5

Christian, M. S., Edwards, B. D., & Bradley, J. C. (2010). Situational judgment tests: Constructs assessed and a meta-analysis of their criterion-related validities. Personnel Psychology, 63(1), 83–117. https://doi.org/10.1111/j.1744-6570.2009.01163.x

Corstjens, J., Lievens, F., & Krumm, S. (2017). Situational Judgement Tests for Selection. The Wiley Blackwell Handbook of the Psychology of Recruitment, Selection and Employee Retention, 226–246. https://doi.org/10.1002/9781118972472.ch11

Fajrianthi, & Zein, R. A. (2017). Development of a psychological test to measure ability-based emotional intelligence in the indonesian workplace using an item response theory. Psychology Research and Behavior Management, 10, 339–352. https://doi.org/10.2147/PRBM.S143113

Greatrix, R., & Dowell, J. (2020). UKCAT and medical student selection in the UK-what has changed since 2006? BMC Medical Education, 20(1). https://doi.org/10.1186/s12909-020-02214-1

Gupta, M., George, J. F., & Xia, W. (2019). Relationships between IT department culture and agile software development practices: an empirical investigation. International Journal of Information Management, 44(June 2018), 13–24. https://doi.org/10.1016/j.ijinfomgt.2018.09.006

Hair, J. F., Tatham, R. L., Anderson, R. E., & Black, W. (1998). Multivariate Data Analysis (Fith Ed.). Prentice-Hall.

Hayes, A. F., & Coutts, J. J. (2020). Use Omega Rather than Cronbach’s Alpha for Estimating Reliability. But…. Communication Methods and Measures, 14(1), 1–24. https://doi.org/10.1080/19312458.2020.1718629

Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

Huber, P. J. (1981). Robust statistics. Wiley.

Johannsen, L. M., Frerichs, W., Inhestern, L., & Bergelt, C. (2020). Assessing competencies of healthcare professionals caring for parents with cancer: The development of an innovative assessment tool. Psycho-Oncology, 29(10), 1670–1677. https://doi.org/10.1002/pon.5507

Junior, G. T., & Saltorato, P. (2021). Workforce agility : a systematic literature review and a research agenda proposal. Innovar, 31(81), 155–168. https://doi.org/https://doi.org/10.15446/innovar.v31n81.95582

Juster, F. R., Baum, R. C., Zou, C., Risucci, D., Ly, A., Reiter, H., Miller, D. D., & Dore, K. L. (2019). Addressing the Diversity-Validity Dilemma Using Situational Judgment Tests. Academic Medicine, 94(8), 1197–1203. https://doi.org/10.1097/ACM.0000000000002769

Karaca, A., Aydogmus, M. E., & Gunbas, N. (2022). Enforced remote work during covid-19 and the importance of technological competency: a job demands-resources perspective. European Review of Applied Psychology, 1–52. https://doi.org/10.1016/j.erap.2022.100867

Kasten, N., Freund, P. A., & Staufenbiel, T. (2020). “Sweet little lies”: An in-depth analysis of faking behavior on Situational Judgment Tests compared to personality questionnaires. European Journal of Psychological Assessment, 36(1), 136–148. https://doi.org/10.1027/1015-5759/a000479

Kim, H.-Y. (2013). Statistical notes for clinical researchers: assessing normal distribution (2) using skewness and kurtosis. Restorative Dentistry & Endodontics, 38(1), 52. https://doi.org/10.5395/rde.2013.38.1.52

Kniffin, K. M., Narayanan, J., Anseel, F., Antonakis, J., Ashford, S. P., Bakker, A. B., Bamberger, P., Bapuji, H., Bhave, D. P., Choi, V. K., Creary, S. J., Demerouti, E., Flynn, F. J., Gelfand, M. J., Greer, L. L., Johns, G., Kesebir, S., Klein, P. G., Lee, S. Y., … Vugt, M. van. (2021). COVID-19 and the workplace: implications, issues, and insights for future research and action. American Psychologist, 76(1), 63–77. https://doi.org/10.1037/amp0000716

Koschmieder, C., & Neubauer, A. C. (2021). Measuring emotion regulation for preservice teacher selection: A theory-driven development of a situational judgment test. Personality and Individual Differences, 168. https://doi.org/10.1016/j.paid.2020.110363

Kreitchmann, R. S., Abad, F. J., Ponsoda, V., Nieto, M. D., & Morillo, D. (2019). Controlling for response biases in self-report scales: forced-choice vs. Psychometric modeling of likert items. Frontiers in Psychology, 10(October), 1–12. https://doi.org/10.3389/fpsyg.2019.02309

Lievens, F., Peeters, H., & Schollaert, E. (2008). Situational judgment tests: A review of recent research. Personnel Review, 37(4), 426–441. https://doi.org/10.1108/00483480810877598

Männiste, T., Pedaste, M., & Schimanski, R. (2019). Situational judgment test for measuring military tactical decision-making skills. Military Psychology, 31(6), 462–473. https://doi.org/10.1080/08995605.2019.1664366

McDonald. (1999). Test theory: A unified treatment. Lawrence Earlbaum Associates.

McDonald, R. P., & Ho, M. H. R. (2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7(1), 64–82. https://doi.org/10.1037/1082-989X.7.1.64

Misra, S., Kumar, V., Kumar, U., Fantazy, K., & Akhter, M. (2012). Agile software development practoces: evolution, principles, and criticisms. International Journal of Quality & Reliability Management, 29(9), 972–980. https://doi.org/10.1108/02656711211272863

Muduli, A., & Pandaya, G. (2018). Psychological empowerment and workforce agility. Psychological Studies, 63(3), 276–285. https://doi.org/10.1007/s12646-018-0456-8

Olaru, G., Burrus, J., MacCann, C., Zaromb, F. M., Wilhelm, O., & Roberts, R. D. (2019). Situational judgment tests as a method for measuring personality: development and validity evidence for a test of dependability. PLoS ONE, 14(2), 1–19. https://doi.org/10.1371/journal.pone.0211884

Patterson, F., Zibarras, L., & Ashworth, V. (2015). Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100. Medical Teacher, 1–15. https://doi.org/10.3109/0142159X.2015.1072619

Paudyal, V., Gunner, E., Bowen, M., Dehele, I., Shamim, A., Khan, K., & Fleming, G. (2021). National recruitment system for pre-registration pharmacist trainees in England and Wales: Context, process and outcomes. International Journal of Pharmacy Practice, 29(3), 285–290. https://doi.org/10.1093/ijpp/riab004

Petermann, M. K. H., & Zacher, H. (2022). Workforce Agility: development and validation of a multidimensional measure. Frontiers in Psychology, 13(March), 1–15. https://doi.org/10.3389/fpsyg.2022.841862

Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. In Developmental Review (Vol. 41, pp. 71–90). Mosby Inc. https://doi.org/10.1016/j.dr.2016.06.004

R Core Team. (2023). R: A Language and Environment for Statistical Computing. https://www.R-project.org/

Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02

Sala-Roca, J., Jariot-Garcia, M., Arnau-Sabatés, L., & Doval, E. (2021). Situational judgment test of basic employability competences development: Validation of a psychoeducative instrument for socioeducative intervention . Pedagogia Social, 37, 39–52. https://doi.org/10.7179/PSRI_2021.37.02

Shashi, Centobelli, P., Cerchione, R., & Ertz, M. (2020). Agile supply chain management: where did it come from and where will it go in the era of digital transformation? Industrial Marketing Management, 90(June), 324–345. https://doi.org/10.1016/j.indmarman.2020.07.011

Sherehiy, B. (2008). Relationships between agility strategy, work organization and workforce agility [Dissertation]. University of Louisville.

Sherehiy, B., & Karwowski, W. (2014). The relationship between work organization and workforce agility in small manufacturing enterprises. International Journal of Industrial Ergonomics, 44(3), 466–473. https://doi.org/10.1016/j.ergon.2014.01.002

Smith, K. J., Flaxman, C., Farland, M. Z., Thomas, A., Buring, S. M., Whalen, K., & Patterson, F. (2020). Development and validation of a situational judgement test to assess professionalism. American Journal of Pharmaceutical Education, 84(7), 985–992. https://doi.org/10.5688/ajpe7771

Sorrel, M. A., Olea, J., Abad, F. J., de la Torre, J., Aguado, D., & Lievens, F. (2016). Validity and Reliability of Situational Judgement Test Scores: A New Approach Based on Cognitive Diagnosis Models. Organizational Research Methods, 19(3), 506–532. https://doi.org/10.1177/1094428116630065

Storme, M., Suleyman, O., Gutlib, M., & Lubart, T. (2020). Who is agile? an investigation of the psychological antecedents of workforce agility agile. Global Business and Organizational Excellence, 1–11. https://doi.org/10.1002/joe.22055

Tarafdar, M., & Qrunfleh, S. (2016). Agile supply chain strategy and supply chain performance: complementary roles of supply chain practices and information systems capability for agility. International Journal of Production Research, 55(4), 925–938. https://doi.org/10.1080/00207543.2016.1203079

Tiffin, P. A., Paton, L. W., O’Mara, D., MacCann, C., Lang, J. W. B., & Lievens, F. (2020). Situational judgement tests for selection: traditional vs construct-driven approaches. Medical Education, 54(2), 105–115. https://doi.org/10.1111/medu.14011

Wang, J., & Wang, X. (2020). Structural equation modeling: applications using Mplus (D. J. Balding, N. A. C. Cressie, G. Fitzmaurice, & H. Goldstein (eds.); 2nd ed.). John Wiley & Sons. https://doi.org/10.1002/9781119422730

Zhang, Z., & Yuan, K. H. (2016). Robust coefficients alpha and omega and confidence intervals with outlying observations and missing data: methods and software. Educational and Psychological Measurement, 76(3), 387–411. https://doi.org/10.1177/0013164415594658

Full Text: PDF

DOI: 10.15408/jp3i.v14i1.39333