Teachers’ Attitudes Toward Generative AI In Assessment Planning: A UTAUT-Based Structural Equation Model
Article Sidebar
Main Article Content
Abstract
The quick rise of generative AI, particularly in educational settings, brings challenges to instructional practices; however, its impact on teachers' attitudes, especially in assessment planning, is still largely unexamined. This study uses the Unified Theory of Acceptance and Use of Technology (UTAUT) framework to explore teachers' attitudes towards generative AI in assessment planning with a focus on key constructs such as performance expectancy (PE), effort expectancy (EE), and social influence (SI), aimed at supporting more effective integration of these technologies in assessment planning. The study collected data from 419 educators and used the Partial Least Squares-Structural Equation Modeling (PLS-SEM). The findings show that performance expectancy significantly affects opinions (β = 0.392, t = 7.122, p < 0.001), indicating that teachers who believe AI will be helpful are likelier to use it. Similarly, effort expectancy (EE) strongly influences attitudes (β = 0.319, t = 5.528, p < 0.001), indicating the significance of ease-of-use beliefs in order for teachers to use generative AI in their assessment planning. Although social influence had a lesser impact (β = 0.133, t = 2.589, p = 0.01), it is still considered significant. These insights stress the importance of providing targeted professional development among teachers to improve their acceptance and implementation of generative AI in assessment planning.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright © 2024 APJETPS, its licensors, and contributors. All rights are reserved. For all open access content, the Creative Commons (CC) Attribution 4.0 International Public License terms apply.
References
Adiguzel, T., Kaya, M., & Cansu, F. (2023). Revolutionizing education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3), 1-19. https://doi.org/10.30935/cedtech/13152
Alamer, A., & Marsh, H. W. (2022). Evaluating the fit of structural equation models in educational research. British Journal of Educational Psychology, 92(2), 257–275. https://doi.org/10.1111/bjep.12437
Alzahrani, L. (2023). Analyzing students’ attitudes and behavior toward artificial intelligence technologies in higher education. International Journal of Recent Technology and Engineering (IJRTE). https://doi.org/10.35940/ijrte.f7475.0311623
Arning, K., Ziefle, M. (2009). Different Perspectives on Technology Acceptance: The Role of Technology Type and Age. In: Holzinger, A., Miesenberger, K. (eds) HCI and Usability for e-Inclusion. USAB 2009. Lecture Notes in Computer Science, vol 5889. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-10308-7_2
Bagozzi, R. P., & Yi, Y. (1988). On the evaluation of structural equation models. Journal of the Academy of Marketing Science, 16, 74– 94.
Becker, J., Ringle, C., Sarstedt, M., & Völckner, F. (2015). How collinearity affects mixture regression results. Marketing Letters, 26(4), 643-659. https://doi.org/10.1007/S11002-014-9299-9
Berg, G., & Plessis, E. (2023). ChatGPT and generative AI: Possibilities for contributing to lesson planning, critical thinking, and openness in teacher education. Education Sciences, 13(10), 998. https://doi.org/10.3390/educsci13100998
Bervell, B., Nyagorme, P., & Arkorful, V. (2020). LMS-enabled blended learning use intentions among distance education tutors: Examining the mediation role of attitude based on a technology-related stimulus-response theoretical framework (TR-SR-TF). Contemporary Educational Technology, 12(1), 1–25. https://doi.org/10.30935/cedtech/8317
Bevans, R. (2020, July 16). Understanding p-values | Definition and examples. Scribbr. https://www.scribbr.com
Buabeng-Andoh, C., & Baah, C. (2020). Pre-service teachers' intention to use learning management system: An integration of UTAUT and TAM. Interactive Technology and Smart Education, 17(4), 455-474. https://doi.org/10.1108/itse-02- 2020-0028
Chen, K., & Chang, M. (2013). User acceptance of ‘near field communication’ mobile phone service: An investigation based on the ‘unified theory of acceptance and use of technology’ model. The Service Industries Journal, 33(6), 609-623. https://doi.org/10.1080/02642069.2011.622369
Cheung, G. W., Cooper-Thomas, H. D., Lau, R. S., et al. (2024). Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations. Asia Pacific Journal of Management, 41, 745–783. https://doi.org/10.1007/s10490-023-09871-y
Cialdini, R. B., & Goldstein, N. J. (2004). Social influence: Compliance and conformity. Annual Review of Psychology, 55, 591– 621. https://doi.org/10.1146/annurev.psych.55.090901.102040
Dai, M., Patel, S., Song, J., Jin, W., Green, C., & Khera, M. (2023). MP36-19 Machine-learning based model for prediction of curvature improvement following CCH injections for Peyronie's disease. The Journal of Urology, 209, e487. https://doi.org/10.1097/JU.0000000000003270.19
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35(8), 982–1003. https://doi.org/10.1287/mnsc.35.8.982
Dwivedi, Y., Rana, N., Jeyaraj, A., Clement, M., & Williams, M. (2017). Re-examining the unified theory of acceptance and use of technology (UTAUT): Towards a revised theoretical model. Information Systems Frontiers, 21(3), 719–734. https://doi.org/10.1007/s10796-017-9774-y
Everitt, B., & Skrondal, A. (2010). The Cambridge Dictionary of Statistics (4th ed.). Cambridge University Press.
Fadda, D., & Morin, A. (2017). A systematic review of structural equation modeling in educational research: What do we know and need to know? Educational Psychology Review, 29(3), 493-516. https://doi.org/10.1007/s10648-016-9407-6
Farrell, A. M. (2021). The role of confirmatory factor analysis in assessing the dimensionality of constructs in educational research. Educational and Psychological Measurement, 81(3), 371–390. https://doi.org/10.1177/0013164420940242
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). GPower 3.1 manual* [Manual]. Heinrich-Heine-Universität Düsseldorf. https://www.psychologie.hhu.de/fileadmin/redaktion/Fakultaeten/Mathematisch- Naturwissenschaftliche_Fakultaet/Psychologie/AAP/gpower/GPowerManual.pdf
Fornell, C., & Larcker, D. F. (1981). Structural equation models with unobservable variables and measurement error: Algebra and statistics. Journal of Marketing Research, 18(3), 382-388. http://dx.doi.org/10.2307/3150980
Fu’adin, A., Bila, S., Saidah, Z., & Hidayat, Z. (2023). Potret penggunaan Google Form sebagai media evaluasi pembelajaran SMA Al-Falah Bandung. Jurnal Pendidikan, Sains Dan Teknologi, 2(2). https://doi.org/10.47233/jpst.v2i2.788
Garone, A., Pynoo, B., Tondeur, J., Cocquyt, C., Vanslambrouck, S., Bruggeman, B., & Struyven, K. (2019). Clustering university teaching staff through UTAUT: Implications for accepting a new learning management system. British Journal of Educational Technology, 50(5), 2466-2483.
Gegenfurtner, A. (2022). Bifactor exploratory structural equation modeling: A meta-analytic review of model fit. Frontiers in Psychology, 13. https://doi.org/10.3389/fpsyg.2022.1037111
Gomes, C., & Farias, J. (2017). The influence of performance and effort expectancy on using a purchasing application in
Brazil. Contabilidade, Gestão e Governança, 20, 72-90. https://doi.org/10.21714/1984-3925_2017V20N1A5
González-Calatayud, V., Prendes-Espinosa, P., & Roig-Vila, R. (2021). Artificial intelligence for student assessment: A systematic review. Applied Sciences, 11(12), 5467. https://doi.org/10.3390/APP11125467
Hair Jr., J.F., et al. (2014). Partial Least Squares Structural Equation Modeling (PLS-SEM): An Emerging Tool in Business Research. European Business Review, pp. 26, 106–121. https://doi.org/10.1108/EBR-10-2013-0128
Hair, J. F. Jr., Hult, G. T. M., Ringle, C. M., Sarstedt, M., Danks, N. P., & Ray, S. (2021). Partial least squares structural equation modeling (PLS-SEM) using R. Springer. https://doi.org/10.1007/978-3-030-80519-7
Hair, J. F., Hult, G. T. M., Ringle, C. M., & Sarstedt, M. (2022). A primer on partial least squares structural equation modeling (PLS- SEM) (3rd ed.). SAGE Publications.
Hair, J.F., Hult, G.T.M., Ringle, C.M. and Sarstedt, M. (2017) A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM). 2nd Edition, Sage Publications Inc., Thousand Oaks, CA. https://www.scirp.org/reference/referencespapers?referenceid=2297757
Hair, J.F., Risher, J.J., Sarstedt, M. and Ringle, C.M. (2019) When to Use and How to Report the Results of PLS-SEM. European Business Review, 31, 2-24. https://doi.org/10.1108/EBR-11-2018-0203
Hamid, M. R. A., Sami, W., & Sidek, M. H. M. (2017). Discriminant Validity Assessment: Use of Fornell & Larcker criterion versus HTMT Criterion. Journal of Physics: Conference Series, 890, 012163. https://doi. org/10.1088/1742- 6596/890/1/012163
Hayes, A., & Coutts, J. (2020). Use Omega rather than Cronbach’s Alpha for estimating reliability. But… Communication Methods and Measures, 14, 1-24. https://doi.org/10.1080/19312458.2020.1718629
Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43(1), 115-135. https://doi.org/10.1007/s11747-014-0403-8
Henseler, J., Ringle, C. M., & Sinkovics, R. R. (2015). The use of partial least squares path modeling in international marketing. Advances in International Marketing, 26, 277–319. https://doi.org/10.1108/S1474-797920150000026014
Hopfenbeck, T., Zhang, Z., Sun, S., Robertson, P., & McGrane, J. (2023). Challenges and opportunities for classroom-based formative assessment and AI: A perspective article. Frontiers in Education. https://doi.org/10.3389/feduc.2023.1270700
Ikwuka, O. I., Onyali, L. C., Olugbemi, O. P., Etodike, C. E., Igbokwe, I. C., & Adigwe, E. J. (2020). Teachers’ attitude towards using ICT for quality instructional delivery in Onitsha North secondary schools, Anambra State, Nigeria. International Journal of Academic Research in Progressive Education & Development, 9(3), 1-11. https://doi.org/10.6007/IJARPED/v9-i3/7980
Im, I., Hong, S., & Kang, M. (2011). An international comparison of technology adoption: Testing the UTAUT model. Information Management, 48(1), 1-8. https://doi.org/10.1016/j.im.2010.09.001
Jauhiainen, J., & Garagorry, A. (2023). Generative AI and ChatGPT in school children’s education: Evidence from a school lesson. Sustainability, 15, Article 14025. https://doi.org/10.3390/su151814025
Jöreskog, Karl. (1971). Statistical Analysis of Sets of Congeneric Tests. Psychometrika. 36. 109-133. 10.1007/BF02291393.
Kim, J. (2019). Multicollinearity and misleading statistical results. Korean Journal of Anesthesiology, 72, 558-569. https://doi.org/10.4097/kja.19087
Kim, J. M. (2017). Study on intention and attitude of using artificial intelligence technology in healthcare. Journal of Convergence for Information Technology, 7(4), 53-60. https://doi.org/10.22156/CS4SMB.2017.7.4.053
Kline, R. B. (2016). Principles and practice of structural equation modeling (4th ed.). Guilford Press.
Kovalchuk, V. (2013). ICT tools application in the pedagogical experiment: Online questionnaire. https://doi.org/10.33407/itlt.v35i3.832
Lombardi, L., & Pastore, M. (2012). Sensitivity of fit indices to fake perturbation of ordinal data: A sample by replacement approach. Multivariate Behavioral Research, 47, 519-546. https://doi.org/10.1080/00273171.2012.692616
Malazzab, B. N., et al. (2024). Analyzing university students’ attitudes and behavior toward AI using the extended unified theory of acceptance and use of technology model. American Journal of Applied Statistics and Economics, 3(1), 99–108. https://doi.org/10.54536/ajase.v3i1.2510
Marikyan, D., & Papagiannidis, S. (2023). The role of AI in higher education: A framework for understanding its impact on teaching and learning. Education and Information Technologies. https://doi.org/10.1007/s10639-023-11102-2
Marsh, H. W., Morin, A., Parker, P., & Kaur, G. (2014). Exploratory structural equation modeling: Integrating the best exploratory and confirmatory factor analysis features. Annual Review of Clinical Psychology, pp. 10, 85–110. https://doi.org/10.1146/annurev-clinpsy-032813-153700
Mason, C. H., & Perreault, W. D. (1991). Collinearity, power, and interpretation of multiple regression analysis. Journal of Marketing Research, 28(3), 268–280. https://doi.org/10.2307/3172863
Mondal, H., Mondal, S., Ghosal, T., & Mondal, S. (2019). Using Google Forms for Medical Survey: A Technical Note. International Journal of Clinical and Experimental Physiology, 5, 216–218. https://doi.org/10.5530/IJCEP.2018.5.4.26
Nakagawa, S., & Schielzeth, H. (2013). A general and simple method for obtaining R² from generalized linear mixed‐effects models. Methods in Ecology and Evolution, 4, 133-142. https://doi.org/10.1111/j.2041-210x.2012.00261.x
Nazaretsky, T., Ariely, M., Cukurova, M., & Alexandron, G. (2022). Teachers' trust in AI-powered educational technology and a professional development program to improve it. British Journal of Educational Technology, 53(3), 914-931. https://doi.org/10.1111/bjet.13232
Obenza, B. N., Baguio, J. S. I. E., Bardago, K. M. W., Granado, L. B., Loreco, K. C. A., Matugas, L. P., Talaboc, D. J., Zayas, R. K. D. D., Caballo, J. H. S., & Caangay, R. B. R. (2023). The Mediating Effect of AI Trust on AI Self-Efficacy and Attitude Toward AI of College Students. International Journal of Metaverse, 2(1), 1–10. https://doi.org/10.54536/ijm.v2i1.2286
Obenza, B. N., Caballo, J. H. S., Caangay, R. B. R., Makigod, T. E. C., Almocera, S. M., Bayno, J. L. M., Camposano, J. R. Jr., Cena, S. J. G., Garcia, J. A. K., Labajo, B. F. M., & Tua, A. G. (2023). Analyzing university students’ attitudes and behavior toward AI using the extended Unified Theory of Acceptance and Use of Technology model. Asian Journal of Assessment in Teaching and Learning, 3(1), 1–14. https://doi.org/10.54536/ajase.v3i1.2510
Obenza, B. N., Go, L. E., Francisco, J. A. M., Buit, E. E. T., Mariano, F. V. B., Cuizon Jr, H. L., Cagabhion, A. J. D., & Agbulos, K. A. J. L. (2024). The Nexus between Cognitive Absorption and AI Literacy of College Students as Moderated by Sex. American Journal of Smart Technology and Solutions, 3(1), 32–39. https://doi.org/10.54536/ajsts.v3i1.2603
Obenza, B. N., Salvahan, A., Rios, A. N., Solo, A., Alburo, R. A., & Gabila, R. J. (2023). University Students’ Perception and Use of ChatGPT: Generative Artificial Intelligence (AI) in Higher Education. International Journal of Human Computing Studies, 5(12), 5-18. https://doi.org/10.31149/ijhcs.v5i12.5033
Owan, V., Abang, K., Idika, D., & Bassey, B. (2023). Exploring the potential of artificial intelligence tools in educational measurement and assessment. Eurasia Journal of Mathematics, Science and Technology Education, 19, Article ID em2307. https://doi.org/10.29333/ejmste/13428
Parsons, V. (2017). Stratified sampling. In Wiley StatsRef: Statistics Reference Online. https://doi.org/10.1002/9781118445112.stat05999.pub2
Petras, N., & Meiser, T. (2023). Problems of domain factors with small factor loadings in bi-factor models. Multivariate Behavioral Research, 1-25. https://doi.org/10.1080/00273171.2023.2228757
Przymuszała, P., et al. (2021). Affective and cognitive components of students' attitudes towards communication learning. BMC Medical Education, 21, 190. https://doi.org/10.1186/s12909-021-02626-7
Raykov, T., Anthony, J., & Menold, N. (2022). On the importance of coefficient alpha for measurement research: Loading equality is not necessary for alpha’s utility as a scale reliability index. Educational and Psychological Measurement, 83, 766- 781. https://doi.org/10.1177/00131644221104972
Sánchez-Prieto, J., Cruz-Benito, J., Therón, R., & García-Peñalvo, F. (2019). How do we measure teachers' acceptance of technology? A systematic review of technology acceptance literature in education. Computers in Human Behavior, 95, 73-84. https://doi.org/10.1016/j.chb.2019.01.027
Sarstedt, M., Ringle, C., & Hair, J. (2017). Partial least squares structural equation modeling. In Handbook of market research (pp. 1-40). Springer. https://doi.org/10.1007/978-3-319-05542-8_15-1
Sharples, M. (2023). Towards social generative AI for education: Theory, practices, and ethics. Learning: Research and Practice, 9, 159-167. https://doi.org/10.1080/23735082.2023.2261131
Smolansky, A., Cram, A., Raduescu, C., Zeivots, S., Huber, E., & Kizilcec, R. (2023). Educator and student perspectives on the impact of generative AI on assessments in higher education. Proceedings of the Tenth ACM Conference on Learning @ Scale. https://doi.org/10.1145/3573051.3596191
Taber, Keith. (2018). The Use of Cronbach’s Alpha When Developing and Reporting Research Instruments in Science Education. Research in Science Education. 48. 1-24. 10.1007/s11165-016-9602-2.
Teo, T. (2011). Factors influencing teachers' intention to use technology: Model development and test. Computers & Education, 57(4), 2432–2440.
Tóth, G., Bodai, Z., & Héberger, K. (2013). Estimation of influential points in any data set from coefficient of determination and its leave-one-out cross-validated counterpart. Journal of Computer-Aided Molecular Design, 27, 837-844. https://doi.org/10.1007/s10822-013-9680-4
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. https://doi.org/10.2307/30036540
Wu, X., Wang, L., & Li, Y. (2022). The impact of educational environment on teachers' acceptance of educational technology: Evidence from China. Education and Information Technologies, 27(1), 469-488. https://doi.org/10.1007/s10639-021- 10584-2
Xia, Y., & Yang, Y. (2018). The influence of number of categories and threshold values on fit indices in structural equation modeling with ordered categorical data. Multivariate Behavioral Research, 53, 731-755. https://doi.org/10.1080/00273171.2018.1480346
Yılmaz, R., & Karaoğlan Yılmaz, F. G. (2023). Augmented intelligence in programming learning: Examining student views on using ChatGPT for programming learning. Computers in Human Behavior: Artificial Humans, 1, Article 100005. https://doi.org/10.1016/j.chbah.2023.100005
Zha, Q. (2003). Internationalization of higher education: Towards a conceptual framework. Policy Futures in Education, 1(2), 248– 270. https://doi.org/10.2304/pfie.2003.1.2.5