Prediction of Student Engagement Using Deep Learning-Based Student Face Expression Detection Method (DL-SFEDM)
Abstract
Nowadays, Academics and teachers have paid a lot of attention to computational thinking (CT) because of the wonderful opportunities it presents for developing students' problem-solving abilities, which are in high demand in a technologically advanced world. However, research has shown that educators lack a solid grasp of CT and often misunderstand its idea, which might impede their implementation of these initiatives. This problem is even worse because very little research has explored ways to engage students with their learning. In this paper, a Deep Learning-Based Student Face Expression Detection Method (DL-SFEDM) is a method for finding out how engaged students are with online lecture videos that don't depend on data generated by educational management systems. To conduct the classroom engagement analysis, the whole class is treated as a single group and their corresponding group engagement score is calculated. Emotions play a crucial part in learning. Computer vision (CV)-based approaches evaluate online and offline lecture recordings and extract students' emotions. This expressive emotion analysis looks at how Students feel in four distinct states: positive, negative, neutral, and negative. The experimental results show that the proposed DL-SFEDM enhances the teaching technique, motivates better student learning and increases student computational thinking.
References
S. Gupta, P. Kumar, and R. K. Tekchandani, "Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models," Multimedia Tools and Applications, vol. 82, no. 8, pp. 11365–11394, 2023. doi: 10.1007/s11042-022-13558-9.
P. Bhardwaj, P. K. Gupta, H. Panwar, M. K. Siddiqui, R. Morales-Menendez, and A. Bhaik, "Application of deep learning on student engagement in e-learning environments," Computers & Electrical Engineering, vol. 93, p. 107277, 2021. doi: 10.1016/j.compeleceng.2021.107277.
X. Tian, B. P. Nunes, Y. Liu, and R. Manrique, "Predicting student engagement using sequential ensemble model," IEEE Transactions on Learning Technologies, 2023. doi: 10.1109/TLT.2023.3342860.
N. Xie, Z. Liu, Z. Li, W. Pang, and B. Lu, "Student engagement detection in online environment using computer vision and multi-dimensional feature fusion," Multimedia Systems, vol. 29, no. 6, pp. 3559–3577, 2023. doi: 10.1007/s00530-023-01153-3.
B. Abdellaoui, A. Moumen, Y. E. B. El Idrissi, and A. Remaida, "Face detection to recognize students’ emotion and their engagement: A systematic review," in Proceedings of the IEEE 2nd International Conference on Electronics, Control, Optimization and Computer Science (ICECOCS), Kenitra, Morocco, Dec. 2020, pp. 1–6. doi: 10.1109/ICECOCS50124.2020.9314600.
X. Li, R. Yue, W. Jia, H. Wang, and Y. Zheng, "Recognizing students' emotions based on facial expression analysis," in Proceedings of the 11th International Conference on Information Technology in Medicine and Education (ITME), Fuzhou, China, Nov. 2021, pp. 96–100. doi: 10.1109/ITME53901.2021.00030.
N. Ahmad, A. Gupta, and D. Singh, "Using deep transfer learning to predict student engagement in online courses," in Proceedings of the International Conference on Machine Learning, Image Processing, Network Security and Data Sciences, Cham, Switzerland, Dec. 2022, pp. 27–36. Springer. doi: 10.1007/978-3-031-24367-7_3.
N. S. Nuha, T. Mahmud, N. Rezaoana, M. S. Hossain, and K. Andersson, "An approach of analyzing classroom student engagement in multimodal environment by using deep learning," in Proceedings of the IEEE 9th International Women in Engineering Conference on Electrical and Computer Engineering (WIECON-ECE), Dhaka, Bangladesh, Nov. 2023, pp. 286–291. doi: 10.1109/WIECON-ECE60392.2023.10456488.
X. Ma, M. Xu, Y. Dong, and Z. Sun, "Automatic student engagement in online learning environment based on neural turing machine," International Journal of Information and Education Technology, vol. 10.18178/ijiet.2021.11.3.1497. 11, no. 3, pp. 107–111, 2021. doi:
D. Dresvyanskiy, W. Minker, and A. Karpov, "Deep learning-based engagement recognition in highly imbalanced data," in Proceedings of the 23rd International Conference on Speech and Computer (SPECOM), St. Petersburg, Russia, Sep. 2021, pp. 166–178. doi: 10.1007/978-3-030-87802-3_16.
S. G. Khenkar et al., "Deep analysis of student body activities to detect engagement state in E-learning sessions," Applied Sciences, vol. 13, no. 4, p. 2591, 2023. doi: 10.3390/app13042591.
R. Abdulkader et al., "Optimizing student engagement in edge-based online learning with advanced analytics," Array, vol. 19, p. 100301, 2023. doi: 10.1016/j.array.2023.100301.
Y. Wang, A. Kotha, P. H. Hong, and M. Qiu, "Automated student engagement monitoring and evaluation during learning in the wild," in Proceedings of the IEEE 7th International Conference on Cyber Security and Cloud Computing (CSCloud) / IEEE 6th International Conference on Edge Computing and Scalable Cloud (EdgeCom), 2020, pp. 270–275. doi: 10.1109/CSCloud-EdgeCom49738.2020.00054.
H. J. Lee and D. Lee, "Study of process-focused assessment using an algorithm for facial expression recognition based on a deep neural network model," Electronics, vol. 10, no. 1, p. 54, 2020. doi: 10.3390/electronics10010054.
X. Zheng, S. Hasegawa, M. T. Tran, K. Ota, and T. Unoki, "Estimation of learners’ engagement using face and body features by transfer learning," in Proceedings of the International Conference on Human-Computer Interaction, Cham, Switzerland, Jul. 2021, pp. 541–552. doi: 10.1007/978-3-030-77772-2_36.
C. Pabba and P. Kumar, "An intelligent system for monitoring students' engagement in large classroom teaching through facial expression recognition," Expert Systems, vol. 39, no. 1, p. e12839, 2022. doi: 10.1111/exsy.12839.
V. Savchenko, L. V. Savchenko, and I. Makarov, "Classifying emotions and engagement in online learning based on a single facial expression recognition neural network," IEEE Transactions on Affective Computing, vol. 13, no. 4, pp. 2132–2143, 2022. doi: 10.1109/TAFFC.2022.3188390.
A. Dubbaka and A. Gopalan, "Detecting learner engagement in MOOCs using automatic facial expression recognition," in Proceedings of the IEEE Global Engineering Education Conference (EDUCON), Porto, Portugal, Apr. 2020, pp. 447–456. doi: 10.1109/EDUCON45650.2020.9125149.
O. Mohamad Nezami, M. Dras, L. Hamey, D. Richards, S. Wan, and C. Paris, "Automatic recognition of student engagement using deep learning and facial expression," in Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Cham, Switzerland, 2020, pp. 273–289. doi: 10.1007/978-3-03046133-1_17.
Z. Zhang, Z. Li, H. Liu, T. Cao, and S. Liu, "Data-driven online learning engagement detection via facial expression and mouse behavior recognition technology," Journal of Educational Computing Research, vol. 58, no. 1, pp. 63–86, 2020. doi: 10.1177/0735633119825575.
R. B. R. Maddu and S. Murugappan, "Online learners’ engagement detection via facial emotion recognition in online learning context using hybrid classification model," Social Network Analysis and Mining, vol. 14, no. 1, p. 43, 2024. doi: 10.1007/s13278-02301181-x. deep
A. Sukumaran and A. Manoharan, "Multimodal engagement recognition from image traits using learning 10.1109/ACCESS.2024.3353053. techniques," IEEE Access, 2024. doi:
B. Rajpal, N. Prasad, K. K. Rao Mangalore, N. Pradeep, and R. Shastri, "A modified technique for recognizing facial expression," Journal of Applied Research on Industrial Engineering, vol. 7, no. 4, pp. 424–434, 2020. doi: 10.22105/jarie.2020.259293.1216.
M. Karahan, F. Lacinkaya, K. Erdonmez, E. D. Eminagaoglu, and C. Kasnakoglu, "Age and gender classification from facial features and object detection with machine learning," Journal of Fuzzy Extension and Applications, vol. 3, no. 3, pp. 219–230, 2022. doi: 10.22105/jfea.2022.328472.1201.
R. Safa, S. A. Edalatpanah, and A. Sorourkhah, "Predicting mental health using social media: A roadmap for future development," in Deep Learning in Personalized Healthcare and Decision Support, London, U.K.: Academic Press, 2023, pp. 285–303. doi: 10.1016/B978-0-443-19413-9.00014-X.
N. Alruwais and M. Zakariah, "Student recognition and activity monitoring in e-classes using deep learning in higher education," IEEE Access, 2024. doi: 10.1109/ACCESS.2024.3354981.
F. Mirsaeedi, H. Koosha, and M. Ghodoosi, "Evaluation of data mining algorithms on educational data using multi-criteria decision-making methods," Journal of Decisions and Operations Research, vol. 10.22105/dmor.2021.239599.1182. 6, no. 1, pp. 41–55, 2021. doi:
S. Mohammadi, N. Hemati, and N. Mohammadi, "Speech recognition system based on machine learning in Persian language," Computational Algorithms and Numerical Dimensions, vol. 1, no. 2, pp. 72–83, 2022. doi: 10.22105/cand.2022.146462.
P. Moradipour, S. A. Edalatpanah, and A. Sorourkhah, "Identifying the factors affecting students' satisfaction and dissatisfaction with the university (case study: postgraduate management students of a higher education institute)," Modern Research in Performance Evaluation, vol. 2, no. 3, pp. 202–213, 2023. doi: 10.22105/mrpe.2023.185975.
A. Panda and A. Shemshad, "Automated class student counting through image processing," Big Data and Computing Visions, vol. 1, no. 1, pp. 24–29, 2021. doi: 10.22105/bdcv.2021.142509.
S. H. Chabok, "Designing management methods in elementary school by creating enthusiasm to promote learning in students," Innovation Management and Operational Strategies, vol. 2, no. 3, pp. 244–256, 2021. doi: 10.22105/imos.2021.301027.1160.
Face Expression Recognition Dataset, Kaggle. [Online]. Available: https://www.kaggle.com/datasets/jonathanoheix/face-expression-recognition-dataset.
P. Bhardwaj, P. K. Gupta, H. Panwar, M. K. Siddiqui, R. Morales-Menendez, and A. Bhaik, "Application of deep learning on student engagement in e-learning environments," Computers & Electrical Engineering, vol. 93, p. 107277, 2021. doi: 10.1016/j.compeleceng.2021.107277.
X. Tian, B. P. Nunes, Y. Liu, and R. Manrique, "Predicting student engagement using sequential ensemble model," IEEE Transactions on Learning Technologies, 2023. doi: 10.1109/TLT.2023.3342860.
N. Xie, Z. Liu, Z. Li, W. Pang, and B. Lu, "Student engagement detection in online environment using computer vision and multi-dimensional feature fusion," Multimedia Systems, vol. 29, no. 6, pp. 3559–3577, 2023. doi: 10.1007/s00530-023-01153-3.
B. Abdellaoui, A. Moumen, Y. E. B. El Idrissi, and A. Remaida, "Face detection to recognize students’ emotion and their engagement: A systematic review," in Proceedings of the IEEE 2nd International Conference on Electronics, Control, Optimization and Computer Science (ICECOCS), Kenitra, Morocco, Dec. 2020, pp. 1–6. doi: 10.1109/ICECOCS50124.2020.9314600.
X. Li, R. Yue, W. Jia, H. Wang, and Y. Zheng, "Recognizing students' emotions based on facial expression analysis," in Proceedings of the 11th International Conference on Information Technology in Medicine and Education (ITME), Fuzhou, China, Nov. 2021, pp. 96–100. doi: 10.1109/ITME53901.2021.00030.
N. Ahmad, A. Gupta, and D. Singh, "Using deep transfer learning to predict student engagement in online courses," in Proceedings of the International Conference on Machine Learning, Image Processing, Network Security and Data Sciences, Cham, Switzerland, Dec. 2022, pp. 27–36. Springer. doi: 10.1007/978-3-031-24367-7_3.
N. S. Nuha, T. Mahmud, N. Rezaoana, M. S. Hossain, and K. Andersson, "An approach of analyzing classroom student engagement in multimodal environment by using deep learning," in Proceedings of the IEEE 9th International Women in Engineering Conference on Electrical and Computer Engineering (WIECON-ECE), Dhaka, Bangladesh, Nov. 2023, pp. 286–291. doi: 10.1109/WIECON-ECE60392.2023.10456488.
X. Ma, M. Xu, Y. Dong, and Z. Sun, "Automatic student engagement in online learning environment based on neural turing machine," International Journal of Information and Education Technology, vol. 10.18178/ijiet.2021.11.3.1497. 11, no. 3, pp. 107–111, 2021. doi:
D. Dresvyanskiy, W. Minker, and A. Karpov, "Deep learning-based engagement recognition in highly imbalanced data," in Proceedings of the 23rd International Conference on Speech and Computer (SPECOM), St. Petersburg, Russia, Sep. 2021, pp. 166–178. doi: 10.1007/978-3-030-87802-3_16.
S. G. Khenkar et al., "Deep analysis of student body activities to detect engagement state in E-learning sessions," Applied Sciences, vol. 13, no. 4, p. 2591, 2023. doi: 10.3390/app13042591.
R. Abdulkader et al., "Optimizing student engagement in edge-based online learning with advanced analytics," Array, vol. 19, p. 100301, 2023. doi: 10.1016/j.array.2023.100301.
Y. Wang, A. Kotha, P. H. Hong, and M. Qiu, "Automated student engagement monitoring and evaluation during learning in the wild," in Proceedings of the IEEE 7th International Conference on Cyber Security and Cloud Computing (CSCloud) / IEEE 6th International Conference on Edge Computing and Scalable Cloud (EdgeCom), 2020, pp. 270–275. doi: 10.1109/CSCloud-EdgeCom49738.2020.00054.
H. J. Lee and D. Lee, "Study of process-focused assessment using an algorithm for facial expression recognition based on a deep neural network model," Electronics, vol. 10, no. 1, p. 54, 2020. doi: 10.3390/electronics10010054.
X. Zheng, S. Hasegawa, M. T. Tran, K. Ota, and T. Unoki, "Estimation of learners’ engagement using face and body features by transfer learning," in Proceedings of the International Conference on Human-Computer Interaction, Cham, Switzerland, Jul. 2021, pp. 541–552. doi: 10.1007/978-3-030-77772-2_36.
C. Pabba and P. Kumar, "An intelligent system for monitoring students' engagement in large classroom teaching through facial expression recognition," Expert Systems, vol. 39, no. 1, p. e12839, 2022. doi: 10.1111/exsy.12839.
V. Savchenko, L. V. Savchenko, and I. Makarov, "Classifying emotions and engagement in online learning based on a single facial expression recognition neural network," IEEE Transactions on Affective Computing, vol. 13, no. 4, pp. 2132–2143, 2022. doi: 10.1109/TAFFC.2022.3188390.
A. Dubbaka and A. Gopalan, "Detecting learner engagement in MOOCs using automatic facial expression recognition," in Proceedings of the IEEE Global Engineering Education Conference (EDUCON), Porto, Portugal, Apr. 2020, pp. 447–456. doi: 10.1109/EDUCON45650.2020.9125149.
O. Mohamad Nezami, M. Dras, L. Hamey, D. Richards, S. Wan, and C. Paris, "Automatic recognition of student engagement using deep learning and facial expression," in Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Cham, Switzerland, 2020, pp. 273–289. doi: 10.1007/978-3-03046133-1_17.
Z. Zhang, Z. Li, H. Liu, T. Cao, and S. Liu, "Data-driven online learning engagement detection via facial expression and mouse behavior recognition technology," Journal of Educational Computing Research, vol. 58, no. 1, pp. 63–86, 2020. doi: 10.1177/0735633119825575.
R. B. R. Maddu and S. Murugappan, "Online learners’ engagement detection via facial emotion recognition in online learning context using hybrid classification model," Social Network Analysis and Mining, vol. 14, no. 1, p. 43, 2024. doi: 10.1007/s13278-02301181-x. deep
A. Sukumaran and A. Manoharan, "Multimodal engagement recognition from image traits using learning 10.1109/ACCESS.2024.3353053. techniques," IEEE Access, 2024. doi:
B. Rajpal, N. Prasad, K. K. Rao Mangalore, N. Pradeep, and R. Shastri, "A modified technique for recognizing facial expression," Journal of Applied Research on Industrial Engineering, vol. 7, no. 4, pp. 424–434, 2020. doi: 10.22105/jarie.2020.259293.1216.
M. Karahan, F. Lacinkaya, K. Erdonmez, E. D. Eminagaoglu, and C. Kasnakoglu, "Age and gender classification from facial features and object detection with machine learning," Journal of Fuzzy Extension and Applications, vol. 3, no. 3, pp. 219–230, 2022. doi: 10.22105/jfea.2022.328472.1201.
R. Safa, S. A. Edalatpanah, and A. Sorourkhah, "Predicting mental health using social media: A roadmap for future development," in Deep Learning in Personalized Healthcare and Decision Support, London, U.K.: Academic Press, 2023, pp. 285–303. doi: 10.1016/B978-0-443-19413-9.00014-X.
N. Alruwais and M. Zakariah, "Student recognition and activity monitoring in e-classes using deep learning in higher education," IEEE Access, 2024. doi: 10.1109/ACCESS.2024.3354981.
F. Mirsaeedi, H. Koosha, and M. Ghodoosi, "Evaluation of data mining algorithms on educational data using multi-criteria decision-making methods," Journal of Decisions and Operations Research, vol. 10.22105/dmor.2021.239599.1182. 6, no. 1, pp. 41–55, 2021. doi:
S. Mohammadi, N. Hemati, and N. Mohammadi, "Speech recognition system based on machine learning in Persian language," Computational Algorithms and Numerical Dimensions, vol. 1, no. 2, pp. 72–83, 2022. doi: 10.22105/cand.2022.146462.
P. Moradipour, S. A. Edalatpanah, and A. Sorourkhah, "Identifying the factors affecting students' satisfaction and dissatisfaction with the university (case study: postgraduate management students of a higher education institute)," Modern Research in Performance Evaluation, vol. 2, no. 3, pp. 202–213, 2023. doi: 10.22105/mrpe.2023.185975.
A. Panda and A. Shemshad, "Automated class student counting through image processing," Big Data and Computing Visions, vol. 1, no. 1, pp. 24–29, 2021. doi: 10.22105/bdcv.2021.142509.
S. H. Chabok, "Designing management methods in elementary school by creating enthusiasm to promote learning in students," Innovation Management and Operational Strategies, vol. 2, no. 3, pp. 244–256, 2021. doi: 10.22105/imos.2021.301027.1160.
Face Expression Recognition Dataset, Kaggle. [Online]. Available: https://www.kaggle.com/datasets/jonathanoheix/face-expression-recognition-dataset.
Published
2025-07-23
How to Cite
SHAH, Chirag Vinalbhai et al.
Prediction of Student Engagement Using Deep Learning-Based Student Face Expression Detection Method (DL-SFEDM).
Yugoslav Journal of Operations Research, [S.l.], july 2025.
ISSN 2334-6043.
Available at: <https://yujor.fon.bg.ac.rs/index.php/yujor/article/view/1358>. Date accessed: 01 aug. 2025.
doi: https://doi.org/10.2298/YJOR240515007V.
Issue
Section
Research Articles
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
