Supervisors: Dr. J. Mostow

I worked with Dr. Jack Mostow on Relating Student Behaviour with their Facial Expressions in RoboTutor. RoboTutor is an open-source Android tablet app that enables children aged between 7-10 to learn basic reading, writing, and arithmetic without adult assistance. %It is also one of five $1M Finalists in the $15M Global Learning XPrize Competition.

As part of this project, we developed a real-time emotion recognition model for pedagogically relevant emotions using facial action units for conducting an offline analysis relating kids’ automatically recognised facial expressions to their logged behavior in RoboTutor. We were able to relate the occurrence of the frustration emotion with the action of the child exiting the application prematurely and the delight emotion to the action of the child submitting the correct answer during the game. I extended my software to the Android platform so that it could be integrated into RoboTutor, allowing the application to recognize the emotion displayed by the child in real-time and predict his/her corresponding behavior action.

This contribution led to a first author accepted paper submission to the Student Abstract and Poster Program at the 32nd Conference of Artificial Intelligence (AAAI) 2018.

View Report