Plenary Lecture
Facial Expression Recognition of Speaker Using Vowel Judgment and Features of Thermal Face Image
Professor Yasunari Yoshitomi
Information Communication System Lab
Graduate School of Life and Environmental Sciences
Kyoto Prefectural University
JAPAN
E-mail: yoshitomi@kpu.ac.jp
Abstract: To better integrate robots into our society, a robot should be able to interact in a friendly manner with humans. The goal of our research is to develop a robot that can perceive human feelings and mental states. For example, a robot could encourage a human who looks sad, advise a person to stop working and rest for a while when the individual looks tired, or take care of a person advanced in years. The present investigation concerns the first stage of the development of a robot that acquires the ability to detect human feeling or inner mental states visually. Although the mechanism for recognizing facial expressions of human feelings has received considerable attention in computer vision research, it currently falls far short of human capability. This is due to the decreased accuracy of facial expression recognition, which is influenced by the inevitable change of gray levels due to nuances of shade, reflection, and local darkness. To avoid this problem and to develop a robust method for facial expression recognition applicable under widely varied lighting conditions, we use an image registered by infrared rays to describe the thermal distribution of the face. The timing of recognizing facial expressions is also important for a robot because the processing can be time-consuming. We adopted an utterance as the key of expressing human feelings because humans tend to say something when expressing their feelings.
In this lecture, I introduce our method for facial expression recognition of a speaker. For facial expression recognition, we pick three images: (i) just before speaking, in speaking (ii) the first and (iii) the last vowels of an utterance. The face direction is estimated to select front-view faces as the targets of facial expression recognition using thermal image processing. A two-dimensional discrete cosine transformation is performed for transforming grayscale values on each block in focused face parts of an image into their frequency components, which are used for generating feature vectors. We use pattern recognition through our heuristic rules for facial expression recognition. In this method, the facial expressions are discriminable with good recognition accuracy when a person exhibits one of the intentional facial expressions of “angry”, “happy”, “neutral”, “sad”, and “surprise”.
We also proposed a method for efficiently updating training data, such that the training data of only the facial expressions of “happy” and “neutral” were updated after an interval such as approximately three and a half years. Using this proposed method, the facial expressions of two subjects were discriminable with good recognition accuracy for the facial expressions of “happy,” “neutral,” and “others” when they exhibited one of the intentional facial expressions of “angry,” “happy,” “neutral,” “sad,” and “surprised.” Based on our previously reported method, we proposed an on-line system for recognizing the facial expression of a speaker using front-view face judgment, vowel judgment, and thermal image processing. We expect the proposed system to be applicable for recognizing facial expressions in daily conversations.
Brief Biography of the Speaker: Yasunari Yoshitomi received his B.E., M.E. degrees and Ph.D. in Engineering from Kyoto University in 1980, 1982, and 1991, respectively. He had worked in Nippon Steel Corporation from 1982 to 1995 and had been engaged in image analysis application and development of soft magnetic materials. From 1995 to 2001, he had been in Miyazaki University as an associate professor at the Department of Computer Science and Systems Engineering. From 2001 to 2008, he had been in Kyoto Prefectural University as a professor at the Department of Environmental Informatics. Since 2008, he has been in Kyoto Prefectural University as a professor at the Environmental Information System Subdivision, Division of Environmental Sciences, Graduate School of Life and Environmental Sciences. He is a member of IEEE, IPSJ, IEICE, JSIAM, ORSJ, HIS, SSJ and IIEEJ. He received a Best Paper Award from IEEE International Workshop on Robot and Human Communication in 1998, and a Best Paper Award from IEEE International Workshop on Robot and Human Interactive Communication in 2000. He has published 132 papers, two reviews, 10 books, and 214 patents. He is an associate editor of Artificial Intelligence Research, and an executive editorial board member of the Journal of Robotics and Artificial Intelligence. He has been listed in the 2010, 2011, and 2012 editions of Marquis Who's Who in the World. His current research interests are communication between human and computer, media information processing, authentication on digital content, stochastic programming problem, simulation on emission trading of greenhouse effect gas, and system for watching infringement of human rights in cyber space.