Automatic facial expression recognition has attracted much attention since the early nineties, especially in human-computer interaction. As computers start becoming a part of our life, they need to become more and more intelligent. Expression recognition systems will enhance this intelligent interaction between the human and the computer.
Although humans can recognize facial expressions easily, a reliable expression recognition system is still a challenge. In this chapter, we will introduce a basic implementation of facial expression using various algorithms from the OpenCV library, including feature extraction and classification using the ml module.
In this chapter, we will be going through the following topics in brief:
Automatic facial expression recognition is an interesting and challenging problem and has several important applications in many areas such as human-computer interaction, human behavior understanding, and data-driven animation. Unlike face recognition, facial expression recognition needs to discriminate between the same expression in different individuals. The problem becomes more difficult as a person may show the same expression in different ways.
The current existing approaches for measuring facial expressions can be categorized into two types: static image and image sequence. In the static image approach, the system analyzes the facial expression in each image frame separately. In the image sequence approach, the system tries to capture the temporal pattern of the motion and changes seen on the face in the sequence of image frames. Recently, attention has been shifted toward the image sequence approach. However, this approach is more difficult and requires more computation than the static approach. In this chapter, we will follow the static image approach and compare several algorithms using the OpenCV 3 library.
The problem of automatic facial expression recognition includes three sub-problems:
Flandmark is an open source C library implementing a facial landmark detector. You can get more information about flandmark in the following sections. Basically, you can use whatever library you want to extract the landmarks. In our implementation, we will use this library to reduce complexity while integrating the library into our project.
In the following sections, we will show you a complete process for implementing a facial expression system. In the next section, you will find several approaches to improve system performance to suit your needs.
In order to simplify the chapter, we will use a dataset to demonstrate the process instead of a live camera. We will use a standard dataset, Japanese Female Facial Expression (JAFFE).There are 214 images of 10 people in the dataset. Each person has three images of each expression. The dataset includes seven expressions (happy, sad, angry, disgust, fear, surprise, and neutral) as shown in the following figure:
You need to download the dataset from the following link: http://www.kasrl.org/jaffe.html