Automatic facial expression recognition has attracted much attention since the early nineties, especially in human-computer interaction. As computers start becoming a part of our life, they need to become more and more intelligent. Expression recognition systems will enhance this intelligent interaction between the human and the computer.
Although humans can recognize facial expressions easily, a reliable expression recognition system is still a challenge. In this chapter, we will introduce a basic implementation of facial expression using various algorithms from the OpenCV library, including feature extraction and classification using the ml module.
In this chapter, we will be going through the following topics in brief:
- A simple architecture to recognize human facial expressions
- Feature extraction algorithms in the OpenCV library
- The learning and testing stage, with various machine learning algorithms
Automatic facial expression recognition is an interesting and challenging problem and has several important applications in many areas such as human-computer interaction, human behavior understanding, and data-driven animation. Unlike face recognition, facial expression recognition needs to discriminate between the same expression in different individuals. The problem becomes more difficult as a person may show the same expression in different ways.
The current existing approaches for measuring facial expressions can be categorized into two types: static image and image sequence. In the static image approach, the system analyzes the facial expression in each image frame separately. In the image sequence approach, the system tries to capture the temporal pattern of the motion and changes seen on the face in the sequence of image frames. Recently, attention has been shifted toward the image sequence approach. However, this approach is more difficult and requires more computation than the static approach. In this chapter, we will follow the static image approach and compare several algorithms using the OpenCV 3 library.
The problem of automatic facial expression recognition includes three sub-problems:
- Finding the face region in the image: The precise position of the face is very important for facial analysis. In this problem, we want to find the face region in the image. This problem can be viewed as a detection problem. In our implementation, we will use the cascade classifier in OpenCV's objdetect module to detect the faces. However, the cascade classifier is prone to alignment error. Therefore, we apply the flandmark library to extract the facial landmarks from the face region and use these landmarks to extract the precise face region.
Note
Flandmark is an open source C library implementing a facial landmark detector. You can get more information about flandmark in the following sections. Basically, you can use whatever library you want to extract the landmarks. In our implementation, we will use this library to reduce complexity while integrating the library into our project.
- Extracting features from the face region: Given the face region, the system will extract facial expression information as a feature vector. The feature vector encodes the relevant information from the input data. In our implementation, the feature vector is obtained by using the combination of the feature detector from the feature2d module and the kmeans algorithm from the core module.
- Classifying the features into emotion categories: This is a classification problem. The system uses classification algorithms to map the extracted feature from the previous step to an emotion category (such as happy, neutral, or sad). This is the main subject of the chapter. We will evaluate machine learning algorithms from the ml module, including neural networks, the support vector machine, and K-Nearest-Neighbor.
In the following sections, we will show you a complete process for implementing a facial expression system. In the next section, you will find several approaches to improve system performance to suit your needs.
In order to simplify the chapter, we will use a dataset to demonstrate the process instead of a live camera. We will use a standard dataset, Japanese Female Facial Expression (JAFFE).There are 214 images of 10 people in the dataset. Each person has three images of each expression. The dataset includes seven expressions (happy, sad, angry, disgust, fear, surprise, and neutral) as shown in the following figure:
Note
You need to download the dataset from the following link: http://www.kasrl.org/jaffe.html
Sample image from the JAFFE dataset.