Facial Action Unit Recognition under Incomplete Data Based on Multi-label Learning with Missing Labels

Yongqiang Li, Baoyuan Wu, Bernard Ghanem, Yongping Zhao, Hongxun Yao, Qiang Ji

Research output: Contribution to journalArticlepeer-review

32 Scopus citations


Facial action unit (AU) recognition has been applied in a wild range of fields, and has attracted great attention in the past two decades. Most existing works on AU recognition assumed that the complete label assignment for each training image is available, which is often not the case in practice. Labeling AU is expensive and time consuming process. Moreover, due to the AU ambiguity and subjective difference, some AUs are difficult to label reliably and confidently. Many AU recognition works try to train the classifier for each AU independently, which is of high computation cost and ignores the dependency among different AUs. In this work, we formulate AU recognition under incomplete data as a multi-label learning with missing labels (MLML) problem. Most existing MLML methods usually employ the same features for all classes. However, we find this setting is unreasonable in AU recognition, as the occurrence of different AUs produce changes of skin surface displacement or face appearance in different face regions. If using the shared features for all AUs, much noise will be involved due to the occurrence of other AUs. Consequently, the changes of the specific AUs cannot be clearly highlighted, leading to the performance degradation. Instead, we propose to extract the most discriminative features for each AU individually, which are learned by the supervised learning method. The learned features are further embedded into the instance-level label smoothness term of our model, which also includes the label consistency and the class-level label smoothness. Both a global solution using st-cut and an approximated solution using conjugate gradient (CG) descent are provided. Experiments on both posed and spontaneous facial expression databases demonstrate the superiority of the proposed method in comparison with several state-of-the-art works.
Original languageEnglish (US)
Pages (from-to)890-900
Number of pages11
JournalPattern Recognition
StatePublished - Jul 8 2016

Bibliographical note

KAUST Repository Item: Exported on 2020-10-01
Acknowledgements: Yongqiang Li is supported by National Natural Science Foundation of China (No. 61402129), and Postdoctoral Foundation Projects (No. LBH-Z14090 and No. 2015M571417). Bernard Ghanem and Baoyuan Wu are supported by funding from King Abdullah University of Science and Technology (KAUST). Hongxun Yao is partially supported by National Natural Science Foundation of China (No. 61472103) and Key Program (No. 61133003).

ASJC Scopus subject areas

  • Artificial Intelligence
  • Signal Processing
  • Software
  • Computer Vision and Pattern Recognition


Dive into the research topics of 'Facial Action Unit Recognition under Incomplete Data Based on Multi-label Learning with Missing Labels'. Together they form a unique fingerprint.

Cite this