Can We See More? Joint Frontalization and Hallucination of Unaligned Tiny Faces

Xin Yu, Fatemeh Shiri, Bernard Ghanem, Fatih Porikli

Research output: Contribution to journalArticlepeer-review

30 Scopus citations

Abstract

In popular TV programs (such as CSI), a very low-resolution face image of a person, who is not even looking at the camera in many cases, is digitally super-resolved to a degree that suddenly the person's identity is made visible and recognizable. Of course, we suspect that this is merely a cinematographic special effect and such a magical transformation of a single image is not technically possible. Or, is it? In this paper, we push the boundaries of super-resolving (hallucinating to be more accurate) a tiny, non-frontal face image to understand how much of this is possible by leveraging the availability of large datasets and deep networks. To this end, we introduce a novel Transformative Adversarial Neural Network (TANN) to jointly frontalize very-low resolution (i.e. $16\times 16$ pixels) out-of-plane rotated face images (including profile views) and aggressively super-resolve them ($8\times$), regardless of their original poses and without using any 3D information. TANN is composed of two components: a transformative upsampling network which embodies encoding, spatial transformation and deconvolutional layers, and a discriminative network that enforces the generated high-resolution frontal faces to lie on the same manifold as real frontal face images. We evaluate our method on a large set of synthesized non-frontal face images to assess its reconstruction performance. Extensive experiments demonstrate that TANN generates both qualitatively and quantitatively superior results achieving over 4 dB improvement over the state-of-the-art.
Original languageEnglish (US)
Pages (from-to)1-1
Number of pages1
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
DOIs
StatePublished - 2019

Bibliographical note

KAUST Repository Item: Exported on 2020-10-01
Acknowledgements: This work was supported by the Australian Research Council‘s Discovery Projects funding scheme (project DP150104645), Australian Research Council Centre of Excellence for Robotic Vision (project number CE140100016) and King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research.

Fingerprint

Dive into the research topics of 'Can We See More? Joint Frontalization and Hallucination of Unaligned Tiny Faces'. Together they form a unique fingerprint.

Cite this