Text Feature Adversarial Learning for Text Generation With Knowledge Transfer From GPT2

Hao Zhang, Yulai Cong, Zhengjue Wang, Lei Zhang, Miaoyun Zhao, Liqun Chen, Shijing Si, Ricardo Henao, Lawrence Carin

Research output: Contribution to journalArticlepeer-review

1 Scopus citations


Text generation is a key component of many natural language tasks. Motivated by the success of generative adversarial networks (GANs) for image generation, many text-specific GANs have been proposed. However, due to the discrete nature of text, these text GANs often use reinforcement learning (RL) or continuous relaxations to calculate gradients during learning, leading to high-variance or biased estimation. Furthermore, the existing text GANs often suffer from mode collapse (i.e., they have limited generative diversity). To tackle these problems, we propose a new text GAN model named text feature GAN (TFGAN), where adversarial learning is performed in a continuous text feature space. In the adversarial game, GPT2 provides the “true” features, while the generator of TFGAN learns from them. TFGAN is trained by maximum likelihood estimation on text space and adversarial learning on text feature space, effectively combining them into a single objective, while alleviating mode collapse. TFGAN achieves appealing performance in text generation tasks, and it can also be used as a flexible framework for learning text representations.
Original languageEnglish (US)
JournalIEEE Transactions on Neural Networks and Learning Systems
StatePublished - Jan 1 2022
Externally publishedYes

Bibliographical note

Generated from Scopus record by KAUST IRTS on 2023-02-15


Dive into the research topics of 'Text Feature Adversarial Learning for Text Generation With Knowledge Transfer From GPT2'. Together they form a unique fingerprint.

Cite this