Compositional Continual Language Learning

Yuanpeng Li, Liang Zhao, Kenneth Church, Mohamed Elhoseiny

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Motivated by the human's ability to continually learn and gain knowledge over time, several research efforts have been pushing the limits of machines to constantly learn while alleviating catastrophic forgetting. Most of the existing methods have been focusing on continual learning of label prediction tasks, which have fixed input and output sizes. In this paper, we propose a new scenario of continual learning which handles sequence-to-sequence tasks common in language learning. We further propose an approach to use label prediction continual learning algorithm for sequence-to-sequence continual learning by leveraging compositionality. Experimental results show that the proposed method has significant improvement over state-of-the-art methods. It enables knowledge transfer and prevents catastrophic forgetting, resulting in more than 85% accuracy up to 100 stages, compared with less than 50% accuracy for baselines in instruction learning task. It also shows significant improvement in machine translation task. This is the first work to combine continual learning and compositionality for language learning, and we hope this work will make machines more helpful in various tasks.
Original languageEnglish (US)
Title of host publicationInternational Conference on Learning Representations (ICLR) 2020
StatePublished - Mar 11 2020

Bibliographical note

KAUST Repository Item: Exported on 2020-10-01
Acknowledgements: Work partially done while visiting Baidu Research.


Dive into the research topics of 'Compositional Continual Language Learning'. Together they form a unique fingerprint.

Cite this