Continual Learning with Structured Inheritance for Semantic Segmentation in Aerial Imagery

Yingchao Feng, Xian Sun, Wenhui Diao, Jihao Li, Xin Gao, Kun Fu

Research output: Contribution to journalArticlepeer-review

27 Scopus citations


With the rapid update and iteration of current aerial image data, the continual learning scenarios and catastrophic forgetting problem attracted increased attention, especially in the semantic segmentation task. However, the existing methods mainly focus on the class continual learning in a single task and are not satisfactory when extended to multiple tasks. In this article, we consider more realistic and complicated settings, namely task continual learning. We revisit the characteristics of semantic segmentation and knowledge distillation (KD) strategy, then propose a general and effective framework, named structured inheritance, to learn new tasks while retaining high performance on old tasks. Specifically, we present two structure-preserving penalties: pixel affinity structure loss and representation consistency structure loss. The former breaks the isolation of pixels and retains the pixel interactive information learned by the old tasks. At the same time, the latter protects high-frequency stationary information between sequence semantic segmentation tasks. Our approach does not need to add extra parameters nor does it need to access the data stream of the old tasks. Therefore, it can be applied in practical applications with strict computational burden, memory cost, and storage budget. Extensive continual learning experiments on four semantic segmentation datasets of Vaihingen, Potsdam, DeepGlobe, and Gaofen challenge semantic segmentation dataset (GCSS) prove the effectiveness of our proposed framework, which outperforms the current state-of-the-art methods and even exceeds the theoretical upper-bound performance of multitask learning. The code and models will be made publicly available.
Original languageEnglish (US)
JournalIEEE Transactions on Geoscience and Remote Sensing
StatePublished - Jan 1 2022
Externally publishedYes

Bibliographical note

Generated from Scopus record by KAUST IRTS on 2023-09-21


Dive into the research topics of 'Continual Learning with Structured Inheritance for Semantic Segmentation in Aerial Imagery'. Together they form a unique fingerprint.

Cite this