elBERto: Self-supervised commonsense learning for question answering

Xunlin Zhan, Yuan Li, Xiao Dong, Xiaodan Liang*, Zhiting Hu, Lawrence Carin

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

6 Scopus citations


Commonsense question answering requires reasoning about everyday situations and causes and effects implicit in context. Typically, existing approaches first retrieve external evidence and then perform commonsense reasoning using these evidence. In this paper, we propose a Self-supervised Bidirectional Encoder Representation Learning of Commonsense (elBERto) framework, which is compatible with off-the-shelf QA model architectures. The framework comprises five self-supervised tasks to force the model to fully exploit the additional training signals from contexts containing rich commonsense. The tasks include a novel Contrastive Relation Learning task to encourage the model to distinguish between logically contrastive contexts, a new Jigsaw Puzzle task that requires the model to infer logical chains in long contexts, and three classic self-supervised learning(SSL) tasks to maintain pre-trained models’ language encoding ability. On the representative WIQA, CosmosQA, and ReClor datasets, elBERto outperforms all other methods using the same backbones and the same training set, including those utilizing explicit graph reasoning and external knowledge retrieval. Moreover, elBERto achieves substantial improvements on out-of-paragraph and no-effect questions where simple lexical similarity comparison does not help, indicating that it successfully learns commonsense and is able to leverage it when given dynamic context.

Original languageEnglish (US)
Article number109964
JournalKnowledge-Based Systems
StatePublished - Dec 22 2022

Bibliographical note

Funding Information:
This work is supported by the National Natural Science Foundation of China (Grant No. 61976233 and U19A2073 ), Guangdong Provincial Natural Science Foundation of China (Grant No. 2019B1515120039 ), and the Office of Naval Research, United States under grant N00014-18-1-2871 .

Publisher Copyright:
© 2022 Elsevier B.V.


  • Commonsense reasoning
  • Question answering
  • Self-supervised learning

ASJC Scopus subject areas

  • Software
  • Management Information Systems
  • Information Systems and Management
  • Artificial Intelligence


Dive into the research topics of 'elBERto: Self-supervised commonsense learning for question answering'. Together they form a unique fingerprint.

Cite this