What Makes Good In-Context Examples for GPT-3?

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, Weizhu Chen

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

191 Scopus citations

Abstract

GPT-3 has attracted lots of attention due to its superior performance across a wide range of NLP tasks, especially with its in-context learning abilities. Despite its success, we found that the empirical results of GPT-3 depend heavily on the choice of in-context examples. In this work, we investigate whether there are more effective strategies for judiciously selecting in-context examples (relative to random sampling) that better leverage GPT-3's in-context learning capabilities. Inspired by the recent success of leveraging a retrieval module to augment neural networks, we propose to retrieve examples that are semantically-similar to a test query sample to formulate its corresponding prompt. Intuitively, the examples selected with such a strategy may serve as more informative inputs to unleash GPT-3's power of text generation. We evaluate the proposed approach on several natural language understanding and generation benchmarks, where the retrieval-based prompt selection approach consistently outperforms the random selection baseline. Moreover, it is observed that the sentence encoders fine-tuned on task-related datasets yield even more helpful retrieval results. Notably, significant gains are observed on tasks such as table-to-text generation (44.3% on the ToTTo dataset) and open-domain question answering (45.5% on the NQ dataset).

Original languageEnglish (US)
Title of host publicationDeeLIO 2022 - Deep Learning Inside Out
Subtitle of host publication3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, Proceedings of the Workshop
EditorsEneko Agirre, Marianna Apidianaki, Ivan Vulic
PublisherAssociation for Computational Linguistics (ACL)
Pages100-114
Number of pages15
ISBN (Electronic)9781955917322
StatePublished - 2022
EventDeep Learning Inside Out: 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO 2022 - Virtual, Dublin, Ireland
Duration: May 27 2022 → …

Publication series

NameDeeLIO 2022 - Deep Learning Inside Out: 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, Proceedings of the Workshop

Conference

ConferenceDeep Learning Inside Out: 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO 2022
Country/TerritoryIreland
CityVirtual, Dublin
Period05/27/22 → …

Bibliographical note

Publisher Copyright:
© 2022 Association for Computational Linguistics.

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Software
  • Hardware and Architecture

Fingerprint

Dive into the research topics of 'What Makes Good In-Context Examples for GPT-3?'. Together they form a unique fingerprint.

Cite this