SAGES consensus recommendations on an annotation framework for surgical video

the SAGES Video Annotation for AI Working Groups

Research output: Contribution to journalArticlepeer-review

38 Scopus citations


Background: The growing interest in analysis of surgical video through machine learning has led to increased research efforts; however, common methods of annotating video data are lacking. There is a need to establish recommendations on the annotation of surgical video data to enable assessment of algorithms and multi-institutional collaboration. Methods: Four working groups were formed from a pool of participants that included clinicians, engineers, and data scientists. The working groups were focused on four themes: (1) temporal models, (2) actions and tasks, (3) tissue characteristics and general anatomy, and (4) software and data structure. A modified Delphi process was utilized to create a consensus survey based on suggested recommendations from each of the working groups. Results: After three Delphi rounds, consensus was reached on recommendations for annotation within each of these domains. A hierarchy for annotation of temporal events in surgery was established. Conclusions: While additional work remains to achieve accepted standards for video annotation in surgery, the consensus recommendations on a general framework for annotation presented here lay the foundation for standardization. This type of framework is critical to enabling diverse datasets, performance benchmarks, and collaboration.

Original languageEnglish (US)
Pages (from-to)4918-4929
Number of pages12
JournalSurgical Endoscopy
Issue number9
StatePublished - Sep 2021

Bibliographical note

Funding Information:
This work was supported by the SAGES Foundation, Digital Surgery, Imagestream, Intuitive Surgical, Johnson & Johnson CSATS, Karl Storz, Medtronic, Olympus, Stryker, Theator, and Verb Surgical.

Funding Information:
Ozanan Meireles is a consultant for Olympus and Medtronic and has received research support from Olympus. Guy Rosman is an employee of Toyota Research Institute (TRI); the views expressed in this paper do not reflect those of TRI or any other Toyota entity. He has received research support from Olympus. Amin Madani is a consultant for Activ Surgical. Gregory Hager is a consultant for and has an equity interest in the company. Nicolas Padoy is a consultant for Caresyntax and has received research support from Intuitive Surgical. Thomas Ward has received research support from Olympus. Daniel Hashimoto is a consultant for Johnson & Johnson and Verily Life Sciences. He has received research support from Olympus and the Intuitive Foundation. Maria S. Altieri, Lawrence Carin, Carla M. Pugh and Patricia Sylla have no conflicts of interest or financial ties to disclose.

Publisher Copyright:
© 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.


  • Annotation
  • Artificial intelligence
  • Computer vision
  • Consensus
  • Minimally invasive surgery
  • Surgical video

ASJC Scopus subject areas

  • Surgery


Dive into the research topics of 'SAGES consensus recommendations on an annotation framework for surgical video'. Together they form a unique fingerprint.

Cite this