SAGES consensus recommendations on an annotation framework for surgical video

the SAGES Video Annotation for AI Working Groups

doi:10.1007/s00464-021-08578-9

SAGES consensus recommendations on an annotation framework for surgical video

the SAGES Video Annotation for AI Working Groups

Computer, Electrical and Mathematical Sciences and Engineering

Research output: Contribution to journal › Article › peer-review

47 Scopus citations

Abstract

Background: The growing interest in analysis of surgical video through machine learning has led to increased research efforts; however, common methods of annotating video data are lacking. There is a need to establish recommendations on the annotation of surgical video data to enable assessment of algorithms and multi-institutional collaboration. Methods: Four working groups were formed from a pool of participants that included clinicians, engineers, and data scientists. The working groups were focused on four themes: (1) temporal models, (2) actions and tasks, (3) tissue characteristics and general anatomy, and (4) software and data structure. A modified Delphi process was utilized to create a consensus survey based on suggested recommendations from each of the working groups. Results: After three Delphi rounds, consensus was reached on recommendations for annotation within each of these domains. A hierarchy for annotation of temporal events in surgery was established. Conclusions: While additional work remains to achieve accepted standards for video annotation in surgery, the consensus recommendations on a general framework for annotation presented here lay the foundation for standardization. This type of framework is critical to enabling diverse datasets, performance benchmarks, and collaboration.

Original language	English (US)
Pages (from-to)	4918-4929
Number of pages	12
Journal	Surgical Endoscopy
Volume	35
Issue number	9
DOIs	https://doi.org/10.1007/s00464-021-08578-9
State	Published - Sep 2021

Bibliographical note

Funding Information:
This work was supported by the SAGES Foundation, Digital Surgery, Imagestream, Intuitive Surgical, Johnson & Johnson CSATS, Karl Storz, Medtronic, Olympus, Stryker, Theator, and Verb Surgical.

Funding Information:
Ozanan Meireles is a consultant for Olympus and Medtronic and has received research support from Olympus. Guy Rosman is an employee of Toyota Research Institute (TRI); the views expressed in this paper do not reflect those of TRI or any other Toyota entity. He has received research support from Olympus. Amin Madani is a consultant for Activ Surgical. Gregory Hager is a consultant for theator.io and has an equity interest in the company. Nicolas Padoy is a consultant for Caresyntax and has received research support from Intuitive Surgical. Thomas Ward has received research support from Olympus. Daniel Hashimoto is a consultant for Johnson & Johnson and Verily Life Sciences. He has received research support from Olympus and the Intuitive Foundation. Maria S. Altieri, Lawrence Carin, Carla M. Pugh and Patricia Sylla have no conflicts of interest or financial ties to disclose.

Publisher Copyright:
© 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.

Keywords

Annotation
Artificial intelligence
Computer vision
Consensus
Minimally invasive surgery
Surgical video

ASJC Scopus subject areas

Surgery

Access to Document

10.1007/s00464-021-08578-9

Cite this

@article{62e107d8cf2243308675fae03780b7f5,

title = "SAGES consensus recommendations on an annotation framework for surgical video",

abstract = "Background: The growing interest in analysis of surgical video through machine learning has led to increased research efforts; however, common methods of annotating video data are lacking. There is a need to establish recommendations on the annotation of surgical video data to enable assessment of algorithms and multi-institutional collaboration. Methods: Four working groups were formed from a pool of participants that included clinicians, engineers, and data scientists. The working groups were focused on four themes: (1) temporal models, (2) actions and tasks, (3) tissue characteristics and general anatomy, and (4) software and data structure. A modified Delphi process was utilized to create a consensus survey based on suggested recommendations from each of the working groups. Results: After three Delphi rounds, consensus was reached on recommendations for annotation within each of these domains. A hierarchy for annotation of temporal events in surgery was established. Conclusions: While additional work remains to achieve accepted standards for video annotation in surgery, the consensus recommendations on a general framework for annotation presented here lay the foundation for standardization. This type of framework is critical to enabling diverse datasets, performance benchmarks, and collaboration.",

keywords = "Annotation, Artificial intelligence, Computer vision, Consensus, Minimally invasive surgery, Surgical video",

author = "{the SAGES Video Annotation for AI Working Groups} and Meireles, {Ozanan R.} and Guy Rosman and Altieri, {Maria S.} and Lawrence Carin and Gregory Hager and Amin Madani and Nicolas Padoy and Pugh, {Carla M.} and Patricia Sylla and Ward, {Thomas M.} and Hashimoto, {Daniel A.} and Yutong Ban and Fillipo Filicori and Pietro Mascagni and John Mellinger and Christopher Schlacta and Stefanie Speidel and Thorsten Juergens and Pablo Garcia-Kilroy and Dotan Asselman and Jordan Bohnen and Draelos, {Rachel Ballantyne} and Hans Fuchs and Ricardo Henao and Duygu Sarikaya and Christopher Boyle and Danyal Fer and Zhen Li and Arvind Ramadorai and Danail Stoyanov and Andrew Yoo and Cristians Gonzalez and Dmitry Oleynikov and Janey Pratt and Danny Scott and Swaroop Vedula and Elan Witkowski and Takayuki Shimizu and Mark Tousignant and Dan Azagury and Flavien Bridault and Brian Dunkin and Teodor Grantcharov and Pierre Jannin and Anand Malpani and Silvana Perretta and Steven Schwaitzberg and Anthony Jarc and Kurt Landfors and Amit Mahadik",

note = "Funding Information: This work was supported by the SAGES Foundation, Digital Surgery, Imagestream, Intuitive Surgical, Johnson & Johnson CSATS, Karl Storz, Medtronic, Olympus, Stryker, Theator, and Verb Surgical. Funding Information: Ozanan Meireles is a consultant for Olympus and Medtronic and has received research support from Olympus. Guy Rosman is an employee of Toyota Research Institute (TRI); the views expressed in this paper do not reflect those of TRI or any other Toyota entity. He has received research support from Olympus. Amin Madani is a consultant for Activ Surgical. Gregory Hager is a consultant for theator.io and has an equity interest in the company. Nicolas Padoy is a consultant for Caresyntax and has received research support from Intuitive Surgical. Thomas Ward has received research support from Olympus. Daniel Hashimoto is a consultant for Johnson & Johnson and Verily Life Sciences. He has received research support from Olympus and the Intuitive Foundation. Maria S. Altieri, Lawrence Carin, Carla M. Pugh and Patricia Sylla have no conflicts of interest or financial ties to disclose. Publisher Copyright: {\textcopyright} 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.",

year = "2021",

month = sep,

doi = "10.1007/s00464-021-08578-9",

language = "English (US)",

volume = "35",

pages = "4918--4929",

journal = "Surgical Endoscopy",

issn = "0930-2794",

publisher = "Springer New York",

number = "9",

}

TY - JOUR

T1 - SAGES consensus recommendations on an annotation framework for surgical video

AU - the SAGES Video Annotation for AI Working Groups

AU - Meireles, Ozanan R.

AU - Rosman, Guy

AU - Altieri, Maria S.

AU - Carin, Lawrence

AU - Hager, Gregory

AU - Madani, Amin

AU - Padoy, Nicolas

AU - Pugh, Carla M.

AU - Sylla, Patricia

AU - Ward, Thomas M.

AU - Hashimoto, Daniel A.

AU - Ban, Yutong

AU - Filicori, Fillipo

AU - Mascagni, Pietro

AU - Mellinger, John

AU - Schlacta, Christopher

AU - Speidel, Stefanie

AU - Juergens, Thorsten

AU - Garcia-Kilroy, Pablo

AU - Asselman, Dotan

AU - Bohnen, Jordan

AU - Draelos, Rachel Ballantyne

AU - Fuchs, Hans

AU - Henao, Ricardo

AU - Sarikaya, Duygu

AU - Boyle, Christopher

AU - Fer, Danyal

AU - Li, Zhen

AU - Ramadorai, Arvind

AU - Stoyanov, Danail

AU - Yoo, Andrew

AU - Gonzalez, Cristians

AU - Oleynikov, Dmitry

AU - Pratt, Janey

AU - Scott, Danny

AU - Vedula, Swaroop

AU - Witkowski, Elan

AU - Shimizu, Takayuki

AU - Tousignant, Mark

AU - Azagury, Dan

AU - Bridault, Flavien

AU - Dunkin, Brian

AU - Grantcharov, Teodor

AU - Jannin, Pierre

AU - Malpani, Anand

AU - Perretta, Silvana

AU - Schwaitzberg, Steven

AU - Jarc, Anthony

AU - Landfors, Kurt

AU - Mahadik, Amit

N1 - Funding Information: This work was supported by the SAGES Foundation, Digital Surgery, Imagestream, Intuitive Surgical, Johnson & Johnson CSATS, Karl Storz, Medtronic, Olympus, Stryker, Theator, and Verb Surgical. Funding Information: Ozanan Meireles is a consultant for Olympus and Medtronic and has received research support from Olympus. Guy Rosman is an employee of Toyota Research Institute (TRI); the views expressed in this paper do not reflect those of TRI or any other Toyota entity. He has received research support from Olympus. Amin Madani is a consultant for Activ Surgical. Gregory Hager is a consultant for theator.io and has an equity interest in the company. Nicolas Padoy is a consultant for Caresyntax and has received research support from Intuitive Surgical. Thomas Ward has received research support from Olympus. Daniel Hashimoto is a consultant for Johnson & Johnson and Verily Life Sciences. He has received research support from Olympus and the Intuitive Foundation. Maria S. Altieri, Lawrence Carin, Carla M. Pugh and Patricia Sylla have no conflicts of interest or financial ties to disclose. Publisher Copyright: © 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.

PY - 2021/9

Y1 - 2021/9

N2 - Background: The growing interest in analysis of surgical video through machine learning has led to increased research efforts; however, common methods of annotating video data are lacking. There is a need to establish recommendations on the annotation of surgical video data to enable assessment of algorithms and multi-institutional collaboration. Methods: Four working groups were formed from a pool of participants that included clinicians, engineers, and data scientists. The working groups were focused on four themes: (1) temporal models, (2) actions and tasks, (3) tissue characteristics and general anatomy, and (4) software and data structure. A modified Delphi process was utilized to create a consensus survey based on suggested recommendations from each of the working groups. Results: After three Delphi rounds, consensus was reached on recommendations for annotation within each of these domains. A hierarchy for annotation of temporal events in surgery was established. Conclusions: While additional work remains to achieve accepted standards for video annotation in surgery, the consensus recommendations on a general framework for annotation presented here lay the foundation for standardization. This type of framework is critical to enabling diverse datasets, performance benchmarks, and collaboration.

AB - Background: The growing interest in analysis of surgical video through machine learning has led to increased research efforts; however, common methods of annotating video data are lacking. There is a need to establish recommendations on the annotation of surgical video data to enable assessment of algorithms and multi-institutional collaboration. Methods: Four working groups were formed from a pool of participants that included clinicians, engineers, and data scientists. The working groups were focused on four themes: (1) temporal models, (2) actions and tasks, (3) tissue characteristics and general anatomy, and (4) software and data structure. A modified Delphi process was utilized to create a consensus survey based on suggested recommendations from each of the working groups. Results: After three Delphi rounds, consensus was reached on recommendations for annotation within each of these domains. A hierarchy for annotation of temporal events in surgery was established. Conclusions: While additional work remains to achieve accepted standards for video annotation in surgery, the consensus recommendations on a general framework for annotation presented here lay the foundation for standardization. This type of framework is critical to enabling diverse datasets, performance benchmarks, and collaboration.

KW - Annotation

KW - Artificial intelligence

KW - Computer vision

KW - Consensus

KW - Minimally invasive surgery

KW - Surgical video

UR - http://www.scopus.com/inward/record.url?scp=85109676129&partnerID=8YFLogxK

U2 - 10.1007/s00464-021-08578-9

DO - 10.1007/s00464-021-08578-9

M3 - Article

C2 - 34231065

AN - SCOPUS:85109676129

SN - 0930-2794

VL - 35

SP - 4918

EP - 4929

JO - Surgical Endoscopy

JF - Surgical Endoscopy

IS - 9

ER -

SAGES consensus recommendations on an annotation framework for surgical video

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this