Non-local Attention Improves Description Generation for Retinal Images

Jia-Hong Huang, Ting-Wei Wu, C.-H. Huck Yang, Zenglin Shi, I-Hung Lin, Jesper Tegner, Marcel Worring

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations


Automatically generating medical reports from retinal images is a difficult task in which an algorithm must generate semantically coherent descriptions for a given retinal image. Existing methods mainly rely on the input image to generate descriptions. However, many abstract medical concepts or descriptions cannot be generated based on image information only. In this work, we integrate additional information to help solve this task; we observe that early in the diagnosis process, ophthalmologists have usually written down a small set of keywords denoting important information. These keywords are then subsequently used to aid the later creation of medical reports for a patient. Since these keywords commonly exist and are useful for generating medical reports, we incorporate them into automatic report generation. Since we have two types of inputs expert-defined unordered keywords and images - effectively fusing features from these different modalities is challenging. To that end, we propose a new keyword-driven medical report generation method based on a non-local attention-based multi-modal feature fusion approach, TransFuser, which is capable of fusing features from different types of inputs based on such attention. Our experiments show the proposed method successfully captures the mutual information of keywords and image content. We further show our proposed keyword-driven generation model reinforced by the TransFuser is superior to baselines under the popular text evaluation metrics BLEU, CIDEr, and ROUGE. Trans-Fuser Github:
Original languageEnglish (US)
Title of host publication2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
ISBN (Print)978-1-6654-0916-2
StatePublished - 2022

Bibliographical note

KAUST Repository Item: Exported on 2022-03-15
Acknowledgements: This work is supported by competitive research funding from University of Amsterdam and King Abdullah University of Science and Technology (KAUST)


Dive into the research topics of 'Non-local Attention Improves Description Generation for Retinal Images'. Together they form a unique fingerprint.

Cite this