Automatic annotation of structured facts in images

Mohamed Elhoseiny; Scott Cohen; Walter Chang; Brian Price; Ahmed Elgammal

Automatic annotation of structured facts in images

Mohamed Elhoseiny, Scott Cohen, Walter Chang, Brian Price, Ahmed Elgammal

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

Motivated by the application of fact-level image understanding, we present an automatic method for data collection of structured visual facts from images with captions. Example structured facts include attributed objects (e.g., <flower, red>), actions (e.g., <baby, smile>), interactions (e.g., <man, walking, dog>), and positional information (e.g., <vase, on, table>). The collected annotations are in the form of fact-image pairs (e.g.,<man, walking, dog> and an image region containing this fact). With a language approach, the proposed method is able to collect hundreds of thousands of visual fact annotations with accuracy of 83% according to human judgment. Our method automatically collected more than 380,000 visual fact annotations and more than 110,000 unique visual facts from images with captions and localized them in images in less than one day of processing time on standard CPU platforms. We will make the data publically available.

Original language	English (US)
Title of host publication	Proceedings of the 5th Workshop on Vision and Language, VL 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016
Editors	Anya Belz, Erkut Erdem, Krystian Mikolajczyk, Katerina Pastra
Publisher	Association for Computational Linguistics (ACL)
Pages	1-9
Number of pages	9
ISBN (Electronic)	9781945626111
State	Published - 2016
Event	5th Workshop on Vision and Language, VL 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 - Berlin, Germany Duration: Aug 12 2016 → …

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	5th Workshop on Vision and Language, VL 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016
Country/Territory	Germany
City	Berlin
Period	08/12/16 → …

Bibliographical note

Publisher Copyright:
© ACL 2016.All right reserved.

ASJC Scopus subject areas

Computer Science Applications
Linguistics and Language
Language and Linguistics

Cite this

Elhoseiny, M., Cohen, S., Chang, W., Price, B., & Elgammal, A. (2016). Automatic annotation of structured facts in images. In A. Belz, E. Erdem, K. Mikolajczyk, & K. Pastra (Eds.), Proceedings of the 5th Workshop on Vision and Language, VL 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 (pp. 1-9). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). Association for Computational Linguistics (ACL).

Elhoseiny, Mohamed ; Cohen, Scott ; Chang, Walter et al. / Automatic annotation of structured facts in images. Proceedings of the 5th Workshop on Vision and Language, VL 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016. editor / Anya Belz ; Erkut Erdem ; Krystian Mikolajczyk ; Katerina Pastra. Association for Computational Linguistics (ACL), 2016. pp. 1-9 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{8f1bf9efa27a4bf59259b63138e6b122,

title = "Automatic annotation of structured facts in images",

abstract = "Motivated by the application of fact-level image understanding, we present an automatic method for data collection of structured visual facts from images with captions. Example structured facts include attributed objects (e.g., ), actions (e.g., ), interactions (e.g., ), and positional information (e.g., ). The collected annotations are in the form of fact-image pairs (e.g., and an image region containing this fact). With a language approach, the proposed method is able to collect hundreds of thousands of visual fact annotations with accuracy of 83% according to human judgment. Our method automatically collected more than 380,000 visual fact annotations and more than 110,000 unique visual facts from images with captions and localized them in images in less than one day of processing time on standard CPU platforms. We will make the data publically available.",

author = "Mohamed Elhoseiny and Scott Cohen and Walter Chang and Brian Price and Ahmed Elgammal",

note = "Publisher Copyright: {\textcopyright} ACL 2016.All right reserved.; 5th Workshop on Vision and Language, VL 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 ; Conference date: 12-08-2016",

year = "2016",

language = "English (US)",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "1--9",

editor = "Anya Belz and Erkut Erdem and Krystian Mikolajczyk and Katerina Pastra",

booktitle = "Proceedings of the 5th Workshop on Vision and Language, VL 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016",

address = "United States",

}

Elhoseiny, M, Cohen, S, Chang, W, Price, B & Elgammal, A 2016, Automatic annotation of structured facts in images. in A Belz, E Erdem, K Mikolajczyk & K Pastra (eds), Proceedings of the 5th Workshop on Vision and Language, VL 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics (ACL), pp. 1-9, 5th Workshop on Vision and Language, VL 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Berlin, Germany, 08/12/16.

Automatic annotation of structured facts in images. / Elhoseiny, Mohamed; Cohen, Scott; Chang, Walter et al.
Proceedings of the 5th Workshop on Vision and Language, VL 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016. ed. / Anya Belz; Erkut Erdem; Krystian Mikolajczyk; Katerina Pastra. Association for Computational Linguistics (ACL), 2016. p. 1-9 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Automatic annotation of structured facts in images

AU - Elhoseiny, Mohamed

AU - Cohen, Scott

AU - Chang, Walter

AU - Price, Brian

AU - Elgammal, Ahmed

PY - 2016

Y1 - 2016

N2 - Motivated by the application of fact-level image understanding, we present an automatic method for data collection of structured visual facts from images with captions. Example structured facts include attributed objects (e.g., ), actions (e.g., ), interactions (e.g., ), and positional information (e.g., ). The collected annotations are in the form of fact-image pairs (e.g., and an image region containing this fact). With a language approach, the proposed method is able to collect hundreds of thousands of visual fact annotations with accuracy of 83% according to human judgment. Our method automatically collected more than 380,000 visual fact annotations and more than 110,000 unique visual facts from images with captions and localized them in images in less than one day of processing time on standard CPU platforms. We will make the data publically available.

AB - Motivated by the application of fact-level image understanding, we present an automatic method for data collection of structured visual facts from images with captions. Example structured facts include attributed objects (e.g., ), actions (e.g., ), interactions (e.g., ), and positional information (e.g., ). The collected annotations are in the form of fact-image pairs (e.g., and an image region containing this fact). With a language approach, the proposed method is able to collect hundreds of thousands of visual fact annotations with accuracy of 83% according to human judgment. Our method automatically collected more than 380,000 visual fact annotations and more than 110,000 unique visual facts from images with captions and localized them in images in less than one day of processing time on standard CPU platforms. We will make the data publically available.

UR - http://www.scopus.com/inward/record.url?scp=85090248209&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85090248209

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 1

EP - 9

BT - Proceedings of the 5th Workshop on Vision and Language, VL 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016

A2 - Belz, Anya

A2 - Erdem, Erkut

A2 - Mikolajczyk, Krystian

A2 - Pastra, Katerina

PB - Association for Computational Linguistics (ACL)

T2 - 5th Workshop on Vision and Language, VL 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016

Y2 - 12 August 2016

ER -

Elhoseiny M, Cohen S, Chang W, Price B, Elgammal A. Automatic annotation of structured facts in images. In Belz A, Erdem E, Mikolajczyk K, Pastra K, editors, Proceedings of the 5th Workshop on Vision and Language, VL 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016. Association for Computational Linguistics (ACL). 2016. p. 1-9. (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Automatic annotation of structured facts in images

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this