Real-time Hyperspectral Imaging in Hardware via Trained Metasurface Encoders

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

10 Scopus citations

Abstract

Hyperspectral imaging has attracted significant attention to identify spectral signatures for image classification and automated pattern recognition in computer vision. State-of-the-art implementations of snapshot hyperspectral imaging rely on bulky, non-integrated, and expensive optical elements, including lenses, spectrometers, and filters. These macroscopic components do not allow fast data processing for, e.g. real-time and high-resolution videos. This work introduces Hyplex™, a new integrated architecture addressing the limitations discussed above. Hyplex™ is a CMOS-compatible, fast hyperspectral camera that replaces bulk optics with nanoscale metasurfaces inversely designed through artificial intelligence. Hyplex™ does not require spectrometers but makes use of conventional monochrome cameras, opening up the possibility for real-time and high-resolution hyperspectral imaging at inexpensive costs. Hyplex™ exploits a model-driven optimization, which connects the physical metasurfaces layer with modern visual computing approaches based on end-to-end training. We design and implement a prototype version of Hyplex™ and compare its performance against the state-of-the-art for typical imaging tasks such as spectral reconstruction and semantic segmentation. In all benchmarks, Hyplex™ reports the smallest reconstruction error. We additionally present what is, to the best of our knowledge, the largest publicly available labeled hyperspectral dataset for semantic segmentation.11Dataset available on https://github.com/makamoa/hyplex.

Original languageEnglish (US)
Title of host publicationProceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
PublisherIEEE Computer Society
Pages12682-12692
Number of pages11
ISBN (Electronic)9781665469463
DOIs
StatePublished - 2022
Event2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 - New Orleans, United States
Duration: Jun 19 2022Jun 24 2022

Publication series

NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Volume2022-June
ISSN (Print)1063-6919

Conference

Conference2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
Country/TerritoryUnited States
CityNew Orleans
Period06/19/2206/24/22

Bibliographical note

Funding Information:
Improved results could also be obtained if we augment the publicly available hyperspectral datasets with more scenes obtained at different wavelengths and in different settings such as, e.g., medical. Such study could generalize the results of Hyplex™ to provide high impact systems for personalized healthcare and precision medicine. Hyplex™ could provide a game-changer technology in this field, leveraging its vast capacity to fast-process high-resolution hyperspec-tral images (see Sec. 7 of Supplementary Material) at speed comparable with current RGB cameras. Acknowledgements. This work was supported by the King Abdullah University of Science and Technology (KAUST) through the Artificial Intelligence Initiative (AII) funding. This research received funding from KAUST (Award OSR-2016-CRG5-2995). Parallel simulations are performed on KAUST’s Shaheen supercomputer.

Publisher Copyright:
© 2022 IEEE.

Keywords

  • Low-level vision
  • Physics-based vision and shape-from-X
  • Vision applications and systems

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition

Fingerprint

Dive into the research topics of 'Real-time Hyperspectral Imaging in Hardware via Trained Metasurface Encoders'. Together they form a unique fingerprint.

Cite this