Seeing through obstructions with diffractive cloaking

Zheng Shi, Yuval Bahat, Seung-Hwan Baek, Qiang Fu, Hadi Amata, Xiao Li, Praneeth Chakravarthula, Wolfgang Heidrich, Felix Heide

Research output: Contribution to journalArticlepeer-review

9 Scopus citations

Abstract

Unwanted camera obstruction can severely degrade captured images, including both scene occluders near the camera and partial occlusions of the camera cover glass. Such occlusions can cause catastrophic failures for various scene understanding tasks such as semantic segmentation, object detection, and depth estimation. Existing camera arrays capture multiple redundant views of a scene to see around thin occlusions. Such multi-camera systems effectively form a large synthetic aperture, which can suppress nearby occluders with a large defocus blur, but significantly increase the overall form factor of the imaging setup. In this work, we propose a monocular single-shot imaging approach that optically cloaks obstructions by emulating a large array. Instead of relying on different camera views, we learn a diffractive optical element (DOE) that performs depth-dependent optical encoding, scattering nearby occlusions while allowing paraxial wavefronts to be focused. We computationally reconstruct unobstructed images from these superposed measurements with a neural network that is trained jointly with the optical layer of the proposed imaging system. We assess the proposed method in simulation and with an experimental prototype, validating that the proposed computational camera is capable of recovering occluded scene information in the presence of severe camera obstruction.
Original languageEnglish (US)
Pages (from-to)1-15
Number of pages15
JournalACM Transactions on Graphics
Volume41
Issue number4
DOIs
StatePublished - Jul 22 2022

Bibliographical note

KAUST Repository Item: Exported on 2022-09-14
Acknowledgements: This work was in part supported by KAUST baseline funding. The DOE design was fabricated in the KAUST Nanofabrication Core Lab. Yuval Bahat was supported by the European Union’s Horizon 2020 research and innovation programme under the Marie SkłodowskaCurie grant agreement No 945422. Felix Heide was supported by an NSF CAREER Award (2047359), a Sony Young Faculty Award, and a Project X Innovation Award.

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design

Fingerprint

Dive into the research topics of 'Seeing through obstructions with diffractive cloaking'. Together they form a unique fingerprint.

Cite this