Single-photon 3D imaging with deep sensor fusion

David B. Lindell, Matthew O'Toole, Gordon Wetzstein

Research output: Contribution to journalArticlepeer-review

119 Scopus citations


Sensors which capture 3D scene information provide useful data for tasks in vehicle navigation, gesture recognition, human pose estimation, and geometric reconstruction. Active illumination time-of-flight sensors in particular have become widely used to estimate a 3D representation of a scene. However, the maximum range, density of acquired spatial samples, and overall acquisition time of these sensors is fundamentally limited by the minimum signal required to estimate depth reliably. In this paper, we propose a data-driven method for photon-efficient 3D imaging which leverages sensor fusion and computational reconstruction to rapidly and robustly estimate a dense depth map from low photon counts. Our sensor fusion approach uses measurements of single photon arrival times from a low-resolution single-photon detector array and an intensity image from a conventional high-resolution camera. Using a multi-scale deep convolutional network, we jointly process the raw measurements from both sensors and output a high-resolution depth map. To demonstrate the efficacy of our approach, we implement a hardware prototype and show results using captured data. At low signal-to-background levels, our depth reconstruction algorithm with sensor fusion outperforms other methods for depth estimation from noisy measurements of photon arrival times.
Original languageEnglish (US)
Pages (from-to)1-12
Number of pages12
JournalACM Transactions on Graphics
Issue number4
StatePublished - Aug 10 2018
Externally publishedYes

Bibliographical note

KAUST Repository Item: Exported on 2022-06-09
Acknowledgements: This project was supported by a Stanford Graduate Fellowship, a Banting Postdoctoral Fellowship, an NSF CAREER Award (IIS 1553333), a Terman Faculty Fellowship, a Sloan Fellowship, by the KAUST Office of Sponsored Research through the Visual Computing Center CCF grant, and by the DARPA REVEAL program.
This publication acknowledges KAUST support, but has no KAUST affiliated authors.


Dive into the research topics of 'Single-photon 3D imaging with deep sensor fusion'. Together they form a unique fingerprint.

Cite this