PanoVerse: automatic generation of stereoscopic environments from single indoor panoramic images for Metaverse applications

Giovanni Pintore, Alberto Jaspe-Villanueva, Markus Hadwiger, Enrico Gobbetti, Jens Schneider, Marco Agus

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

We present a novel framework, dubbed PanoVerse, for the automatic creation and presentation of immersive stereoscopic environments from a single indoor panoramic image. Once per 360° shot, a novel data-driven architecture generates a fixed set of panoramic stereo pairs distributed around the current central view-point. Once per frame, directly on the HMD, we rapidly fuse the precomputed views to seamlessly cover the exploration workspace. To realize this system, we introduce several novel techniques that combine and extend state-of-the art data-driven techniques. In particular, we present a gated architecture for panoramic monocular depth estimation and, starting from the re-projection of visible pixels based on predicted depth, we exploit the same gated architecture for inpainting the occluded and disoccluded areas, introducing a mixed GAN with self-supervised loss to evaluate the stereoscopic consistency of the generated images. At interactive rates, we interpolate precomputed panoramas to produce photorealistic stereoscopic views in a lightweight WebXR viewer. The system works on a variety of available VR headsets and can serve as a base component for Metaverse applications. We demonstrate our technology on several indoor scenes from publicly available data.

Original languageEnglish (US)
Title of host publicationProceedings - Web3D 2023
Subtitle of host publication28th International Conference on Web3D Technology
EditorsStephen N. Spencer
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9798400703249
DOIs
StatePublished - Oct 9 2023
Event28th International Conference on Web3D Technology, Web3D 2023 - San Sebastian, Spain
Duration: Oct 9 2023Oct 11 2023

Publication series

NameProceedings - Web3D 2023: 28th International Conference on Web3D Technology

Conference

Conference28th International Conference on Web3D Technology, Web3D 2023
Country/TerritorySpain
CitySan Sebastian
Period10/9/2310/11/23

Bibliographical note

Publisher Copyright:
© 2023 Owner/Author.

Keywords

  • Data-driven Methods
  • Immersive Stereoscopic Exploration
  • Indoor Environments
  • Metaverse Applications
  • Omnidirectional Images
  • WebXR

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Graphics and Computer-Aided Design
  • Human-Computer Interaction
  • Software

Fingerprint

Dive into the research topics of 'PanoVerse: automatic generation of stereoscopic environments from single indoor panoramic images for Metaverse applications'. Together they form a unique fingerprint.

Cite this