Driving Policy Transfer via Modularity and Abstraction

Matthias Muller, Alexey Dosovitskiy, Bernard Ghanem, Vladlen Koltun

Research output: Contribution to conferencePaperpeer-review

45 Scopus citations


End-to-end approaches to autonomous driving have high sample complexity and are difficult to scale to realistic urban driving. Simulation can help end-to-end driving systems by providing a cheap, safe, and diverse training environment. Yet training driving policies in simulation brings up the problem of transferring such policies to the real world. We present an approach to transferring driving policies from simulation to reality via modularity and abstraction. Our approach is inspired by classic driving systems and aims to combine the benefits of modular architectures and end-to-end deep learning approaches. The key idea is to encapsulate the driving policy such that it is not directly exposed to raw perceptual input or low-level vehicle dynamics. We evaluate the presented approach in simulated urban environments and in the real world. In particular, we transfer a driving policy trained in simulation to a 1/5-scale robotic truck that is deployed in a variety of conditions, with no finetuning, on two continents.

Original languageEnglish (US)
Number of pages15
StatePublished - 2018
Event2nd Conference on Robot Learning, CoRL 2018 - Zurich, Switzerland
Duration: Oct 29 2018Oct 31 2018


Conference2nd Conference on Robot Learning, CoRL 2018

Bibliographical note

Funding Information:
Acknowlegdement. This work was partially supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research.

Publisher Copyright:
© CoRL 2018. All rights reserved.


  • Autonomous Driving
  • Sim-to-Real
  • Transfer Learning

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Statistics and Probability


Dive into the research topics of 'Driving Policy Transfer via Modularity and Abstraction'. Together they form a unique fingerprint.

Cite this