Abstract
Optical flow is a regression task where convolutional neural networks (CNNs) have led to major breakthroughs. However, this comes at major computational demands due to the use of cost-volumes and pyramidal representations. This was mitigated by producing flow predictions at quarter the resolution, which are upsampled using bilinear interpolation during test time. Consequently, fine details are usually lost and post-processing is needed to restore them. We propose the Normalized Convolution UPsampler (NCUP), an efficient joint upsampling approach to produce the full-resolution flow during the training of optical flow CNNs. Our proposed approach formulates the upsampling task as a sparse problem and employs the normalized convolutional neural networks to solve it. We evaluate our upsampler against existing joint upsampling approaches when trained end-to-end with a a coarse-to-fine optical flow CNN (PWCNet) and we show that it outperforms all other approaches on the FlyingChairs dataset while having at least one order fewer parameters. Moreover, we test our upsampler with a recurrent optical flow CNN (RAFT) and we achieve state-of-the-art results on Sintel benchmark with ∼ 6% error reduction, and on-par on the KITTI dataset, while having 7.5% fewer parameters (see Figure 1). Finally, our upsampler shows better generalization capabilities than RAFT when trained and evaluated on different datasets.
Original language | English (US) |
---|---|
Title of host publication | VISAPP |
Editors | Giovanni Maria Farinella, Petia Radeva, Jose Braz, Kadi Bouatouch |
Publisher | SciTePress |
Pages | 742-752 |
Number of pages | 11 |
ISBN (Electronic) | 9789897584886 |
State | Published - 2021 |
Event | 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, VISIGRAPP 2021 - Virtual, Online Duration: Feb 8 2021 → Feb 10 2021 |
Publication series
Name | VISIGRAPP 2021 - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications |
---|---|
Volume | 5 |
Conference
Conference | 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, VISIGRAPP 2021 |
---|---|
City | Virtual, Online |
Period | 02/8/21 → 02/10/21 |
Bibliographical note
Funding Information:This work was supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) and Swedish Research Council grant 2018-04673.
Publisher Copyright:
Copyright © 2021 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved.
Keywords
- Joint image upsampling
- Normalized convolution
- Optical flow estimation CNNs
- Spare CNNS
ASJC Scopus subject areas
- Computer Graphics and Computer-Aided Design
- Computer Science Applications
- Computer Vision and Pattern Recognition