UAMD-Net: A Unified Adaptive Multimodal Neural Network for Dense Depth Completion about digital_image_processing HOT 2 OPEN

Depth prediction is a critical problem in robotics applications, especially autonomous driving. However, the former usually suffers from overfitting while building cost volume, and the latter has a limited generalization due to the lack of geometric constraint. To solve these problems, we propose a novel multimodal neural network, namely UAMD-Net, for dense depth completion based on the fusion of binocular stereo matching and the weak constrain from the sparse point clouds. Specifically, the sparse point clouds are converted to sparse depth map and sent to the multimodal feature encoder (MFE) with binocular image, constructing a cross-modal cost volume. Then, it will be further processed by the multimodal feature aggregator (MFA) and the depth regression layer. Furthermore, the existing multimodal methods ignore the problem of modal dependence, that is, the network will not work when a certain modal input has a problem. Therefore, we propose a new training strategy called Modal-dropout which enables the network to be adaptively trained with multiple modal inputs and inference with specific modal inputs. Comprehensive experiments conducted on the KITTI depth completion benchmark demonstrate that our method produces robust results and outperforms other state-of-the-art methods.

from digital_image_processing.

from digital_image_processing.