Autonomous Driving: from Sensor Fusion to End-to-End Control
DMFuser introduces a novel approach to autonomous driving by integrating multi-task learning with sensor fusion for perception and control tasks. The model uses an attention-CNN-based sensor fusion module that combines both RGB and depth camera data, resulting in superior feature extraction. This attention mechanism operates on global and local levels, ensuring no crucial details are missed.
The perception module processes this sensor data into a Semantic Depth Cloud (SDC), a 23-layer bird’s eye view map that improves scene understanding. For the control module, DMFuser uses the fused data to predict waypoints and vehicular controls (throttle, brake, steering) dynamically, employing a GRU-based network for real-time adjustments. One key innovation is the use of knowledge distillation, where single-task teachers provide insights to guide the multi-task student, refining predictions and reducing noisy signals. The result is a more accurate and efficient navigation system, validated through extensive testing in the CARLA simulator.
With future improvements like reinforcement learning and explainable AI, DMFuser offers a promising direction for autonomous vehicle navigation
Here are the links:
Related posts
