Today’s Patent – Camera-radar sensor fusion using local attention mechanism

The said invention (US20230213643A1) was invented by Jyh-Jing,HwangHenrik, KretzschmarDragomir Anguelov patented by USPTO on 05-01-2022. Currently, it stands assigned to Waymo Llc.

Traditional neural networks process data from individual sensors, such as camera sensors or radar sensors, to identify objects and characterize scenes.

However, these approaches often lack robustness, especially in challenging scenarios such as inclement weather or sensor malfunctions. Existing methods for fusing data from camera and radar sensors frequently fail to maintain proper spatial alignment between the data collected by these sensors, leading to inaccuracies in object detection, depth prediction, and characterization. This misalignment hampers precision and limits the effectiveness of such systems in critical applications, such as autonomous vehicle navigation.

The disclosed invention presents a robust and fault-tolerant sensor fusion system that integrates image data from camera sensors and radar data using attention mechanisms to enhance depth predictions and spatial alignment. This system generates a fused point cloud that retains high-resolution image features and accurately incorporates object distance and velocity information from radar data. Unlike conventional methods, this approach ensures accurate object detection and classification, even in adverse conditions or when one sensor fails. When deployed in autonomous vehicles, the system enables precise decision-making for safe and smooth trajectory planning, particularly in long-range scenarios such as highway driving. This advancement significantly improves accuracy, precision, and reliability over prior approaches.