Fusion NPU Scheduling for Heterogeneous Sensor Data Locality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial neural networks for autonomous vehicles face inefficiencies in processing vast amounts of data from heterogeneous sensors due to the need for dedicated networks for each sensor type, leading to prolonged processing times.
Innovation Solution
A neural processing unit (NPU) is designed to efficiently process data from multiple sensors through a fusion neural network, utilizing a controller, processing elements, and on-chip memory to manage operations based on data locality information, and includes a special function unit (SFU) to perform convolution operations and reduce power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If dedicated artificial neural networks are implemented for each sensor type, then each sensor data can be processed with specialized optimization, but processing time increases considerably due to multiple separate networks
Solution Approach 1:
The patent merges multiple dedicated neural networks into a single fusion neural network that processes heterogeneous sensor data (camera, radar, LiDAR, ultrasonic) together. This consolidation reduces the number of separate processing pipelines while maintaining specialized handling for each sensor type through dedicated processing units within the unified network, thereby decreasing overall processing time.
Solution Approach 2:
The fusion neural network is designed with multi-functional capability to handle diverse sensor data types simultaneously. It incorporates universal processing structures that can adapt to different sensor inputs (image data, point cloud data, radar signals) while maintaining optimized processing paths for each type, enabling one network to perform the work of multiple dedicated networks.
2Measurement precision
If multiple heterogeneous sensors are mounted on the vehicle for comprehensive data collection, then sensing coverage and accuracy are improved, but data volume increases leading to longer processing times
Solution Approach 1:
The patent combines data from multiple heterogeneous sensors (camera, radar, LiDAR, ultrasonic sensors) into a unified fusion neural network processing pipeline. This merging approach processes all sensor inputs simultaneously through coordinated processing units rather than sequentially through separate networks, improving productivity despite increased data volume.
Solution Approach 2:
The fusion neural network is segmented into dedicated processing units for different sensor types (image processing unit for camera, point cloud processing unit for LiDAR, radar signal processing unit). Each unit processes specific sensor data with optimized algorithms, then results are integrated. This segmentation maintains high processing efficiency while handling diverse sensor inputs.
3Productivity
If a fusion neural network processes heterogeneous sensor data, then processing efficiency is improved, but device complexity increases due to integration requirements
Solution Approach 1:
The fusion neural network is divided into distinct processing units: image processing unit for camera data, point cloud processing unit for LiDAR, radar signal processing unit, and ultrasonic signal processing unit. Each unit has specialized architecture optimized for its sensor type, reducing the complexity of designing a single unified structure while maintaining high processing efficiency through modular organization.
Solution Approach 2:
The patent introduces a data fusion layer that acts as an intermediary between the specialized processing units and the final decision-making layer. This mediator integrates outputs from different sensor processing units, reconciles data in different formats and coordinate systems, and prepares unified input for subsequent processing stages, simplifying the overall integration complexity.
Data Source
AI summary
A neural processing unit (NPU) includes a controller including a scheduler, the controller configured to receive from a compiler a machine code of an artificial neural network (ANN) including a fusion ANN, the machine code including data locality information of the fusion ANN, and receive heterogeneous sensor data from a plurality of sensors corresponding to the fusion ANN; at least one processing element configured to perform fusion operations of the fusion ANN including a convolution operation and at least one special function operation; a special function unit (SFU) configured to perform a special function operation of the fusion ANN; and an on-chip memory configured to store operation data of the fusion ANN, wherein the scheduler is configured to control the at least one processing element and the on-chip memory such that all operations of the fusion ANN are processed in a predetermined sequence according to the data locality information.