Neural Processor Spatial Cropping for Data Access Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural processor circuits face performance bottlenecks due to slow data access, which increases power consumption and reduces efficiency in performing machine learning operations, especially when relying solely on central processing units (CPUs) for various configurations of neural networks.
Innovation Solution
A system-on-a-chip circuit incorporating a neural processor with multiple neural engines, a data processor DMA circuit, and a buffer that supports arbitrary spatial cropping in both dimensions, allowing for efficient data fetching and formatting, realignment, and broadcasting to neural engines, thereby reducing reliance on system memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is fetched from system memory for neural network operations, then computation can be performed, but data access becomes a bottleneck reducing performance and increasing power consumption
Solution Approach 1:
The patent implements spatial cropping operations that pre-process and prepare data in advance by determining crop regions and offsets before neural network computation. The compiler analyzes the computational graph and identifies cropping operations that can be performed beforehand, allowing data to be prepared and staged for faster access during the actual computation phase, thereby reducing data access bottlenecks.
Solution Approach 2:
The patent segments the data processing pipeline into distinct stages: spatial cropping operations that extract relevant regions from input data, followed by neural network computation on the cropped data. This segmentation allows the system to process only the necessary portions of data rather than handling complete datasets, reducing the overall data access burden and improving computation efficiency.
2Speed
If spatial cropping operations are implemented to improve data access efficiency, then data handling speed increases, but system complexity increases due to additional processing steps
Solution Approach 1:
The patent implements a universal spatial cropping mechanism that can be applied across different neural network configurations and data types. The cropping circuit and compiler integration provide a multi-functional solution that handles various cropping scenarios (spatial, temporal, channel-wise) through a unified framework, avoiding the need for separate specialized circuits for each cropping type and thus managing complexity while maintaining versatility.
Solution Approach 2:
The patent introduces a compiler as an intermediary layer that translates high-level neural network operations into optimized machine code with embedded spatial cropping instructions. This intermediary handles the complexity of cropping operation generation and optimization, allowing the hardware circuits to execute straightforward, pre-computed cropping operations without needing to understand or manage the complex logic of when and how to crop data.
3Adaptability or versatility
If arbitrary spatial cropping is supported in both x and y dimensions, then adaptability to different neural network configurations improves, but processing complexity increases
Solution Approach 1:
The patent implements dynamic buffer realignment capabilities that automatically adjust to different cropping configurations. The buffer management system dynamically reconfigures data layout and alignment based on the specific cropping operations required for each neural network layer, allowing arbitrary cropping in both x and y dimensions while maintaining efficient data access patterns. This dynamic adaptation eliminates the need for static, complex hardware configurations for each possible cropping scenario.
Solution Approach 2:
The patent utilizes parameter changes in the compiler-generated code to control cropping behavior. By modifying parameters such as crop offsets, crop sizes, and data layout configurations in the compiled task descriptors, the system achieves high adaptability to different neural network configurations. These parameter changes allow the same hardware circuit to efficiently handle diverse cropping requirements without increasing physical circuit complexity, as the flexibility is achieved through configurable parameters rather than additional hardware components.
Data Source
AI summary
An SoC circuit includes a neural processor circuit coupled to a CPU. The neural processor circuit includes neural engines, a data processor DMA circuit, a system memory, and a data processor circuit. The CPU is configured to execute a compiler, which is in turn configured to determine to perform a mode of spatial cropping and the associated crop offset. The neural processor circuit is configured to support arbitrary cropping in the x and y dimensions. The compiler is configured to generate task descriptor(s), the task descriptor(s) distributed to components of the neural processor circuit. The data processor DMA circuit is configured to fetch and format data corresponding to the crop from a source to the buffer. The buffer is configured to realign the data according to the crop origin for broadcast to the neural engines. The neural engines is configured to perform a computation operation which uses the cropped data.