Neural Network Subgraph Compilation for Pipelined Compute Circuits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network architectures face significant latency and throughput issues due to memory constraints, particularly when processing high-resolution data, as the memory demands exceed available on-chip resources, leading to substantial time penalties from off-chip memory access and non-linear latency.
Innovation Solution
The method involves grouping neural network layers into subgraphs based on profiled compute times and equalizing compute times between groups, allowing for pipelined execution by multiple compute circuits, and decomposing input activations into tiles to reduce off-chip memory access, enabling efficient processing and linear scaling of performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If neural network layers are processed sequentially using available on-chip memory resources, then memory constraints are managed, but processing time increases substantially due to off-chip memory access
Solution Approach 1:
The patent divides the neural network computation into multiple independent stages or subgraphs, where each stage processes a portion of the network. This segmentation allows different stages to be executed in parallel on separate compute circuits, reducing the need for sequential off-chip memory access and enabling more data to remain in on-chip memory throughout the processing pipeline.
Solution Approach 2:
The patent transitions from a single-dimensional sequential processing model to a multi-dimensional parallel processing architecture. By organizing compute circuits in multiple dimensions and enabling parallel execution of different network stages, the system increases effective memory bandwidth and reduces processing time without requiring proportional increases in on-chip memory capacity.
2Measurement precision
If high-resolution data is processed, then data quality and detail are improved, but memory demands exceed available on-chip resources
Solution Approach 1:
The patent segments the high-resolution data processing into multiple stages that can be pipelined across different compute circuits. This allows different portions of the high-resolution data to be processed simultaneously at different stages, maintaining data quality while managing memory requirements through parallel processing rather than requiring all data to reside in on-chip memory simultaneously.
Solution Approach 2:
The patent implements preliminary data preparation and tiling operations that organize high-resolution input data into smaller manageable tiles before processing. This preliminary action allows the data to be staged in on-chip memory in a structured way that enables efficient parallel processing while minimizing the peak memory requirements for handling high-resolution data.
3Quantity of substance
If off-chip memory access is used to handle large activations, then memory capacity requirements are met, but substantial time penalties are incurred
Solution Approach 1:
The patent segments the neural network into multiple subgraphs that can be processed in parallel pipelines. This segmentation enables intermediate activations to remain in on-chip memory across stage boundaries, eliminating the need for frequent off-chip memory access and maintaining high throughput even when processing large activations that would otherwise require extensive off-chip memory.
Solution Approach 2:
The patent establishes continuous data flow through pipelined processing stages, where output from one stage becomes input to the next without interruption. This continuity of useful action keeps compute circuits constantly utilized and minimizes idle time waiting for memory operations, thereby maintaining high productivity while managing large activation sizes through efficient on-chip memory utilization.
Data Source
AI summary
Processing of a neural network specification includes gathering first layers of a neural network graph into groups of layers based on profiled compute times of the layers and equalized compute times between the groups. Each group is a subgraph of one or more of the layers of the neural network. The neural network graph is compiled into instructions for pipelined execution of the neural network graph by compute circuits. The compiling includes designating, for each first subgraph of the subgraphs having output activations that are input activations of a second subgraph of the subgraphs, operations of the first subgraph to be performed by a first compute circuit and operations of the second subgraph to be performed by a second compute circuit. The compute circuits are configured to execute the instructions.