Neural Processing Layout with Line-Based Scheduling for Edge Generative AI
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Edge devices face challenges in efficiently processing complex generative AI models due to resource limitations, including limited on-chip memory and processing power, which lead to excessive power consumption and inefficient memory access, especially when handling branched and residual connections in neural networks.
Innovation Solution
A heterogeneous neural processing system with specialized and general-purpose processing units collaboratively executes autoencoder and denoising operations, employing line-based depth-first scheduling and H-reuse caching to optimize memory access and latency, while handling branched and residual connections efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If complex generative AI models are deployed on edge devices, then AI functionality and user experience are improved, but power consumption and memory usage increase excessively
Solution Approach 1:
The patent segments the neural network into multiple operators and divides computation into micro-batches, allowing partial processing and early termination when confidence thresholds are met. This segmentation enables the system to process only necessary portions of data, reducing overall power consumption while maintaining AI functionality on edge devices.
Solution Approach 2:
The system performs partial computation by processing micro-batches and stopping early when confidence thresholds are satisfied. This partial action approach avoids completing unnecessary computations, significantly reducing power consumption while still delivering acceptable AI inference results for practical applications.
2Adaptability or versatility
If complex generative AI models are deployed on edge devices, then AI functionality is improved, but on-chip memory requirements exceed available capacity
Solution Approach 1:
The patent segments feature maps into multiple tiles and processes them in a depth-first manner, loading only necessary tile portions into on-chip memory at any given time. This segmentation strategy reduces peak memory requirements from needing to hold entire feature maps to holding only current processing tiles, enabling complex models to run on edge devices with limited memory.
Solution Approach 2:
The system introduces a new dimension of processing by dividing feature maps into spatial tiles and processing them in a depth-first search pattern through the network layers. This dimensional transformation allows the system to manage memory usage by processing data in manageable chunks rather than loading complete high-dimensional feature maps into limited on-chip memory.
3Device complexity
If traditional neural network processing is used on edge devices, then implementation simplicity is maintained, but external memory access frequency increases
Solution Approach 1:
The patent implements ping-pong buffers that pre-load the next data segment while the current segment is being processed. This preliminary action ensures that data is ready in on-chip memory before it is needed, eliminating external memory access delays and reducing latency without significantly increasing architectural complexity.
Solution Approach 2:
The system maintains continuous processing by overlapping data transfer and computation operations. While one data segment is being processed, the next segment is being loaded into buffers, ensuring that the processing pipeline never stalls and external memory access is minimized. This continuity eliminates idle time and reduces overall memory access latency.
4Device complexity
If conventional memory access patterns are used, then implementation simplicity is maintained, but bandwidth utilization is inefficient
Solution Approach 1:
The patent segments feature maps into tiles and processes them in a depth-first manner, accessing memory in a systematic pattern that maximizes spatial locality. This segmentation enables the memory system to efficiently cache and reuse data blocks, improving bandwidth utilization without requiring complex memory access patterns or breaking conventional memory hierarchies.
Data Source
AI summary
A heterogeneous neural processing system includes a first processor configured to execute encoding and decoding operations of an autoencoder, and a second processor configured to execute task-specific neural network operations with iterative processing. The processors execute computational tasks with synchronized data exchange to implement generative AI models. The first processor processes feature maps divided into lines of data with line-based depth-first scheduling, caches data in activation memory, and selects operations deeper in network hierarchy while handling branched inputs, outputs, and residual connections. An H-reuse cache stores boundary pixels between spatial segments, enabling concurrent execution of convolution and element-wise operations. A neural network conditioning device analyzes models to identify layer dependencies, applies search space constraints, performs iterative searches to generate fusion schedules, and selects optimal schedules based on external memory access and execution latency.