Neural Network Partitioning for Latency Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks face significant computational demands and latency issues due to the large number of weights required for complex tasks like natural language processing and autonomous navigation, which exceed the capacity of on-chip memory in processing integrated circuit devices, leading to prolonged latencies from frequent system memory access.
Innovation Solution
The neural network model is partitioned for serial execution across multiple processing cores in a pipelined fashion, allowing each core to cache weights locally and reducing the need for system memory access, with optimization techniques balancing execution latencies and adjusting partitions based on compute load, weight usage, and tensor transfer sizes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If neural network weights are stored in system memory due to exceeding on-chip memory capacity, then the neural network can perform complex tasks, but execution latency increases significantly
Solution Approach 1:
The patent divides the neural network computation into multiple partitions, each assigned to different processing cores. Each partition processes a subset of weights and computations, allowing parallel execution and reducing the time any single core needs to wait for system memory access. This segmentation transforms a single long-latency computation into multiple shorter parallel computations.
Solution Approach 2:
The patent introduces a new dimension of parallelism by distributing neural network partitions across multiple processing cores simultaneously. Instead of sequential execution on a single core, the system executes multiple partitions in parallel across different cores, effectively adding a temporal parallelism dimension that reduces overall execution latency despite weights residing in system memory.
2Productivity
If neural network is executed on multiple processing cores in parallel, then throughput improves, but execution latency increases due to coordination overhead
Solution Approach 1:
The patent performs preliminary partitioning of the neural network into discrete computation units before execution. Each partition is pre-configured with its required weights and computation operations, so that during parallel execution, processing cores can immediately begin computation without coordination overhead. This preliminary organization eliminates runtime synchronization delays.
Solution Approach 2:
The patent creates multiple copies of the same neural network partition across different processing cores, allowing identical computations to execute in parallel on different input data batches. This copying strategy enables throughput scaling while maintaining consistent latency characteristics across parallel executions.
3Stability of the object's composition
If partition boundaries are adjusted to balance execution latency, then latency uniformity improves, but system complexity increases
Solution Approach 1:
The patent implements a feedback mechanism where the system measures actual execution latency of each partition and uses this information to dynamically adjust partition boundaries. The latency measurements feed back into the partitioning algorithm, which iteratively refines partition configurations to achieve balanced execution times across all processing cores, automating the complexity management.
Solution Approach 2:
The patent changes the partitioning parameters from fixed computation-based divisions to latency-based divisions. By measuring actual execution time and using latency as the primary partitioning parameter rather than purely computational metrics, the system automatically adapts to varying workload characteristics and achieves balanced execution without manual intervention.
Data Source
AI summary
Techniques to partition a neural network model for serial execution on multiple processing integrated circuit devices are described. An initial partitioning of the model into multiple partitions each corresponding to a processing integrated circuit device is performed. For each partition, an execution latency is calculated by aggregating compute clock cycles to perform computations in the partition, and weight loading clock cycles determined based on a number of weights used in the partition. The amount of data being outputted from the partition is also determined. The partitions can be adjusted by moving computations from a source partition to a target partition to change execution latencies of the partitions and the amount of data being transferred between partitions.