Neural Network Processor On-Chip Memory Context Switching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network processing systems face performance bottlenecks due to reliance on off-chip memory for weight values, leading to increased memory latency and reduced computational efficiency, especially when CPUs and GPUs are used for parallel computations.
Innovation Solution
Implementing a neural network processor with on-chip memory to store weight values, allowing direct data transfer between computation engines and reducing memory transactions, thereby minimizing memory latency and enhancing computational efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If off-chip memory is used to store weight values, then memory capacity is sufficient, but memory latency increases and computational efficiency decreases
Solution Approach 1:
The patent segments memory into on-chip memory (for frequently accessed weight values) and off-chip memory (for bulk storage). This segmentation allows the system to store weight values in on-chip memory when available, reducing latency, while maintaining sufficient total capacity through off-chip memory.
Solution Approach 2:
The patent introduces on-chip memory as an intermediary between off-chip memory and computation engines. This intermediary buffer stores weight values closer to the computation engines, reducing the time penalty for memory access while maintaining the capacity benefits of off-chip storage.
2Loss of time
If on-chip memory is used to store weight values, then memory latency is reduced, but device complexity increases
Solution Approach 1:
The patent implements dynamic memory allocation where on-chip memory is allocated for weight values when computation engines need them, and freed when not needed. This dynamic approach allows the system to optimize for low latency during computation while managing complexity through flexible resource allocation.
Solution Approach 2:
The patent changes the parameter of memory location from fixed (either on-chip or off-chip) to variable, allowing weight values to be stored in on-chip memory when available and off-chip memory when not available. This parameter change enables the system to adapt to different operational requirements.
3Ease of operation
If CPUs are used for neural network computations, then sequential processing is efficient, but parallel computation performance is poor
Solution Approach 1:
The patent designs a neural network processing system that can handle both sequential and parallel computations efficiently. The computation engines are configured to perform parallel matrix multiplications while the memory subsystem supports both sequential access patterns (for CPU-like operations) and parallel access patterns (for GPU-like operations), making the system universal in its computational capabilities.
4Productivity
If GPUs are used for neural network computations, then parallel computation performance is improved, but memory access efficiency decreases
Solution Approach 1:
The patent applies preliminary action by pre-loading weight values into on-chip memory before parallel computation begins. This allows computation engines to access frequently used weight values quickly during parallel operations, reducing memory access time penalties while maintaining high parallel computation performance.
Data Source
AI summary
Provided are systems, methods, and integrated circuits neural network processor that can execute a fast context switch between one neural network and another. In various implementations, a neural network processor can include a plurality of memory banks storing a first set of weight values for a first neural network. When the neural network processor receives first input data, the neural network processor can compute a first result using the first set of weight values and the first input data. While computing the first result, the neural network processor can store, in the memory banks, a second set of weight values for a second neural network. When the neural network processor receives second input data, the neural network processor can compute a second result using the second set of weight values and the second input data, where the computation occurs upon completion of computation of the first result.