Neural Inference Chip On-Chip Memory Parallelism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural inference technologies face inefficiencies in terms of time, space, and energy consumption due to reliance on off-chip memory access and lack of parallel processing capabilities, leading to increased latency and energy expenditure.
Innovation Solution
The development of neural inference chips with on-chip memory for storing neural network models and data, along with an on-chip network connecting multiple neural cores, enables parallel processing and reduces the need for off-chip memory access by distributing computation and memory locally among cores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If off-chip memory access is used for storing neural network models and data, then device complexity is reduced, but processing speed decreases and energy consumption increases
Solution Approach 1:
The patent merges memory and computation by integrating on-chip memory (model memory and data memory) directly with the neural core array. This allows neural network models and input data to be stored and processed simultaneously on the same chip, eliminating the need for frequent off-chip memory accesses and thereby improving processing speed while maintaining manageable device complexity through systematic memory organization.
Solution Approach 2:
The patent segments the memory system into distinct on-chip components: model memory for storing neural network parameters, data memory for storing input data, and intermediate buffers for activation values. This segmentation allows each memory type to be optimized for its specific function and enables parallel access patterns that improve overall processing speed without overwhelming system complexity.
2Device complexity
If off-chip memory access is used for storing neural network models and data, then device complexity is reduced, but energy consumption increases
Solution Approach 1:
By merging memory and computation on-chip, the patent eliminates energy-intensive data transfers between off-chip memory and the processor. The integrated architecture keeps neural network models and input data locally available, dramatically reducing the energy required for data movement while organizing the memory system in a manageable way.
Solution Approach 2:
The patent introduces on-chip intermediate buffers and data memory as mediators between the external interface and the neural core array. These intermediaries cache frequently accessed data and activation values, reducing the frequency of high-energy off-chip memory accesses while maintaining a relatively simple overall memory architecture.
3Speed
If multiple neural cores are used for parallel processing, then processing speed increases, but device complexity increases
Solution Approach 1:
The patent segments the neural processing system into multiple independent neural cores, each capable of processing specific neurons or neuron groups in parallel. This segmentation enables speedup through parallelism while keeping each core relatively simple. The memory system is also segmented into model memory and data memory, each serving specific core groups, which manages interconnection complexity systematically.
Solution Approach 2:
The patent designs universal neural cores that can be configured to perform different neural network operations through programmable interconnection patterns. Each core can handle multiple neuron types and connection patterns, reducing the need for specialized hardware for each function. This multi-functionality allows parallel processing with increased speed while controlling device complexity through standardized, reconfigurable core designs.
4Speed
If on-chip memory is used for storing neural network models and data, then processing speed increases, but device complexity increases
Solution Approach 1:
The patent merges memory and computation by integrating on-chip model memory and data memory directly with the neural core array. This integration improves processing speed by keeping data locally available, while the systematic organization of memory types and their dedicated interfaces manages the complexity of the integrated architecture.
Solution Approach 2:
The patent segments the on-chip memory system into distinct functional units: model memory for neural network parameters, data memory for input data, and intermediate buffers for activations. This segmentation allows each memory type to be optimized independently and accessed by specific core groups, improving speed while managing integration complexity through modular design.
Data Source
AI summary
Neural inference chips and cores adapted to provide time, space, and energy efficient neural inference via parallelism and on-chip memory are provided. In various embodiments, the neural inference chips comprise: a plurality of neural cores interconnected by an on-chip network; a first on-chip memory for storing a neural network model, the first on-chip memory being connected to each of the plurality of cores by the on-chip network; a second on-chip memory for storing input and output data, the second on-chip memory being connected to each of the plurality of cores by the on-chip network.