Embedded Neural Network Weights in ROM for Low-Power AI Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI inference solutions, such as GPUs, TPUs, and CPUs, are inefficient and costly for dedicated tasks due to high power consumption, latency, and the need to repeatedly load model weights, while FPGAs require significant programming effort and are not power efficient.
Innovation Solution
A dedicated chip architecture, referred to as models-on-silicon, embeds transformer-based neural network weights and inference architecture directly onto hardware, using sequential read-only memories and custom-built circuits to optimize LLM operations, eliminating the need for repeated weight loading and reducing power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If general-purpose processors (GPUs, TPUs, CPUs) are used for AI inference, then high computational power is achieved, but power consumption and cost increase significantly
Solution Approach 1:
The patent merges the neural network model weights directly into the ROM of the processing circuit, combining storage and computation functions into a single integrated unit. This eliminates the need for separate weight storage memory and repeated data loading, reducing power consumption while maintaining computational power for AI inference tasks.
Solution Approach 2:
The processing circuit serves itself by having the neural network weights embedded within its own structure. The circuit independently performs inference operations using its embedded weights without requiring external memory access or data transfer, thereby reducing energy consumption associated with data movement and external communication.
2Adaptability or versatility
If general-purpose processors are used for AI inference, then computational flexibility is maintained, but execution time and latency increase due to repeated weight loading
Solution Approach 1:
The neural network weights are pre-loaded and embedded into the ROM during circuit fabrication or initialization. This preliminary action eliminates the need for repeated weight loading during inference operations, significantly reducing execution time and latency while allowing the circuit to perform inference tasks immediately upon receiving input data.
3Adaptability or versatility
If FPGAs are used for AI inference, then hardware reconfigurability is achieved, but programming complexity and power consumption increase
Solution Approach 1:
The patent uses a fixed, embedded weight structure in ROM that is simple and cost-effective to implement. Rather than using complex reconfigurable logic like FPGAs, the solution accepts limited adaptability in exchange for dramatically reduced programming complexity and lower power consumption, making it suitable for applications where the neural network model is fixed or changes infrequently.
Data Source
AI summary
A “models-on-silicon” chip can encapsulate Large Language Model weights and inference architecture directly onto the hardware by etching the weights onto the chip and implementing custom circuits to perform operations of a Large Language Model. The weights are stored in sequential read-only memory, and the operations are orchestrated in a feedforward manner. Each line is read at a designated time slot along with the operation that is operating on the data. The architecture eliminates the recurring task of loading weights and the model processing graph onto Graphics Processing Units each time. Moreover, the architecture frees up the need to persistently retrieve weights from memory for each computation, and the data is stored near the circuits performing the operations. Performance is improved, routing is simplified, and data is more quickly accessed. The architecture is cost-effective and can be highly scalable.