Mamba State Space Hardware Embedding for Low-Latency Edge AI
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for implementing large language models (LLMs) on edge devices are hindered by high computation costs, power consumption, and latency issues, making them inefficient for real-time applications due to the need for frequent loading of model weights and data transfer.
Innovation Solution
A dedicated chip architecture, referred to as 'models-on-silicon,' embeds LLM weights and inference architecture directly onto hardware, utilizing sequential read-only memories and custom-built circuits to perform operations efficiently, eliminating the need for repeated weight loading and reducing power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If LLM weights and data are stored in external memory and loaded repeatedly, then model flexibility and adaptability are maintained, but power consumption increases and inference time increases
Solution Approach 1:
The patent embeds LLM weights directly within the inference circuitry, nesting the weight storage functionality inside the processing unit itself. This eliminates the need for external memory loading operations, thereby reducing power consumption while accepting increased on-chip complexity to achieve the nested integration of weights and computation units.
2Productivity
If LLM weights are loaded from external memory for each inference task, then model versatility is maintained, but inference time increases due to repeated loading
Solution Approach 1:
The patent performs preliminary action by pre-loading and embedding LLM weights into the inference circuitry before inference tasks begin. This advance preparation eliminates the need for repeated weight loading during inference, significantly improving inference speed while requiring complex hardware architecture to accommodate the embedded weights.
3Loss of energy
If general-purpose computing hardware is used for LLM inference, then device versatility is maintained, but computation cost and power consumption increase
Solution Approach 1:
The patent extracts the weight storage and retrieval functionality from general-purpose memory systems and embeds it directly within the inference circuitry. This extraction creates a specialized hardware architecture optimized for LLM inference, improving energy efficiency while reducing adaptability to other models compared to general-purpose hardware.
4Loss of time
If LLM weights are frequently transferred between memory and processor, then computation accuracy is maintained, but latency increases
Solution Approach 1:
The patent merges the weight storage memory and the inference processor into a unified embedded architecture. By combining these previously separate components into a single integrated system, the patent eliminates data transfer operations between memory and processor, thereby reducing inference latency while increasing the complexity of the merged hardware structure.
Data Source
AI summary
A state space model with selective updates, also referred to as a Mamba-based block, in a Mamba-based model can be embedded onto a silicon chip. Specialized hardware modules in a models-on-silicon chip, such as an optimized selective scan unit and an optimized 1D convolution unit, can perform the operations of the selective state space model of the Mamba-based model. These modules individually and collectively enhance processing speed, power efficiency, and overall performance. The parameters such as weights of the Mamba-based model are arranged in a sequential order in one or more sequential read memories according to a predetermined timing sequence. By embedding the selective state space model onto the models-on-silicon architecture, which excels in managing larger input context sizes, this solution transforms the Mamba-based model into a highly viable and efficient option for AI tasks being performed on resource-constrained devices.