Computing-in-Memory Processor Layout for Generative Model Bandwidth Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Portable terminal devices face limitations in running generative models due to memory bandwidth restrictions, which hinder efficient execution of large-scale models.
Innovation Solution
A hardware system architecture is introduced with a first processor and a second computing-in-memory processor, along with a target memory, allowing direct data communication and storage, enhancing data processing and reading capacities for running generative models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a conventional processor architecture is used to run generative models on portable terminal devices, then the device structure remains simple and power consumption is controlled, but the model running speed is slow and execution efficiency is low due to memory bandwidth restrictions
Solution Approach 1:
The processor is divided into two distinct types: a first processor for executing general functions and a second computing-in-memory processor for executing model-based data processing functions. This segmentation allows each processor type to be optimized for its specific purpose, with the computing-in-memory processor dedicated to handling generative model workloads, thereby improving model running speed without requiring complete architectural overhaul of the entire system.
Solution Approach 2:
The computing-in-memory processor is designed to handle multiple model-related operations including executing the second function based on model data, performing data communication with both the first processor and target memory, and running generative models directly. This multi-functional design enables a single specialized processor to handle various AI workloads, improving productivity while controlling the overall device complexity through functional consolidation.
2Productivity
If memory bandwidth is increased to improve model execution efficiency, then the model running speed increases, but the hardware resource requirements and device complexity increase
Solution Approach 1:
The computing-in-memory processor merges computation and memory access functions into a single integrated unit. By combining the processing logic with direct memory access capabilities, the system eliminates the need for high-speed separate memory interfaces, thereby improving model execution efficiency without proportionally increasing hardware resource consumption.
Solution Approach 2:
The computing-in-memory processor acts as an intermediary between the general-purpose first processor and the target memory. It handles model data processing locally, reducing the amount of data that needs to be transmitted over high-bandwidth memory interfaces, thus improving execution efficiency while moderating the memory bandwidth requirements of the overall system.
3Productivity
If a computing-in-memory processor is introduced to improve model running speed, then the productivity increases, but the device complexity and processing architecture complexity increase
Solution Approach 1:
The computing-in-memory processor is extracted as a separate, dedicated component from the general-purpose processor. This extraction isolates the complex model execution logic into a specialized unit, allowing the first processor to remain simple while the second processor handles the complexity of generative model workloads, thus improving productivity while managing overall system complexity through functional separation.
Data Source
AI summary
Examples of the disclosure relate to a terminal device, a method and apparatus for processing data based on a model, and a storage medium. The terminal device includes: a first processor, a second processor and a target memory that are communicatively connected; where the first processor is configured to execute a first function; the second processor is a computing-in-memory processor, and is configured to perform data communication with the first processor and execute a second function based on data obtained from communication, the second function is a data processing function based on the model, and the second function is different from the first function; and the target memory is configured to perform data communication with the second processor and store data obtained by running the second processor.