Overlay Code Execution for AI Models in Limited Local Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The limited storage capacity of local memory in embedded systems, due to cost and power consumption constraints, cannot accommodate the increasing storage needs of rapidly evolving artificial intelligence and big data algorithms, leading to challenges in storing code segments and data segments effectively.
Innovation Solution
A method and system that flexibly stores code and data segments of a target model by using a main memory to load executable files and call information into a local memory, divided into application and stack areas, with a dynamic overlay manager managing data transfer using DMA engines, and optionally employing a double buffer architecture for efficient execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If local memory size is increased to store more code segments and data segments, then storage capacity is improved, but cost and power consumption increase
Solution Approach 1:
The patent divides the code execution process into multiple segments, loading only the currently needed code segment and its corresponding data segment into the local memory at a time. The local memory is segmented into a code segment storage area and a data segment storage area, with each area independently managing its content. This segmentation allows the system to maintain a small local memory size while still supporting execution of large programs by loading portions sequentially.
2Quantity of substance
If local memory size is increased to store more code segments and data segments, then storage capacity is improved, but device cost increases
Solution Approach 1:
The patent implements segmentation by dividing both the code and data into manageable segments that are loaded on-demand. The local memory is structured with separate areas for code segments and data segments, allowing efficient utilization of limited storage space. This approach enables the system to support large-scale AI algorithms without requiring proportionally large local memory, thereby reducing manufacturing costs.
3Speed
If code segments are pre-loaded into local memory, then execution speed is improved, but memory usage increases
Solution Approach 1:
The patent applies preliminary action by pre-loading the next code segment and its corresponding data segment into the local memory while the current segment is being executed. This overlapping preparation ensures that when the current segment finishes, the next segment is already ready in memory, maintaining execution speed without requiring all segments to be pre-loaded. The compiler generates segment information that guides this pre-loading process.
Solution Approach 2:
The patent implements dynamic memory management where the content of local memory changes over time based on execution needs. Code segments and data segments are dynamically loaded and unloaded from local memory according to the execution flow. The system transitions between different memory states, loading new segments as old ones complete execution, thereby optimizing the balance between execution speed and memory utilization.
4Loss of time
If all code segments are stored in local memory, then access latency is reduced, but storage space requirements increase
Solution Approach 1:
The patent divides the program into multiple code segments and data segments, with only the active segment residing in local memory at any given time. The segmentation allows the system to achieve fast access latency for the currently executing code while maintaining a small local memory footprint. The compiler generates segment information that enables the system to quickly switch between segments without requiring all code to be simultaneously available in local memory.
Data Source
AI summary
A code execution method and a data processing device and system are provided. The method includes storing, through a data processing device, multiple executable files corresponding to multiple functions and multiple call information obtained by parsing a target model through a compile into a main memory and running, through the data processing device, the target model. The running, through the data processing device, the target model includes the following: loading an executable file from the main memory into an application code area of a local memory based on a running sequence of a plurality of functions and loading corresponding call information into a stack area of the local memory; and running the executable file in the application code area based on the call information in the stack area and transferring a running result from the local memory to the main memory after the running is completed.