Hierarchical Weight Loading for Deep Learning Model Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The limited memory of a central processing unit and graphics processing unit in a single machine or graphics card restricts the simultaneous deployment of multiple deep learning models, leading to significant storage space occupation during model execution.
Innovation Solution
A model data processing system that determines a target execution unit based on a predetermined execution order, utilizing a storage device set to load and store weight data in a hierarchical manner, allowing efficient use of storage resources across multiple storage devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all deep learning models are deployed on a single machine or graphics card, then model execution capability is improved, but storage space occupation increases significantly
Solution Approach 1:
The patent segments the storage system into multiple levels: first storage device (fast memory), second storage device (secondary storage), and third storage device (external or network storage). Weight data is divided and stored across these segments based on access frequency and execution order, allowing models to be executed while reducing peak storage requirements on any single device.
Solution Approach 2:
The system performs preliminary actions by pre-loading weight data into the first storage device according to the predetermined execution order of execution units. This ensures that required weight data is already available in fast memory when needed, eliminating the need to hold all model weights simultaneously in high-speed memory.
Solution Approach 3:
The second storage device acts as an intermediary between the first storage device and the third storage device. It temporarily holds weight data during the transfer process, enabling efficient data movement without requiring all data to reside simultaneously in the highest-performance storage, thus reducing overall storage space occupation.
2Speed
If weight data is loaded into first storage space for execution, then execution speed is improved, but storage space occupation increases
Solution Approach 1:
Weight data is pre-loaded into the first storage space according to the predetermined execution order of execution units. This preliminary action ensures that data is ready in fast memory before execution begins, maintaining high execution speed while avoiding the need to load all weight data simultaneously.
Solution Approach 2:
The system dynamically loads and unloads weight data from the first storage space based on the current execution unit being processed. As execution units are completed, their associated weight data is unloaded to make room for subsequent execution units, maintaining fast execution while managing storage space dynamically.
Data Source
AI summary
The present application discloses a model data processing system and a method, and a storage medium. The system includes: a first storage device, configured to: determine, based on a predetermined execution order of any execution unit in a deep learning model, a target execution unit to be executed currently; at least one first target storage device, configured to: return and store weight data of the target execution unit stored in a first target storage space, into a first storage space via in turn at least one storage device in a storage device set, where the storage device set includes the at least one first target storage device, and the first target storage device is configured to store in advance, in accordance with the predetermined execution order, weight data of each execution unit into the first target storage space in turn; an execution device, configured to execute the target execution unit based on the weight data of the target execution unit stored in the first storage space. The present application solves the technical problem of large storage space occupied by data processing during model execution.