Sharded Model Parameter Storage for Inference Resource Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In machine learning, large-scale feature embeddings require distributed storage and recombination, leading to inefficient resource utilization due to the need for loading complete models during inference, which is limited by memory size and CPU/network resources.
Innovation Solution
A method and apparatus for processing model parameters involve obtaining a to-be-sharded model parameter set, obtaining attribute information of a storage system different from the system on which the model operates, and storing the model parameters based on this information, optimizing storage and resource utilization by separating large-scale feature embeddings from the model's operational system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If large-scale feature embeddings are distributed on multiple parameter servers for training, then the model can handle large-scale data, but the system complexity increases and resource utilization becomes inefficient during inference
Solution Approach 1:
The patent segments the large-scale model parameter set into multiple shards, where each shard is stored in a separate key-value storage system. This allows the model to handle large parameters by dividing them into manageable pieces while simplifying the inference process by only loading necessary shards into memory.
Solution Approach 2:
The patent extracts frequently accessed model parameters from the complete model and stores them in a key-value storage system. This separation allows the inference system to operate with only the necessary parameters in memory, reducing system complexity and improving resource utilization.
2Reliability
If the complete model is loaded during inference, then the model can perform accurate predictions, but the memory size and CPU/network resources are limited
Solution Approach 1:
The patent implements partial loading of model parameters by only loading the necessary shards into memory during inference, rather than loading the complete model. This reduces memory and CPU resource consumption while maintaining model accuracy for the specific inference task.
Solution Approach 2:
The patent introduces a key-value storage system as an intermediary between the distributed parameter servers and the inference system. This intermediary efficiently retrieves and provides model parameters on-demand, reducing the need to load the complete model into memory while maintaining access to all parameters when needed.
3Quantity of substance
If distributed large-scale feature embedding is used for training, then the model can process large data volumes, but the inference process requires loading and recombining complete models which reduces productivity
Solution Approach 1:
The patent segments the model parameters into shards that can be independently stored and retrieved. During inference, only the necessary shards are loaded and combined, eliminating the need to load and recombine the complete model and significantly improving inference efficiency.
Solution Approach 2:
The patent performs preliminary organization of model parameters into shards during the training phase, storing them in a key-value storage system. This preliminary structuring enables efficient retrieval and combination of parameters during inference, improving productivity without compromising the ability to process large data volumes.
Data Source
AI summary
Provided are a method for processing model parameters, and an apparatus. The method comprises: a model parameter set to be sharded is obtained, wherein the model parameter set comprises a multi-dimensional array corresponding to a feature embedding; attribute information for a storage system used for storing the model parameter set to be sharded is obtained, wherein the storage system used for storing the model parameter set to be sharded differs from a system on which a model corresponding to the model parameter set to be sharded is located when operating; the model parameter set to be sharded is stored in the storage system according to the attribute information.