ML Embedding Deployment Split for Memory-Constrained Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The deployment of machine learning models with large embedding layers requires significant memory resources, leading to high costs and inefficient use of hardware capabilities, as a single machine cannot accommodate the entire model and embedding, resulting in wasted resources and increased expenses for customized machines and maintenance.
Innovation Solution
Separate deployment of machine learning models and their associated embedding processing, where the embedding model is run in an embedding environment and the machine learning model in a model environment, allowing for efficient memory utilization and flexible resource allocation by sending input tensors for low-dimensional conversion and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the embedding layer is deployed together with the machine learning model on a single machine, then the model can access the embedding parameters directly, but the memory requirements become extremely high (hundreds of billions of parameters requiring 100 GB to 1 TB memory)
Solution Approach 1:
The patent divides the deployment architecture into two separate environments: an embedding running environment that stores and manages the embedding parameters, and a model running environment that executes the machine learning model. This segmentation allows the model to access embedding parameters through controlled interfaces without requiring the model environment to allocate hundreds of GBs of memory locally, thus resolving the contradiction between reliable parameter access and excessive memory consumption.
Solution Approach 2:
The patent introduces an embedding management system that acts as an intermediary between the model running environment and the embedding parameters. This intermediary manages the embedding parameters centrally, providing efficient access mechanisms (such as memory-mapped files or shared memory) that allow the model to access embedding data without duplicating it in the model environment's memory, thereby reducing the memory burden while maintaining access reliability.
2Reliability
If customized machines with sufficient memory are purchased to accommodate the entire model and embedding, then the model can run with all parameters available, but the costs of device acquisition, running, and maintenance increase significantly
Solution Approach 1:
The patent creates a universal embedding management system that can serve multiple model running environments simultaneously. Instead of requiring each machine to have dedicated embedding storage, the centralized embedding environment can be shared across multiple models and machines, reducing the need for expensive customized hardware and lowering acquisition and maintenance costs while maintaining full model execution capability.
Solution Approach 2:
The patent transitions from a single-machine vertical architecture to a distributed multi-environment horizontal architecture. By moving embedding storage to a separate dimension (independent embedding running environment), the system avoids the need to vertically scale up single machines with expensive memory, instead enabling horizontal scaling across multiple standard machines that share the embedding resource pool.
3Reliability
If a single machine's capacity is occupied by a large embedding layer, then the embedding can be stored locally, but the execution of the model cannot fully utilize the computing capability of underlying hardware resources
Solution Approach 1:
The patent segments the system into independent embedding and model running environments, allowing the model environment to focus entirely on computation while the embedding environment handles parameter storage. This separation enables the model to fully utilize computing resources (GPUs, TPUs, CPUs) without being bottlenecked by local embedding storage constraints, as embedding data can be efficiently retrieved through the dedicated embedding environment's optimized access paths.
Solution Approach 2:
The embedding management system serves as an intermediary that optimizes the data flow between storage and computation. It implements efficient memory management, caching strategies, and data retrieval mechanisms that minimize the impact on model execution performance, allowing computing resources to operate at full capacity while maintaining reliable access to embedding parameters when needed.
Data Source
AI summary
Implementations of the present specification provide a model-based prediction method and apparatus. The method includes: a model running environment receives an input tensor of a machine learning model; the model running environment sends a table query request to an embedding running environment, the table query request including the input tensor, to request low-dimensional conversion of the input tensor; the model running environment receives a table query result returned by the embedding running environment, the table query result being obtained by the embedding running environment by performing embedding query and processing based on the input tensor; and the model running environment inputs the table query result into the machine learning model, and runs the machine learning model to complete model-based prediction.