Meta Learning Model Parameter Distribution Across Processing Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing meta learning methods for deep learning models, such as those using the MAML algorithm, face inefficiencies due to the large number of model parameters, leading to low training efficiency and the inability of a single processing node to accommodate all parameters, especially for large embedding layer parameters.
Innovation Solution
A meta learning method and system that distribute the training of a deep learning model across a cluster of N processing nodes, allowing each node to learn and update parameters in parallel, thereby reducing graphics memory redundancy and improving training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a single processing node is used to perform meta learning, then the model can be trained, but the graphics memory cannot accommodate all parameters especially for large embedding layer parameters
Solution Approach 1:
The patent divides the model parameters into two categories: embedding layer parameters and dense layer parameters. The embedding layer parameters are distributed across multiple processing nodes, while each node stores and processes dense layer parameters locally. This segmentation allows the system to handle large parameter quantities without requiring a single node with sufficient graphics memory.
2Productivity
If existing MAML optimization algorithm is used, then meta learning can be performed, but the training efficiency is low due to large number of model parameters
Solution Approach 1:
The patent segments the training process into two independent loops: an inner loop for optimizing embedding layer parameters and an outer loop for optimizing dense layer parameters. This segmentation allows parallel processing and independent optimization, significantly improving training efficiency when dealing with large numbers of parameters.
Solution Approach 2:
The patent introduces a hierarchical optimization structure with inner and outer loops, adding a temporal dimension to the optimization process. The inner loop performs multiple iterations for embedding parameters before the outer loop updates dense layer parameters, creating a multi-dimensional optimization space that improves convergence efficiency.
3Reliability
If all parameters are learned by each processing node, then complete parameter training is achieved, but graphics memory redundancy increases
Solution Approach 1:
The patent segments parameter storage across processing nodes: each node stores only the dense layer parameters needed for local processing, while embedding layer parameters are shared across nodes. This segmentation eliminates graphics memory redundancy while maintaining training completeness through coordinated parameter updates.
Solution Approach 2:
The patent introduces a parameter server as an intermediary component that manages and distributes embedding layer parameters across processing nodes. This intermediary enables efficient parameter sharing and coordination, ensuring complete parameter training without requiring each node to store all parameters locally.
Data Source
AI summary
This specification provides a meta learning method of a deep learning model and a meta learning system of a deep learning model, and relates to the field of deep learning technologies. The meta learning method of a deep learning model is applied to a cluster including N processing nodes, and the method includes: obtaining a training dataset, where the training dataset includes training samples corresponding to a plurality of tasks; and performing a plurality of times of iterative training on the deep learning model based on the training dataset in parallel by using the N processing nodes in the cluster, to obtain a meta learning parameter of the deep learning model, In each time of iterative training, each of the N processing nodes learns some parameters of the deep learning model by using some training samples in the training dataset, and the some training samples correspond to a same task.