Model Training Parameter Mapping for Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge in machine learning model training is the massive amount of sample data and model parameters, which leads to high storage and computational resource requirements, making it difficult to achieve efficient training due to limited memory and slow access speeds, especially when using large-scale computer clusters.
Innovation Solution
A method and apparatus for model training that involves reading a portion of sample data to form a subset, mapping model parameters from a first feature component to a second feature component, and training the model based on the subset, reducing the size of model parameters and efficiently distributing workload across multiple computers, while minimizing resource overhead and maintaining computational performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If all sample data is stored in computer memory to ensure high access efficiency, then access speed is improved, but memory resource requirements increase significantly
Solution Approach 1:
The patent segments the large sample data set into multiple subsets, where each worker node processes a specific subset. This segmentation allows each node to load only its required subset into memory, reducing individual memory requirements while maintaining fast access speeds for the distributed training process.
Solution Approach 2:
The patent introduces a distributed computing dimension by deploying training across multiple worker nodes in a cluster. Instead of loading all data into a single machine's memory, the system distributes data subsets across multiple nodes, transforming a single-machine memory problem into a multi-machine distributed storage and processing architecture.
2Speed
If a large number of computer hosts are used to store sample data in internal storage, then access efficiency is guaranteed, but network and cluster management complexity increases
Solution Approach 1:
The patent extracts only the necessary portion of sample data (relevant subsets) to each worker node's memory based on the current training iteration requirements. This extraction approach avoids the need for all nodes to access or manage the entire data set, reducing network traffic and simplifying cluster coordination while maintaining efficient local access.
3Quantity of substance
If sample data is stored in external storage devices, then storage capacity is sufficient, but access speed and training efficiency decrease
Solution Approach 1:
The patent performs preliminary action by pre-loading the required sample data subsets into the memory of each worker node before the actual training computation begins. This preliminary loading ensures that when training computations are executed, the data is already available in fast memory, eliminating the need for slow external storage access during the critical training phase.
Solution Approach 2:
The patent applies partial action by loading only the specific subset of data that each worker node needs for its current training tasks, rather than loading the entire data set into each node's memory. This partial loading approach achieves sufficient storage capacity for training while maintaining fast access speeds, avoiding the excessive memory consumption that would result from loading all data everywhere.
4Reliability
If model parameters are maintained in full size for the complete sample set, then model accuracy is preserved, but memory occupancy and computational overhead increase
Solution Approach 1:
The patent segments the model parameters and sample data into coordinated subsets that are distributed across worker nodes. Each node maintains and processes only the parameter subset relevant to its data subset, reducing individual memory occupancy while ensuring that the complete model accuracy is preserved through proper aggregation of results across all nodes during the distributed training process.
Data Source
AI summary
A model training method and an apparatus thereof are provided. The method includes reading a portion of sample data in a sample full set to form a sample subset; mapping a model parameter related to the portion of sample data from a first feature component for the sample full set to a second feature component for the sample subset; and training a model based on the portion of sample data having the second feature component. A size of a copy of model parameters(s) on a sample computer can be reduced after mapping, thus greatly reducing an amount of training data and minimizing the occupancy of memory of the computer. Memory of a sample computer is used to place vectors, and store and load samples, thereby performing machine learning and training large-scale models with relatively low resource overhead under a condition of minimizing the loss of efficiency.