Data Processing with Sequential Gradient Accumulation for Faster Convergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The convergence speed of machine learning models in analyzing enterprise big data is a significant challenge due to the massive data volumes involved.
Innovation Solution
A data processing system and method that involves sequentially reading sample data subsets, calculating gradient information, accumulating multiple sets of gradient information, and using the accumulated information to update the model, thereby reducing the need for frequent communication and optimizing the convergence process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If sample data subsets are sequentially read and processed one at a time with deletion after calculation, then memory usage is reduced and communication overhead is minimized, but the convergence speed may be affected by insufficient sample data for gradient calculation
Solution Approach 1:
The system pre-loads multiple sample data subsets into memory before gradient calculation begins. This preliminary action ensures that sufficient sample data is available for accurate gradient computation while minimizing the need for frequent communication between storage and processing units during the training process.
Solution Approach 2:
The sample data set is divided into multiple subsets that are sequentially processed. After each subset is used for gradient calculation, it is deleted from memory and the next subset is loaded. This segmentation approach balances memory usage with the need for sufficient data for convergence.
2Productivity
If multiple sample data subsets are accumulated in memory for gradient calculation, then convergence speed is improved through sufficient sample data, but memory usage and communication overhead increase
Solution Approach 1:
Multiple sample data subsets are pre-loaded into memory before the training process begins. This preliminary action ensures that the system has sufficient sample data available for accurate gradient computation, improving convergence speed without requiring continuous communication during training.
Solution Approach 2:
After each sample data subset is used for gradient calculation, it is discarded from memory to make space for the next subset. This approach allows the system to maintain a manageable memory footprint while still accumulating sufficient data over time for effective model convergence.
3Loss of time
If sample data subsets are read sequentially and deleted after use, then communication frequency is reduced, but the total training time may increase due to repeated data loading
Solution Approach 1:
Sample data subsets are pre-loaded into memory before gradient calculation begins. This preliminary action reduces the frequency of communication between storage and processing units during training, minimizing communication overhead and time loss.
Solution Approach 2:
The system maintains a continuous training process by keeping multiple sample data subsets in memory and sequentially processing them without interruption. This continuity ensures that the training workflow progresses smoothly with minimal idle time for data loading.
Data Source
AI summary
Embodiments of the present invention disclose a data processing apparatus. The apparatus is configured to: after calculating a set of gradient information of each parameter by using a sample data subset, delete the sample data subset, read a next sample data subset, calculate another set of gradient information of each parameter by using the next sample data subset, and accumulate a plurality of sets of calculated gradient information of each parameter, to obtain an update gradient of each parameter.