ML Model Training via Data Instance Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Network bandwidth constraints limit the amount of data that can be transmitted during the training of deep neural networks (DNNs), resulting in smaller batch sizes and longer training times, especially when the training dataset is remote from the computer system executing the training procedure.
Innovation Solution
Implementing lossy data instance compression schemes, such as global, batch-level, instance-level, and importance sampling-based compression, to increase the batch size within network bandwidth constraints, allowing for faster DNN convergence by reducing the size of data instances while maintaining model accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If larger batch sizes are used for SGD-based training, then DNN convergence speed is improved, but network bandwidth constraints prevent transmitting sufficient data
Solution Approach 1:
The patent extracts and removes redundant information from data instances using compression schemes. By identifying and eliminating unnecessary data components while preserving essential information, the system enables transmission of larger batches within bandwidth constraints, thus improving convergence speed without exceeding network capacity.
Solution Approach 2:
The patent changes the parameter of data representation by applying various compression schemes (global, batch-level, instance-level, importance sampling-based) that transform original data into compressed forms. This parameter transformation reduces data size while maintaining training effectiveness, allowing larger batch sizes to be transmitted.
2Loss of time
If data instances are compressed to increase batch size, then training time is reduced, but data fidelity may be compromised
Solution Approach 1:
The patent applies partial compression actions selectively to different data components based on their importance. Importance sampling-based compression identifies and preserves critical information while compressing less important aspects, ensuring that model accuracy is maintained even with aggressive compression applied to other portions of the data.
Solution Approach 2:
The patent incorporates feedback mechanisms where compression parameters are adjusted based on observed training performance. By monitoring model accuracy and convergence behavior, the system dynamically tunes compression levels to maintain reliability while maximizing time efficiency.
Data Source
AI summary
Techniques for accelerating the training of machine learning (ML) models in the presence of network bandwidth constraints via data instance compression. For example, consider a scenario in which (1) a first computer system is configured to train a ML model on a training dataset that is stored on a second computer system remote from the first computer system, and (2) one or more network bandwidth constraints place a cap on the amount of data that may be transmitted between the two computer systems per training iteration. In this and other similar scenarios, the techniques of the present disclosure enable the second computer system to send, according to one of several schemes, a batch of compressed data instances to the first computer system at each training iteration, such that the data size of the batch is less than or equal to the data cap.