Data Transfer Optimization System Using Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large data transfers in communication networks take extended periods, impacting downstream activities and resource availability, as existing methods do not efficiently optimize data transfer times.
Innovation Solution
A data transfer optimization system utilizing machine learning to split data into groups, monitor transfer times, and dynamically adjust concurrency based on performance metrics, allowing for concurrent archival and reducing transfer times without pre-calculating subtree sizes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data transfers are performed using traditional file system methods, then data integrity is maintained, but transfer time increases significantly
Solution Approach 1:
The patent divides large data transfers into smaller data groups that can be transferred concurrently through multiple paths. The system splits the source directory into multiple data groups and transfers them in parallel, significantly reducing overall transfer time while maintaining data integrity through coordinated reconstruction at the destination.
Solution Approach 2:
The patent implements dynamic adjustment of transfer parameters based on real-time monitoring. The system continuously monitors transfer times and dynamically adjusts the splitting strategy and concurrency level, allowing optimal performance adaptation to changing network conditions and data characteristics without pre-calculating subtree sizes.
2Loss of time
If data is split into multiple groups for concurrent transfer, then transfer time is reduced, but system complexity increases
Solution Approach 1:
The patent implements a self-optimizing system that automatically monitors transfer performance and adjusts splitting strategies without external intervention. The machine learning module continuously learns from transfer patterns and autonomously optimizes data group configuration, reducing the need for manual system configuration and management overhead.
Solution Approach 2:
The system incorporates continuous feedback loops where transfer performance is monitored and fed back to the optimization module. This feedback mechanism allows the system to learn from actual transfer outcomes and continuously refine its splitting and concurrency strategies, achieving optimal performance through adaptive learning rather than complex predetermined rules.
3Productivity
If machine learning is used to optimize transfers, then transfer efficiency improves, but computational overhead increases
Solution Approach 1:
The patent applies machine learning selectively to the most beneficial aspects of transfer optimization rather than attempting to optimize every parameter. The system focuses computational resources on identifying optimal data group splitting and concurrency levels, while using simpler heuristics for routine transfer parameters, achieving significant efficiency gains with moderate computational overhead.
Data Source
AI summary
In one embodiment, a method includes receiving at a network device comprising a data transfer optimization module, input identifying a source directory comprising data and a target directory at a storage device, splitting the data into a plurality of data groups for transfer to the storage device, transmitting the data groups concurrently to the storage device on data paths wherein transfer times of the data groups are monitored, receiving at the data transfer optimization module, identification of a data group with a longest transfer time, splitting the data group with the longest transfer time, and placing the data from the data group into at least two of the data groups for transfer to the storage device. An apparatus and logic are also disclosed herein.