Parallel File Splitting for Upload and Download Time Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network systems face inefficiencies in uploading and downloading data files due to the lengthy process of uploading data files to a file storage node and downloading them from there to a destination node, which affects push efficiency.
Innovation Solution
The method involves splitting data files into multiple sub-files, uploading these sub-files in parallel to a file storage node, and then downloading and merging them in parallel by a slave push node before pushing the merged file to the destination node upon receiving a scheduling instruction, thereby optimizing the push process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If a client uploads a data file to a file storage node sequentially, then the upload process is simple to implement, but the upload time is long
Solution Approach 1:
The patent divides a data file into multiple sub-files before uploading. Each sub-file is uploaded separately to the file storage node in parallel operations. This segmentation allows the upload process to be distributed across multiple concurrent connections, significantly reducing the total upload time while maintaining manageable complexity through automated splitting and parallel handling
2Loss of time
If a slave push node downloads a data file from a file storage node sequentially, then the download process is simple to implement, but the download time is long
Solution Approach 1:
The patent segments the downloaded data file into multiple sub-files and downloads them in parallel from the file storage node. Each sub-file is downloaded simultaneously through separate connections, dramatically reducing download time. The slave push node automatically manages the parallel download operations and reassembles the sub-files into the complete data file
Solution Approach 2:
The patent implements preliminary splitting of the data file into sub-files before the download process begins. This preliminary action enables the slave push node to initiate multiple parallel download operations simultaneously, rather than sequentially, thereby reducing overall download time while keeping the process manageable through pre-organized file structures
3Productivity
If the master push node waits for the slave push node to complete downloading before scheduling the push task, then the push accuracy is ensured, but the push efficiency is reduced
Solution Approach 1:
The patent implements preliminary splitting of the data file into sub-files and initiates parallel download operations before the complete file is available. The slave push node begins processing and pushing sub-files as they are downloaded, rather than waiting for the entire file. This preliminary action significantly improves push efficiency while maintaining reliability through automated integrity verification and error handling mechanisms
Data Source
AI summary
A method, a device and a system for pushing a file are provided. In some embodiments, the method includes: splitting a data file to be pushed into multiple sub-files; uploading the multiple sub-files in parallel to a file storage node; and adding a push task corresponding to the multiple sub-files into a coordinator node. When it is detected that the push task is added into the coordinator node, a slave push node downloads the multiple sub-files in parallel from the file storage node, merges the multiple downloaded sub-files, and pushes a data file obtained by the merging to a destination node upon receipt of a scheduling instruction from the master push node. The solutions shorten the time taken to upload the data file and the time taken to download the data file, and improve the push efficiency.