Distributed File Archiving via Chunk Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current archiving solutions for distributed computing environments face inefficiencies in handling large numbers of varied application data files, as they are optimized for single objects rather than directory structures, leading to low data throughput rates and prolonged archival times for terabyte-sized data.
Innovation Solution
A distributed file processing system that splits large files into chunks, compresses and encrypts them, and uses a cloud-based object-level storage environment to archive and restore data efficiently, leveraging distributed computing for high parallelization and metadata management to improve throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If web service interface is used for archiving, then data availability and security are improved, but data throughput rate deteriorates (averaging about 2 MB/s)
Solution Approach 1:
The patent segments the archiving process into multiple parallel web service calls, each handling a single file. By dividing the overall archiving task into independent file-level operations that can be executed concurrently across multiple distributed hosts, the system achieves high parallelization while maintaining the reliability benefits of the web service interface.
2Reliability
If web service interface is used for archiving, then data security is improved, but archival time deteriorates (several days for terabyte-sized data)
Solution Approach 1:
The patent segments the archiving task into file-level operations that can be performed in parallel across multiple distributed hosts. This segmentation enables the system to maintain secure web service interface usage while dramatically reducing total archival time through concurrent processing of multiple files simultaneously.
Solution Approach 2:
The patent implements preliminary actions by establishing the distributed computing environment and file system structure before archiving begins. Metadata is prepared and organized in advance, and the distributed host network is configured to enable parallel processing, thereby reducing the actual archival execution time while maintaining security protocols.
3Adaptability or versatility
If distributed computing is used, then scalability is improved, but system complexity deteriorates
Solution Approach 1:
The patent implements a universal distributed host architecture where each host can perform multiple functions (archiving, restoring, metadata management) through a standardized web service interface. This multi-functionality enables scalability without proportionally increasing system complexity, as the same infrastructure components serve multiple purposes across the distributed network.
Data Source
AI summary
Systems and methods for distributed file processing are disclosed. In one embodiment, a computer-implemented method for distributed file processing in a distributed network may include: (1) receiving, at a first distributed host in a network of a plurality of distributed hosts, an archive command; (2) the first distributed host identifying a plurality of files in a shared file system to archive in response to the archive command; (3) the first distributed host splitting at least one of the plurality of files that is above a predetermined size into a plurality of file chunks; (4) the first distributed host instructing a second distributed host to archive one of the plurality of files or plurality of file chunks to a store; and (5) the first distributed host and the second distributed host archiving the plurality of files and the plurality of file chunks to the store.