Parallel Data Copy in Distributed File Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Copying large files in a distributed file system is inefficient due to the serialized process, which heavily taxes a single server's hardware resources, leading to significant burdens on CPU, memory, and network bandwidth.
Innovation Solution
The method involves partitioning a source file into regions and copying these regions in parallel across multiple servers, with a coordinating server managing the process and merging the temporary files into a single destination file while preserving file descriptors and attributes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a serialized copy process is used on a single server, then the file copy operation can be completed with simple system architecture, but the copying time and hardware resource consumption increase significantly
Solution Approach 1:
The source file is divided into multiple regions (first region, second region, etc.), and each region is copied in parallel by different servers. This segmentation allows the copy operation to be distributed across multiple processing units, significantly improving copying speed while maintaining manageable system architecture through clear division of labor
2Device complexity
If a single server performs the copy operation, then the system architecture remains simple, but the hardware resources such as CPU, memory, and network bandwidth are heavily taxed
Solution Approach 1:
The copy task is segmented and distributed to multiple servers, with each server handling a specific region of the source file. This distribution reduces the hardware resource burden on any single server, as each server only needs to process a portion of the total data rather than the entire file
Solution Approach 2:
Multiple servers work in parallel to copy different regions of the source file simultaneously. The coordinated effort of multiple servers combines their computing power, memory, and network bandwidth resources, reducing the overall resource consumption per server while maintaining simple individual server architecture
3Productivity
If storage is allocated up front for very large files, then the copy operation can proceed smoothly, but the allocation process itself takes a significant amount of time
Solution Approach 1:
Instead of allocating storage for the entire large file at once, the system allocates storage for individual regions separately. Each server allocates storage only for its assigned region, which is a much smaller and faster operation. This segmented allocation approach maintains smooth copy operations while dramatically reducing total allocation time
Data Source
AI summary
Multiple servers sharing a distributed file system are used to perform copies of regions of a source file in parallel from a source storage unit to corresponding temporary files at a destination storage unit. These temporary files are then merged or combined into a single file at the destination storage unit in a way that preserves the inode structure and attributes of the source file. A substantial speedup is obtained by copying regions of the file in parallel.