Filesystem Data Block Partitioning for Bulk Copy Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional filesystem operations for moving large amounts of data are time-consuming due to high metadata overhead requirements, especially when dealing with numerous files, as each file's metadata needs to be updated separately during bulk data copying operations.
Innovation Solution
The method involves partitioning a filesystem's physical address space into logical partitions, allocating containers of data block identifiers, maintaining a partition map, and updating it efficiently during data block transfers between partitions, thereby minimizing metadata updates and optimizing bulk data copy operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional filesystem operations update metadata for each file separately during bulk data copying, then data movement can be performed, but the metadata overhead greatly increases operation time
Solution Approach 1:
The patent merges metadata updates by implementing a single metadata update operation that handles bulk data copying for multiple files simultaneously, rather than updating metadata for each file separately. This combining approach eliminates the repetitive metadata update overhead that plagues conventional filesystem operations, thereby significantly improving data movement speed while reducing total operation time.
2Reliability
If metadata for each file is updated separately during file moves, then file location tracking is maintained, but the number of metadata updates greatly increases overhead requirements
Solution Approach 1:
The patent combines multiple individual metadata update operations into a single bulk metadata update operation. This approach maintains accurate file location tracking for all moved files while dramatically reducing metadata overhead by processing all updates atomically in one operation, rather than sequentially updating each file's metadata separately.
3Quantity of substance
If bulk data copying is performed on a few multiples of data blocks at a time, then data transfer can occur, but the operation becomes time-consuming when dealing with millions of small files
Solution Approach 1:
The patent merges bulk data copying operations by allowing data blocks to be copied in large batches while performing a single metadata update operation. This approach enables efficient handling of millions of small files by combining multiple data block transfers into one operation, thereby significantly improving copy operation speed while maintaining the ability to track all file locations.
Data Source
AI summary
The disclosed computer-implemented method for efficiently moving data within a filesystem may include (1) partitioning, on a storage device, physical address space of a filesystem into a plurality of logical partitions, (2) allocating, on the storage device, at least one container of data block identifiers representing data blocks for each of the logical partitions, (3) maintaining, on the storage device, a partition map of values identifying the container of data block identifiers, the logical partitions, and at least one offset associated with each of the logical partitions, (4) sending, on the storage device, the data blocks from a source logical partition within the logical partitions to a target logical partition within the logical partitions, and (5) updating, on the storage device, the partition map based on the data blocks sent from the source logical partition to the target logical partition. Various other methods, systems, and computer-readable media are also disclosed.