Distributed File Extent Update Mechanism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed data storage systems face inefficiencies in updating files due to block-based structures, where small changes require rewriting entire blocks, leading to resource wastage and scalability issues, and conflict resolution mechanisms are not scalable in cloud environments.
Innovation Solution
A system that allows byte-level updates by disintegrating files into extents and packets, enabling append-only operations with delta streams to store only updated bytes, and using a copy-write mechanism to prevent overlapping, along with conflict detection and transaction management to handle multiple applications accessing the same file.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If block-based file structure is used for distributed storage, then data can be stored across multiple servers, but updating small portions of files requires rewriting entire blocks, wasting system resources
Solution Approach 1:
The patent segments files into extents and further divides extents into packets at the byte level. This hierarchical segmentation allows updates to be applied at the packet level rather than block level, enabling fine-grained updates without rewriting entire blocks. When a small portion of a block needs updating, only the specific packets containing changed bytes are rewritten, not the entire block.
Solution Approach 2:
The patent implements local quality by allowing different portions of a file to have different update characteristics. Each packet can be independently updated based on whether it contains changed data. The system identifies and processes only the local regions (packets) that contain modifications, rather than treating the entire block uniformly.
2Reliability
If entire blocks are rewritten for small updates, then data consistency is maintained, but system resources such as CPU cycles and memory are insufficiently utilized
Solution Approach 1:
The patent applies partial action by performing updates only on the necessary portions of data. Instead of rewriting entire blocks, the system identifies specific packets containing changed bytes and updates only those packets. This partial action maintains data consistency while avoiding the waste of rewriting unchanged portions of blocks.
Solution Approach 2:
The patent changes the granularity parameter of file updates from block level to packet level. By introducing a finer-grained packet structure within blocks, the system can modify the update granularity parameter to match the actual size of changes, improving both consistency and resource utilization.
3Reliability
If files are locked to prevent conflicts during updates, then data corruption is avoided, but scalability is reduced in distributed cloud environments
Solution Approach 1:
The patent segments the locking mechanism from the file level to the packet level. Instead of locking entire files during updates, the system locks only the specific packets being modified. This segmentation allows other applications to simultaneously access and update different packets of the same file, maintaining data integrity for modified portions while enabling concurrent access to unchanged portions.
Solution Approach 2:
The patent implements local quality in conflict resolution by applying locks only to local regions (packets) being updated rather than the entire file. This allows different applications to work on different portions of the file simultaneously, improving scalability while maintaining data integrity through localized conflict prevention.
4Reliability
If commit dependencies are imposed on transactions, then conflict resolution is achieved, but metadata becomes too large to handle in large distributed computing environments
Solution Approach 1:
The patent segments transaction metadata from the file level to the packet level. Instead of tracking commit dependencies for entire files, the system tracks them at the packet level. This segmentation reduces the quantity of metadata that needs to be managed, as only packets involved in transactions require metadata tracking, not entire files.
Solution Approach 2:
The patent extracts commit dependency tracking from the file-level transaction system and applies it selectively at the packet level. By taking out the metadata requirement from the file level and applying it only where necessary (at packet level), the system reduces overall metadata size while maintaining conflict resolution capabilities.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system for updating a file and resolving conflict in updating of the file in a distributed file store is disclosed. The system includes a receiver, an updater, and a conflict detector. The receiver receives the file formed of a plurality of extents to be updated. An extent is indicative of a piece of data content of the file and formed of a plurality of packets of substantially equal size. The receiver further receives instructions indicative of one or more extents of the file in which the updates are to be made. The updater updates one or more packets, based on the instructions. The conflict detector detects that a portion of the file is being currently updated, receives a request to update the same portion from another application, and decline the request by the other application to update the same portion to resolve the conflict.