Operation Log Coordination for Distributed Data Trim Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing distributed storage systems with a three-layer architecture face issues of increased write amplification and garbage data due to independent garbage collection processes in the intermediate KV class storage engine layer and underlying distributed file system layer, affecting system performance and cost.
Innovation Solution
A data processing method involving a tag operation instruction, where operation information is stored in an operation log, a data operation table is updated based on log information, and a data operation request is sent to a target data node for collaborative processing, including a trim operation to tag data for deletion without actual deletion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If independent garbage collection processing is performed in each layer (KV storage engine layer and distributed file system layer), then data reliability is maintained, but write amplification increases and system performance deteriorates
Solution Approach 1:
The patent merges the independent garbage collection processes of the KV storage engine layer and distributed file system layer into a unified collaborative garbage collection mechanism. The server coordinates trim operations across both layers simultaneously, allowing garbage collection to occur in sync rather than independently, thereby reducing write amplification while maintaining data reliability through coordinated multi-layer processing
2Reliability
If independent garbage collection processing is performed in each layer, then data integrity is preserved, but garbage data increases and storage cost rises
Solution Approach 1:
The patent combines garbage collection operations across multiple layers into a unified process where the server issues coordinated trim operations to both the KV storage engine layer and distributed file system layer simultaneously. This merged approach ensures that garbage data is eliminated from both layers in sync, preventing the accumulation of redundant garbage data while preserving data integrity through coordinated validation
Solution Approach 2:
The server performs preliminary coordination of garbage collection operations before executing trim operations on individual layers. By pre-coordinating the garbage collection timing and scope across layers, the system eliminates garbage data more efficiently and reduces the total volume of garbage data that would otherwise accumulate from independent, uncoordinated collection processes
3Reliability
If append-only manner is used for data deletion and modification in multi-layer architecture, then data reliability is ensured, but write amplification increases
Solution Approach 1:
The patent merges deletion operations across layers by coordinating trim operations that simultaneously target both the KV storage engine layer and distributed file system layer. Instead of performing separate append-only deletions in each layer, the unified garbage collection mechanism eliminates the need for redundant write operations, reducing write amplification while maintaining data reliability through coordinated multi-layer validation
Data Source
AI summary
Embodiments of the specification provide a data processing method and apparatus, where the data processing method includes: receiving a tag operation instruction for a data file, and storing tag operation information associated with the tag operation instruction in an operation log; updating a data operation table associated with the data file according to the tag operation information recorded in the operation log in a case where the operation log meets a log information processing condition; and generating a data operation request corresponding to a target data node based on an updated data operation table, and sending the data operation request to the target data node. The data operation table is updated according to the operation log, thereby determining the target data node according to the data operation table, and executing a tag operation at the target data node.