Barrier Engine for Ordered Dependent Data Deletion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale distributed storage systems face challenges in managing dependent data objects due to out-of-order replication, leading to invalid snapshots and inefficient deletion processes, particularly when dealing with billions of objects and complex dependency sets.
Innovation Solution
A system utilizing a barrier engine and delete manager to manage dependent delete operations by determining dependent data objects and their dependencies, delaying delete processing until completion of dependent delete requests, and employing timestamps and replication barriers to ensure ordered deletions across storage nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If parallel replication is used to improve efficiency and throughput, then replication speed increases, but replication order cannot be guaranteed leading to invalid dependent data objects
Solution Approach 1:
The system performs preliminary actions by creating dependent data objects with embedded dependency metadata before replication occurs. The delete manager pre-processes delete requests by identifying dependency relationships and establishing deletion sequences in advance, ensuring that dependent objects are deleted only after their dependencies are successfully replicated and deleted at the target node.
Solution Approach 2:
The system implements feedback mechanisms through operation logs that track replication status and dependency relationships. The delete manager continuously monitors replication progress and uses this feedback to dynamically adjust delete operations, waiting for confirmation that dependencies have been successfully replicated before proceeding with deletion of dependent objects.
2Reliability
If strictly ordered replication with confirmation is used to ensure data validity, then replication reliability improves, but system scalability deteriorates due to increased time and computational resources
Solution Approach 1:
The system segments the large-scale data deletion task into manageable units by identifying and processing dependency groups. The delete manager divides the dependency set into smaller subsets that can be processed in parallel where possible, reducing the computational burden while maintaining validity guarantees through the operation log tracking system.
Solution Approach 2:
The operation log serves as an intermediary data structure that mediates between replication and deletion operations. It stores dependency relationships and replication status information, allowing the delete manager to make informed decisions about deletion timing without requiring direct communication between all components, thus improving scalability.
3Reliability
If auditing of dependency objects is performed after replication to detect missing objects, then data validity can be verified, but the process becomes impractical for large data sets
Solution Approach 1:
Instead of auditing after replication, the system performs preliminary validation by embedding dependency metadata in the operation log during the replication process. The delete manager uses this pre-established information to determine deletion sequences without requiring post-replication auditing of large data sets.
Solution Approach 2:
The system extracts only the essential dependency relationship information needed for validation into the operation log, separating this critical metadata from the full data set. This allows efficient tracking and verification of dependency satisfaction without requiring comprehensive auditing of all data objects.
Data Source
AI summary
Example distributed storage systems, delete managers, and methods provide for managing dependent delete operations among data stores. Dependent data operation entries and corresponding dependency sets may be identified in an operations log. Dependent data operations may be identified in each shard and data operation entries. A delete process for the data objects in the dependency set may be delayed until the delete process for the dependent data object completes.