Distributed Storage Object Delete via Stack Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, deleting objects in a multiple datacenter environment is complex due to interleaved and out-of-sequence requests, which can lead to unintended data losses if not handled properly, resulting in data integrity issues.
Innovation Solution
A method involving a primary datacenter that receives delete requests, retrieves object metadata, and inserts a delete marker on a stack data structure, while chronologically storing current and previous metadata versions to ensure deterministic behavior and eventual consistency across datacenters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple DELETE and PUT requests are handled in a multiple datacenter environment, then the system can process concurrent operations, but requests may arrive out of sequence and fail, leading to unintended data losses
Solution Approach 1:
The system performs preliminary actions by chronologically storing metadata versions in a stack data structure before processing delete requests. The delete marker is inserted on top of the stack, ensuring that the system has a predefined order for processing requests. This preliminary organization prevents out-of-sequence processing and ensures that even if requests arrive concurrently or out of order, they are handled in the correct chronological sequence, maintaining data integrity while allowing high request processing capability
Solution Approach 2:
The stack data structure acts as an intermediary between incoming DELETE/PUT requests and the actual object deletion process. By inserting delete markers onto the stack and processing them in LIFO (Last In, First Out) order, the system mediates the timing and sequencing of request execution. This intermediary mechanism ensures that delete operations only occur after all preceding operations have been processed, preventing unintended data losses while maintaining high throughput
2Reliability
If the system stores chronologically ordered metadata versions, then deterministic behavior and eventual consistency are achieved, but the complexity of managing metadata increases
Solution Approach 1:
The system segments metadata into discrete versioned units stored in a stack data structure. Each metadata version is a separate, identifiable element that can be independently managed. This segmentation allows the system to track chronological changes without requiring complex interdependencies between metadata elements. The stack structure naturally organizes these segments in order, reducing the complexity of metadata management while ensuring deterministic behavior and eventual consistency across datacenters
Solution Approach 2:
The system creates and stores copies of metadata versions in the stack data structure rather than modifying the original metadata in place. Each DELETE or PUT operation generates a new metadata version that is pushed onto the stack, preserving the historical state. This copying approach simplifies metadata management by avoiding complex update logic and conflict resolution, while the chronological stacking ensures deterministic replay and consistency across distributed datacenters
Data Source
AI summary
System and methods for storage object distribution using dynamic policy controls are provided. An embodiment method deleting an object in a distributed storage system includes receiving, from a client, a delete request for an object, retrieving, by a primary datacenter, an object metadata list corresponding to the object to be deleted, and inserting, by the primary datacenter, a delete marker on a top of a stack data structure in an object metadata list.