Distributed Storage Purge Request Batching and Status Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems like HDFS, handling data purge requests efficiently is challenging due to resource-intensive write operations and file fragmentation, which can impact read performance.
Innovation Solution
The system employs a multi-stage processing approach using separate data structures to manage data purge requests, allowing for batch processing and optimization, which reduces the need for expensive write operations and minimizes fragmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If write operations are used for status adjustments in distributed storage systems, then data purge requests can be handled, but resource utilization increases and read performance deteriorates
Solution Approach 1:
The patent segments data purge requests into multiple stages: initial submission, processing, and completion. Each stage is tracked separately using different columns (initially_purged, fully_purged) without requiring complete file rewrites. This segmentation allows the system to handle purge requests efficiently by tracking status through metadata updates rather than expensive write operations.
Solution Approach 2:
The patent introduces status flag columns as intermediary elements that mediate between data purge requests and actual data deletion. These flags (initially_purged, fully_purged) serve as intermediate indicators that track purge progress without requiring direct manipulation of the underlying data files, thereby reducing resource utilization while maintaining purge functionality.
2Productivity
If write operations are used for data purge in distributed storage systems, then data can be removed, but file fragmentation increases
Solution Approach 1:
The patent extracts the purge tracking functionality from the actual data files and places it in separate metadata columns within the file system's data structure. By taking out the status tracking (initially_purged, fully_purged flags) from the data content itself and storing it as separate metadata, the system can track purge progress without modifying or fragmenting the actual data files.
Solution Approach 2:
The patent creates a logical copy of the purge status information in separate columns rather than modifying the original data structure. The initially_purged and fully_purged flags serve as copies of the purge state, allowing the system to track data removal progress without performing write operations on the actual data files, thereby preventing fragmentation.
3Productivity
If traditional data purge methods are used in distributed storage systems, then data can be deleted, but read performance is impacted
Solution Approach 1:
The patent implements a dynamic, multi-stage purge tracking mechanism that adapts to the purge progress. The system transitions from a single-state deletion model to a dynamic two-stage model (initially_purged and fully_purged flags) that can adaptively track different phases of data removal. This dynamic approach allows read operations to continue efficiently while purge operations progress through different stages without blocking or degrading performance.
Data Source
AI summary
Systems and methods described herein relate to the efficient handling of data purge requests in the context of a distributed storage system. A plurality of data purge requests is stored in a first data structure. The data purge requests may be grouped into batches that are processed at least partially in parallel. A first data purge request from the plurality of data purge requests is successfully processed, and is moved from the first data structure to a second data structure. Processing of a second data purge request from the plurality of data purge requests is unsuccessful. The second data purge request is retained in the first data structure. Purge status data is generated based on the first data purge request being in the second data structure and the second data purge request being in the first data structure. The purge status data may be presented at a user device.