Object Storage Garbage Collection Without Data Structure Corruption
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Public cloud storage systems fail to manage the lifecycle of data effectively, leading to unnecessary costs and security risks due to data retention, and can corrupt larger data structures when individual objects are deleted prematurely.
Innovation Solution
Implement a retention application that analyzes access logs to identify unused objects within object-based storage systems, determining their relationship to active data structures and performing archival or deletion actions accordingly to maintain data integrity and security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If individual objects are deleted from object-based storage systems, then storage costs are reduced, but data structures may be corrupted
Solution Approach 1:
The system performs preliminary analysis of access logs and data structure relationships before deletion. It identifies objects that are truly unused by checking both access patterns and structural dependencies, ensuring safe deletion only after confirming no active references exist.
Solution Approach 2:
The system continuously monitors access logs and updates its understanding of data structure relationships. This feedback mechanism allows it to dynamically adjust deletion decisions based on actual usage patterns, preventing premature deletion of objects that may become unused in the future.
2Reliability
If data is retained in public cloud storage systems, then data availability is maintained, but security risks increase
Solution Approach 1:
The system proactively identifies and removes unused data before security breaches can occur. By analyzing access logs to determine genuine unused status, it eliminates security vulnerabilities while maintaining availability of actively used data.
Solution Approach 2:
The system extracts and removes only the specific portion of data that is truly unused, leaving actively used data intact. This selective removal reduces security risks without impacting data availability for ongoing operations.
3Measurement precision
If access logs are analyzed to identify unused objects, then deletion accuracy is improved, but processing time increases
Solution Approach 1:
The system performs access log analysis in advance to build a comprehensive view of data usage patterns. By pre-processing and caching this information, it reduces the time required for actual deletion decisions while maintaining high accuracy in identifying unused objects.
Data Source
AI summary
Disclosed are various embodiments for garbage collection for object-based storage systems. A first set of objects stored in a data object store is identified as not meeting criteria for deletion or archiving, the criteria specified in a retention policy. A second set of objects is identified by performing a search of the object data store for data objects belonging to a data structure with the first set of objects. The second set of objects is recorded as not meeting the criteria for deletion or archiving. The second set of data objects are retained, based on the recording, to prevent the data structure from being corrupted.