Cloud Object Index Garbage Collection for Tiered Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Organizations face challenges in managing data stored in both private data centers and public cloud storage systems from a single interface, requiring efficient data tiering solutions that reduce external dependencies on cloud providers and ensure data security, while maintaining scalability and performance.
Innovation Solution
A cloud object index is generated to monitor and manage files tiered to cloud storage, using Cloud Metadata Objects and Cloud Data Objects, with expiration dates and garbage collection processes to optimize storage and retrieval, and a caching component to enhance performance by caching frequently accessed data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is tiered to public cloud storage, then storage costs are reduced and scalability is improved, but external dependencies on cloud providers increase and data security concerns arise
Solution Approach 1:
The system segments data into different tiers: hot data remains in private data center storage while cold data is moved to public cloud storage. The data management system is divided into a cloud tiering module that handles cloud storage operations and a garbage collection module that manages data lifecycle, allowing independent optimization of each component.
Solution Approach 2:
The patent introduces an intermediary data management system that sits between the private data center and public cloud storage. This intermediary manages data tiering, maintains metadata about cloud-stored data, and coordinates garbage collection, thereby reducing direct external dependencies while enabling secure cloud storage.
2Quantity of substance
If data is tiered to cloud storage, then storage costs are reduced, but management complexity increases due to needing to track data across multiple interfaces
Solution Approach 1:
The data management system implements a universal interface that can manage both private data center storage and public cloud storage through a single API. The cloud object index and metadata structures enable the system to treat cloud-stored data uniformly with local data, allowing a single management interface to handle tiered data across different storage locations.
Solution Approach 2:
The system implements feedback mechanisms where the cloud tiering module continuously monitors data access patterns and storage usage, automatically adjusting tiering decisions. The garbage collection module receives feedback from the cloud object index about which cloud objects are still referenced, enabling automated management without increasing operational complexity.
3Reliability
If cloud objects are retained indefinitely, then data restoration capability is maintained, but storage costs and external dependencies increase
Solution Approach 1:
The system performs preliminary actions by maintaining a cloud object index that tracks all cloud-stored data and its references before garbage collection occurs. Expiration dates are set in advance for cloud objects based on retention policies, allowing the system to proactively manage cloud storage usage while ensuring data restoration capability is maintained until the expiration date is reached.
Solution Approach 2:
The garbage collection module dynamically changes the retention parameter for cloud objects based on their reference status. Cloud objects that are still referenced by active LINs maintain their retention period, while those with no references have their retention parameter changed to allow deletion, optimizing cloud storage usage while preserving necessary data restoration capability.
4Quantity of substance
If garbage collection is performed on cloud objects, then cloud storage costs are reduced, but risk of deleting still-referenced data increases
Solution Approach 1:
The garbage collection module implements a feedback mechanism where it queries the cloud object index to verify whether any LINs still reference a cloud object before deletion. This feedback loop ensures that cloud objects are only deleted when confirmed to have no active references, reducing cloud storage costs while maintaining data integrity by preventing premature deletion of still-referenced data.
Solution Approach 2:
The cloud object index serves as an intermediary verification layer between the garbage collection module and cloud storage. Before deleting a cloud object, the garbage collection module uses the cloud object index to check reference status, thereby mediating the deletion process to ensure data integrity while enabling cloud storage optimization.
Data Source
AI summary
Systems and methods are provided herein for garbage collecting data previously tiered to cloud storage. A cloud object index can be generated and used to track objects stored in cloud storage. As objects within cloud storage are added or modified, the transactions can be tracked and the cloud object index can updated based on the activity. Expiration dates associated with cloud objects can be maintained in the cloud object index factoring in user defined retention periods, process defined retention periods for services like backup and snapshots, and regulatory defined retention periods depending on the type of data being tiered. Upon a determination that no active processes are linking to a cloud data object, that an expiration date has been met, and other custom factors, cloud objects can be removed from cloud storage permanently.