Synchronous ILM Policy Application for Distributed Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies and visibility issues when applying Information Lifecycle Management (ILM) policies, as they often create redundant copies that are later discarded and lack guaranteed application of ILM rules during ingest, leading to resource wastage and uncertainty in data protection.
Innovation Solution
Implementing synchronous ILM policies at ingest, which ensures that data lifecycle policies are applied immediately, reducing resource consumption and providing guaranteed application of ILM rules, thereby ensuring data protection and efficiency by creating only necessary copies and eliminating interim copies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If asynchronous ILM policies are applied after ingest, then the storage system can return success immediately, but redundant copies are created and discarded later, wasting resources
Solution Approach 1:
The system performs preliminary evaluation of ILM rules during the ingest operation itself, determining the final placement of object copies before the ingest completes. This prevents the creation of redundant interim copies that would later need to be discarded, thereby reducing resource consumption while maintaining fast ingest performance.
2Productivity
If ILM policies are applied asynchronously, then ingest operations complete faster, but there is no guaranteed application of ILM rules and visibility into when policies are applied
Solution Approach 1:
The system provides feedback to the client application regarding the status and completion of ILM policy application. During synchronous ILM evaluation, the system communicates with the client to provide visibility into the ILM application process, ensuring both reliability of policy application and client awareness of the process status.
3Reliability
If synchronous ILM policies are applied at ingest, then resource consumption is reduced and policy application is guaranteed, but the ingest operation must wait for ILM evaluation to complete
Solution Approach 1:
The system segments the ILM evaluation process into discrete, manageable steps that can be performed during ingest without blocking the entire operation. By evaluating ILM rules in a structured sequence and allowing parallel processing where possible, the system minimizes the time impact on ingest operations while ensuring complete policy application.
4Speed
If interim copies are created during asynchronous ILM, then ingest can proceed quickly, but additional storage space is consumed and copies must be deleted later
Solution Approach 1:
The system determines the final destination and placement of object copies during the ingest operation itself, creating only the necessary copies from the start. This preliminary determination eliminates the need to create and later delete interim copies, reducing storage space consumption while maintaining efficient ingest speed.
Data Source
AI summary
A distributed storage system may synchronously apply an Information Lifecycle Management (ILM) policy to objects at ingest. In one embodiment of synchronous ILM, three options are available for a user: balanced, strict, and dual commit. Dual commit refers to the behavior where one will always create two replicated copies in the same site and then apply ILM asynchronously. Strict refers to the behavior where the storage system attempts to apply the ILM policy synchronously on ingest, and if the storage system cannot the ingest of the object will fail. This ensures that the storage system can guarantee that ILM has been applied to recently ingested objects. Balanced refers to the behavior where the storage system attempts to apply ILM synchronously, but if the storage system cannot the storage system may fall-back to dual-commit.