Distributed ILM Scanner Namespace Assignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Implementing Information Lifecycle Management (ILM) policies in large-scale geographically distributed storage systems is challenging due to unreliability factors such as network partitioning, hardware failure, and human error, especially when managing billions of data objects with non-homogenous components and lower cost, higher capacity storage devices.
Innovation Solution
Deploying distributed scanners and ILM rules appliers across nodes in the storage system to iteratively scan and evaluate object metadata against ILM policies, with mechanisms for self-assigning regions, managing queue prioritization, and ensuring consistency to prevent policy violations and data loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a centralized paradigm is used to implement ILM policies, then policy management is simplified, but the system becomes untenable and difficult to manage in large-scale distributed systems
Solution Approach 1:
The patent divides the centralized ILM policy management into distributed scanner processes deployed across multiple storage nodes. Each scanner independently evaluates objects within its assigned namespace range, eliminating the single point of failure and management bottleneck while maintaining policy consistency through deterministic self-assignment of scanning ranges.
2Reliability
If distributed scanners are deployed across all nodes, then system reliability and scalability improve, but the complexity of coordinating and managing scanners increases
Solution Approach 1:
Each scanner process autonomously determines its own scanning range based on the total number of objects and scanner index, without requiring centralized coordination or communication with other scanners. This self-assignment mechanism eliminates coordination overhead while ensuring complete and non-duplicate coverage of the object namespace.
3Reliability
If scanners continuously scan all objects, then ILM policy compliance is ensured, but system performance and resource utilization degrade
Solution Approach 1:
Scanners perform iterative scanning at defined intervals rather than continuously monitoring all objects. Each scanner periodically re-scans its assigned namespace range, balancing ILM compliance verification with system performance by avoiding constant evaluation while ensuring policies are applied to newly ingested objects within acceptable timeframes.
4Reliability
If multiple scanners scan overlapping namespace ranges, then fault tolerance improves, but policy violations occur due to duplicate or conflicting evaluations
Solution Approach 1:
The patent assigns asymmetric, non-overlapping namespace ranges to each scanner based on its index and the total object count. This deterministic assignment ensures that each object falls within exactly one scanner's range, preventing duplicate evaluations and policy conflicts while maintaining fault tolerance through the distributed architecture.
Data Source
AI summary
To effectively implement ILM policies and account for unreliability in a geographically distributed large-scale storage system, “scanners” and “ILM rules appliers” can be deployed on nodes throughout the storage system for large scale ILM implementation. Each scanner is programmed to deterministically self-assign a region of object namespace and scan that region of object namespace. To “scan” a region, a scanner accesses metadata of each object that has an identifier within the scanner's region and inserts the object metadata into one of a set of queues for ILM evaluation. An ILM rules applier dequeues object metadata for evaluation against ILM rules and determines whether an ILM task is to be performed for ILM rule compliance.