Hierarchical Data Protection Using ILM Policies and Erasure Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges with excess storage overhead, high repair loads, and inability to support multiple failure types due to relying on a single data protection scheme, which can leave systems vulnerable to additional failures during rebuild processes.
Innovation Solution
A method for policy-based hierarchical data protection that determines data protection schemes at different levels of a storage system using Information Lifecycle Management (ILM) policies, allowing for the combination of multiple protection schemes across storage node and disk levels, such as erasure coding for site and node failures with reduced storage overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replication is used for data protection, then data availability is improved, but storage overhead increases significantly
Solution Approach 1:
The patent segments data protection into two hierarchical levels: node-level replication for availability and disk-level erasure coding for efficiency. This segmentation allows each level to use the most appropriate protection mechanism, reducing overall storage overhead while maintaining data availability.
Solution Approach 2:
The patent combines two different data protection schemes (replication and erasure coding) into a composite hierarchical structure. This composite approach leverages the strengths of both methods: replication's simplicity and availability guarantees at the node level, and erasure coding's storage efficiency at the disk level.
2Quantity of substance
If RAID protection schemes are used, then storage overhead is reduced, but repair load increases and vulnerability to additional failures occurs during rebuild
Solution Approach 1:
The patent segments the repair process into two levels: failed disk reconstruction at the disk level using erasure coding, and failed node reconstruction at the node level using replication. This segmentation isolates repair operations, allowing individual disk failures to be repaired without compromising overall node availability and reducing the vulnerability window.
Solution Approach 2:
The patent implements node-level replication as a cushioning layer before disk-level failures can propagate. This ensures that even if a disk failure occurs and requires reconstruction, the node remains available through replication, providing a safety buffer against additional failures during the rebuild process.
3Reliability
If DDP is used for data protection, then data distribution and protection are improved, but retrieval time increases
Solution Approach 1:
The patent applies different data protection schemes to different levels of the storage hierarchy based on local requirements. Node-level replication provides fast retrieval for commonly accessed data, while disk-level erasure coding provides efficient protection for stored data. This local optimization balances protection and retrieval performance.
Solution Approach 2:
The patent segments data protection operations to separate retrieval-critical functions from protection-critical functions. Node-level replication handles retrieval operations efficiently, while disk-level erasure coding handles protection with minimal impact on retrieval performance.
4Device complexity
If a single data protection scheme is used across the storage system, then system complexity is reduced, but ability to support multiple failure types decreases
Solution Approach 1:
The patent segments the storage system into two distinct levels (node level and disk level), each with its own data protection scheme optimized for the failure types at that level. This segmentation allows the system to support multiple failure types (node failures, disk failures, site failures) without requiring a single complex protection scheme.
Solution Approach 2:
The hierarchical structure provides universal protection against multiple failure types through its two-level design. Node-level replication protects against node and site failures, while disk-level erasure coding protects against disk failures. Together, they provide multi-functional protection coverage.
Data Source
AI summary
A method, non-transitory computer readable medium, and storage management computing device that obtains an information lifecycle management (ILM) policy. A data protection scheme to be applied at a storage node computing device level is determined and a plurality of storage node computing devices are identified based on an application of the ILM policy to metadata received from one of the storage node computing devices and associated with an object ingested by the one of the storage node computing devices. The one of the storage node computing devices is instructed to generate one or more copies of the object or fragments of the object according to the data protection scheme and to distribute the object copies or one of the object fragments to one or more other of the storage node computing devices to be stored by at least the one or more other storage node computing devices on one or more disk storage devices.