Hierarchical Data Storage Architecture Erasure Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data storage systems face challenges in scaling to handle large volumes of data due to inefficiencies in RAID mechanisms, increased drive failures, and metadata management bottlenecks, leading to high storage costs and reduced device lifespan.
Innovation Solution
A data storage architecture that implements a wide spreading or hierarchical storage approach, using erasure coding to generate redundant data fragments across multiple storage devices, allowing for enhanced storage resiliency and minimizing metadata management by storing it with the data objects, thereby reducing the need for separate metadata servers and improving storage system efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional RAID mechanisms are used for data protection, then storage resiliency is provided, but device lifespan decreases rapidly due to intensive I/O operations
Solution Approach 1:
The patent segments data into multiple fragments and distributes them across different storage devices. When data needs to be protected or recovered, only specific fragments need to be accessed, rather than performing intensive I/O operations on all storage devices as in traditional RAID. This segmentation reduces the I/O burden on individual devices, extending their lifespan while maintaining storage resiliency.
Solution Approach 2:
The patent performs preliminary encoding of data into fragments with embedded redundancy information before storage. This preliminary action ensures that data can be recovered without requiring intensive real-time I/O operations on all devices during failure scenarios, thereby reducing wear on storage devices while maintaining protection capabilities.
2Quantity of substance
If drive capacities are increased without commensurate increases in throughput, then storage capacity is improved, but traditional data protection mechanisms become increasingly difficult to implement
Solution Approach 1:
The patent changes the fundamental parameter of data protection from traditional RAID parity schemes to erasure coding with fragmented data distribution. This allows the system to scale to large drive capacities while maintaining manageable complexity through mathematical encoding rather than complex mechanical or procedural protection mechanisms.
3Reliability
If multiple metadata servers are deployed to eliminate bottlenecks and ensure metadata durability, then metadata availability is improved, but system complexity and consistency management difficulty increase
Solution Approach 1:
The patent merges metadata with data objects by storing them together in the same storage system rather than maintaining separate metadata servers. This consolidation eliminates the complexity of managing multiple metadata servers and their consistency relationships, while still providing durable metadata storage through the same erasure coding and fragmentation mechanisms used for data.
4Quantity of substance
If traditional RAID mechanisms are used with larger deployment sizes, then storage capacity is increased, but mean time between faults is reduced
Solution Approach 1:
The patent segments data across a large number of storage devices in a distributed manner. This segmentation approach allows the system to scale to large deployment sizes while maintaining fault tolerance, as the failure of individual devices does not compromise overall system reliability. The erasure coding ensures that data can be recovered even with multiple device failures, maintaining mean time between faults despite increased deployment size.
Data Source
AI summary
Technology is disclosed for a data storage architecture for providing enhanced storage resiliency for a data object. The data storage architecture can be implemented in a single-tier configuration and/or a multi-tier configuration. In the single-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data fragments, which are stored across many storage devices. In the multi-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data segments, which are sent to one or more tiers of storage nodes. Each of the storage nodes further encodes the data segment to generate many data fragments representing the data segment, which are stored across many storage devices associated with the storage node. The I/O operations for rebuilding the data in case of device failures is spread across many storage devices, which minimizes the wear of a given storage device.