Erasure Coded Data Relocation in Distributed Log Structured Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprise storage systems face challenges in scalability, flexibility, and management overhead due to their centralized model, which can lead to high upfront costs and inefficient adaptation to changing application requirements, and may result in siloed operations that compromise fault tolerance.
Innovation Solution
A distributed storage system implementing erasure coding with a key-to-physical medium map that allows data relocation and parity calculation across multiple storage devices, ensuring fault tolerance by replicating the key-to-physical medium map across disks and using a distributed write leader election algorithm to manage data writes and reads efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a centralized storage model is used, then storage controller functions and storage media are integrated into the same physical unit, but this makes it harder to independently scale storage capacity, performance and cost
Solution Approach 1:
The patent divides the storage system into separate components: storage controllers and storage media are decoupled into independent units that can be scaled separately. Each storage device can be individually managed and scaled without requiring changes to the entire system architecture, enabling independent scaling of capacity, performance, and cost.
Solution Approach 2:
The system transitions from a single centralized appliance to a distributed multi-dimensional architecture where storage resources can be allocated across multiple devices and locations. This allows the system to scale along multiple dimensions (capacity, performance, cost) independently by adding or removing individual storage devices rather than requiring complete appliance replacements.
2Quantity of substance
If new storage appliances are added to increase capacity and performance, then storage requirements are met, but appliances operate in silos and impose significant management overhead
Solution Approach 1:
The patent merges multiple independent storage appliances into a unified distributed storage system through a common namespace and metadata layer. This allows the system to aggregate storage capacity from multiple devices while providing unified management interfaces that reduce operational complexity. The distributed architecture maintains data accessibility across all devices while presenting a consolidated view to users and applications.
Solution Approach 2:
The storage system implements universal interfaces and protocols that allow different storage devices to be managed through common operations. The distributed file system provides multi-functional capabilities including unified data access, cross-device data migration, and centralized management interfaces that work consistently across all storage devices regardless of their physical location or hardware differences.
3Reliability
If erasure coding is implemented across distributed storage devices, then fault tolerance is enhanced, but data relocation and parity calculation require coordinated updates across multiple devices
Solution Approach 1:
The system performs preliminary actions by pre-calculating parity data and preparing data relocation paths before actual failures occur. The distributed file system maintains metadata about data locations and parity relationships in advance, allowing for rapid fault tolerance operations when failures occur. This preliminary preparation reduces the coordination complexity during actual data protection operations.
Solution Approach 2:
The patent introduces an intermediary distributed file system layer that manages erasure coding operations between applications and physical storage devices. This intermediary handles the complexity of parity calculation and data relocation by abstracting these operations into standardized interfaces, reducing the coordination burden on individual storage devices while maintaining enhanced fault tolerance capabilities.
4Adaptability or versatility
If data stripes are relocated across storage devices, then storage flexibility is improved, but file system updates must track physical location changes
Solution Approach 1:
The system creates and maintains a copy of the namespace metadata that maps logical file identifiers to physical storage locations. This copied metadata structure allows the distributed file system to track data stripe locations independently from the actual data, enabling flexible data relocation without requiring real-time updates to file system structures. The metadata copy can be updated asynchronously, reducing the time overhead associated with tracking physical location changes.
Data Source
AI summary
Presented herein are methodologies for implementing erasure coding in a distributed log structured storage system. A method includes receiving a write request for first data from a file system, selecting a physical sector on a selected storage device in an array of storage devices on which to store the first data, assigning a key to the physical sector, storing the key and an indication of the physical sector in a key-to-physical medium map, erasure coding the data, including generating parity data associated with the first data, writing the first data and the parity data as a data stripe to each storage device in the array of storage devices and, in response to receiving the write request, sending the key to the file system. Read, update, and delete procedures in the context of a log structured framework are also described.