Distributed Erasure Coding Across Storage Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Centralized erasure coding techniques in data storage systems cause processing bottlenecks, inefficient use of computational storage devices, lengthy recovery processes, and write amplification due to update-driven parity rewrites, especially in systems with diverse storage devices and computational resources.
Innovation Solution
Implementing offloaded, device-driven erasure coding where storage devices autonomously partition and encode data, allowing for cooperative erasure coding operations, delayed parity recalculations, and efficient use of computational resources, reducing bottlenecks and write amplification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If centralized erasure coding is implemented, then data reliability is improved, but processing bottlenecks occur and productivity decreases
Solution Approach 1:
The patent segments the centralized erasure coding function into distributed operations across multiple storage devices. Each storage device independently performs erasure coding on its local data blocks and generates parity blocks locally, eliminating the single-point processing bottleneck while maintaining data reliability through distributed redundancy.
Solution Approach 2:
The patent extracts the erasure coding computation from the centralized controller and relocates it to the storage devices themselves. This extraction of computational workload from the central processing point distributes the burden across the storage network, improving overall processing efficiency while preserving data protection capabilities.
2Reliability
If frequent parity recalculations are performed, then data reliability is maintained, but write amplification increases
Solution Approach 1:
The patent implements preliminary action by having storage devices buffer data blocks and perform erasure coding computations in advance before actual write operations are complete. Parity blocks are pre-calculated and staged, allowing the system to maintain data protection without immediately triggering additional write amplification during peak operation periods.
Solution Approach 2:
The patent employs periodic action by scheduling parity recalculation operations at intervals rather than continuously. Storage devices periodically update parity blocks based on changes detected in data blocks, maintaining data reliability while significantly reducing the frequency of write operations and associated write amplification.
3Device complexity
If computational resources are centralized, then control is simplified, but device complexity increases at the controller
Solution Approach 1:
The patent implements self-service by enabling storage devices to autonomously perform erasure coding computations on their own data blocks without requiring centralized control for each operation. Each storage device independently manages its own parity generation and updates, reducing controller complexity while improving computational resource utilization across the distributed system.
4Productivity
If recovery operations are performed quickly, then productivity is improved, but read amplification increases
Solution Approach 1:
The patent applies preliminary action in recovery operations by pre-identifying and staging available parity blocks and data blocks before recovery is requested. When a failure occurs, the system can immediately begin reconstruction using pre-positioned blocks, accelerating recovery speed while minimizing the need for additional read operations and reducing read amplification.
Data Source
AI summary
A method for storing data may include receiving user data at a group of storage devices, wherein the storage devices are interconnected, erasure coding the user data into redundancy blocks at the group of storage devices, and storing the redundancy blocks on at least two of the storage devices. The erasure encoding may be distributed among at least two of the storage devices. The redundancy blocks may be arranged in reliability groups. The redundancy blocks may be grouped by the storage devices independently of the partitioning of the user data by the user. The method may further include recovering data based on redundancy blocks. A storage device may include a storage medium, a network interface configured to communicate with one or more other storage devices, and a storage processing unit configured to erasure code user data into redundancy blocks cooperatively with the one or more other storage devices.