Access-Node Block Storage With Inline Erasure Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current persistent storage solutions in large data centers face challenges with data reduction techniques like erasure coding being computationally intensive and difficult to scale, especially for small data blocks, leading to inefficiencies in flash memory storage and high costs.
Innovation Solution
The implementation of a Durable Block Device (DBD) with inline erasure coding, where access nodes configure and manage storage volumes to perform erasure coding in real-time, reducing computational overhead and optimizing storage efficiency by aggregating data blocks into larger chunks for storage across multiple devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data reduction techniques like erasure coding are implemented for persistent storage, then storage efficiency and redundancy are improved, but computational overhead increases and scalability deteriorates
Solution Approach 1:
The system segments data into fixed-size blocks and processes them independently through the log structured volume. Each block can be erasure coded and stored separately, allowing parallel processing and improving computational throughput while maintaining storage efficiency.
Solution Approach 2:
The patent introduces access nodes as intermediary devices between servers and storage devices. These access nodes perform erasure coding operations, transferring the computational burden from servers to specialized access nodes, thereby improving server productivity while maintaining data redundancy.
2Quantity of substance
If erasure coding is performed on small data blocks, then storage efficiency is improved, but processing time and latency increase
Solution Approach 1:
The system performs preliminary actions by pre-processing data blocks and organizing them in a log structured volume before final storage. Data blocks are aggregated and prepared in advance, reducing the processing time required during actual write operations while maintaining efficient storage utilization.
Solution Approach 2:
The log structured volume enables continuous data flow and processing without interruption. Data blocks are continuously aggregated, erasure coded, and written to storage in a streamlined manner, minimizing idle time and reducing overall processing latency while maintaining storage efficiency.
3Reliability
If durable block devices with redundancy are implemented, then data reliability is improved, but device complexity and cost increase
Solution Approach 1:
The access nodes serve multiple functions including data reception, erasure coding, and storage management. This multi-functionality reduces the need for separate specialized components, simplifying the overall storage system architecture while maintaining data durability through erasure coding redundancy.
Solution Approach 2:
The system allows flexible configuration of erasure coding parameters such as data block size, redundancy factor, and chunk size. By optimizing these parameters based on specific workload requirements, the system achieves data durability with minimized complexity and cost for different application scenarios.
Data Source
AI summary
Techniques are described in which network devices, such as one or more data center access nodes, are configured to support durable block storage with inline erasure coding, i.e., erasure coding in real time as data is updated. A Durable Block Device (DBD) supports a block level API for one or more storage volumes that may be mapped to one or more applications executed by servers in communication with the data center access nodes. The disclosure describes the operation of the data plane of the DBD that is hosted on one or more access nodes, and its interactions with the management and control planes of the DBD that are hosted on one or more of the servers. The disclosure describes generation of a log structured volume in the DBD configured to gather multiple data blocks into larger chunks of data for inline erasure coding for storage across multiple storage devices.