Erasure-Coded Volume Writes Without Prior Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing amount of data stored on object storage devices leads to higher latency due to an insufficient increase in the number of Input/Output (I/O) operations that these devices can perform, despite the need for efficient data management and fault tolerance.
Innovation Solution
The technology reduces I/O operations by skipping initial replication for non-latency-sensitive clients and directly performing erasure coding on data blocks, thereby optimizing data storage and retrieval processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replication is performed before erasure coding, then data availability and fault tolerance are improved, but I/O operations increase and latency increases
Solution Approach 1:
The system performs preliminary erasure coding on data blocks before writing to storage devices, creating encoded fragments in advance. This preliminary action eliminates the need for real-time replication operations during data retrieval, thereby reducing I/O latency while maintaining data availability through the distributed encoded fragments.
Solution Approach 2:
The patent extracts the replication operation from the data writing process, separating it into a distinct phase. By taking out the replication step and replacing it with direct erasure coding, the system reduces the number of I/O operations required during normal data access while preserving fault tolerance through the encoded data distribution across multiple devices.
2Reliability
If replication is performed before erasure coding, then data integrity is improved, but storage space consumption increases
Solution Approach 1:
The system changes the parameter of data representation from full replicas to erasure-coded fragments. By transforming data into encoded fragments with redundancy information, the system maintains data integrity and enables reconstruction from fewer fragments, while significantly reducing the total storage space required compared to storing multiple complete replicas.
Solution Approach 2:
Instead of creating multiple complete copies of data through replication, the patent uses erasure coding to create multiple fragments that collectively represent the original data. These fragments act as distributed copies that can be recombined to reconstruct the original data, providing the same integrity benefits as replication but with reduced storage overhead.
3Productivity
If erasure coding is performed without prior replication, then I/O efficiency is improved, but system complexity increases
Solution Approach 1:
The patent segments the data processing workflow into distinct phases: data reception, erasure coding, and distributed storage. By segmenting the complex erasure coding process into manageable stages with clear interfaces, the system improves I/O efficiency while making the overall complexity more tractable through modular organization of operations.
Data Source
AI summary
The present technology pertains to storing blocks in a storage system that requires fewer I/O operations for processes that are non-latency-sensitive. The present technology collects blocks in a buffer and then performs erasure coding while writing the blocks into storage. The erasure coding can occur without the blocks first being replicated. And the present technology can acknowledge the request to store the blocks to a client providing the blocks even before the blocks are written into the storage.