Erasure Coding Data Alignment for In-Storage Computation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems employing erasure coding techniques face inefficiencies and errors due to unknown data distribution across storage devices, leading to increased overhead and incorrect calculations during in-storage computations.
Innovation Solution
Implementing erasure coding methods that are conscious of in-storage computation by allowing user applications to control data alignment and partitioning, thereby preventing related data from being split across multiple storage devices, and using metadata indices for efficient memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the EC layer partitions and stores data chunks across multiple storage devices without user application control, then data redundancy and reliability are improved, but in-storage computation efficiency deteriorates due to data transfers between devices and synchronization overhead
Solution Approach 1:
The user application performs preliminary actions by controlling data alignment and partitioning before EC processing. The application divides data into chunks and determines their placement on storage devices, ensuring related data remains together. This preliminary organization prevents subsequent data transfers during computation while maintaining EC reliability through proper redundancy distribution.
2Reliability
If the EC layer splits data into chunks and stores them on different storage devices, then data distribution and fault tolerance are improved, but computation accuracy deteriorates due to erroneous results from distributed calculations
Solution Approach 1:
The system applies local quality by allowing the user application to control the specific placement and organization of data chunks on individual storage devices. Related data maintains local coherence on the same device while EC provides global fault tolerance. This localized control ensures computation accuracy by keeping related data together, while overall system reliability is maintained through distributed redundancy.
3Adaptability or versatility
If the EC layer performs erasure coding operations without knowledge of user application data structures, then system independence and modularity are improved, but data alignment and computation efficiency deteriorate
Solution Approach 1:
The user application serves as an intermediary between the EC layer and storage devices. It receives data, performs alignment and partitioning into chunks, and then passes these organized chunks to the EC layer for redundancy processing. This intermediary role enables the EC layer to maintain independence while the application optimizes data organization for computation efficiency, bridging the gap between modular design and performance requirements.
Data Source
AI summary
A method includes sending, from an application layer, a chunk size setting to an erasure coding layer. The method further includes receiving, at the application layer, user data. The method further includes aligning, at the application layer, the user data based on the chunk size setting. The method further includes sending the aligned user data to the erasure coding layer. The method further includes partitioning, at the erasure coding layer, the aligned user data into a first data chunk and a second data chunk. The method further includes generating, at the erasure coding layer, a parity chunk based on the first data chunk and the second data chunk. The method further includes sending, from the erasure coding layer, the first data chunk, the second data chunk, and the parity chunk to a storage system.