Erasure Coding Fragment Segmentation for Storage Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Erasure-coding-based storage systems face inefficiencies in data retrieval due to the need for extensive computations and higher storage requirements when incorporating locality properties, which can result in slower retrieval speeds and reduced storage efficiency compared to replication-based systems.
Innovation Solution
A system that utilizes finite field solutions to generate enhanced and regular coded fragments, reducing the number of coded fragments needed for data retrieval while maintaining maximum distance separable (MDS) properties, thereby improving network performance and storage efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If erasure coding with locality properties is implemented to improve storage efficiency, then storage space is reduced, but data retrieval speed deteriorates due to extensive computations
Solution Approach 1:
The coded fragments are segmented into two distinct types: enhanced coded fragments with locality properties and regular coded fragments. This segmentation allows the system to use fewer fragments for retrieval while maintaining storage efficiency, resolving the contradiction between reduced storage space and retrieval speed.
Solution Approach 2:
Different types of coded fragments are created with different properties: enhanced fragments optimized for locality and fast retrieval, and regular fragments for complete reconstruction. This local quality differentiation enables fast data retrieval using only enhanced fragments while maintaining overall storage efficiency.
2Speed
If the number of coded fragments is reduced to improve retrieval efficiency, then retrieval speed is improved, but storage reliability deteriorates
Solution Approach 1:
The system segments coded fragments into enhanced and regular types, where enhanced fragments can independently reconstruct data for fast retrieval, while regular fragments provide additional redundancy for reliability. This segmentation allows fast retrieval with fewer fragments while maintaining storage reliability through the complete set.
Solution Approach 2:
The system prepares multiple types of coded fragments in advance: enhanced fragments that can independently reconstruct data without requiring all fragments, providing a cushion against failures. This beforehand preparation ensures both fast retrieval and storage reliability.
3Productivity
If enhanced coded fragments are generated with locality properties, then data retrieval requires fewer fragments, but computational overhead increases during encoding
Solution Approach 1:
The system applies different encoding strategies to different fragment types: enhanced fragments use locality-optimized encoding for fast retrieval, while regular fragments use standard encoding. This local quality approach improves retrieval efficiency without excessively increasing overall computational overhead.
Solution Approach 2:
The system changes encoding parameters specifically for enhanced fragments to optimize for locality and fast retrieval. By modifying only the parameters for enhanced fragments rather than all fragments, the system improves retrieval efficiency while limiting the increase in computational overhead.
Data Source
AI summary
One embodiment provides a system that facilitates efficient storage and retrieval using erasure coding. During operation, the system determines a finite field solution that conforms to both locality and maximum distance separable (MDS) properties of an erasure-coding system. The system determines a generator matrix of the erasure-coding system based on the finite field solution and generates, from a data element, a plurality of coded fragments based on the generator matrix of the erasure-coding system. The plurality of coded fragments includes a set of enhanced coded fragments that allows reconstruction of the data element and a set of regular coded fragments. The number of the enhanced coded fragments can be fewer than a threshold number of coded fragments for the erasure-coding system.