Volume-Level Erasure Coding for Low-Overhead Sequential Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in optimizing data performance and integrity on network computing and storage systems, particularly in managing large volumes of data with redundancy coding techniques that ensure availability and durability while minimizing storage overhead.
Innovation Solution
The use of redundancy codes, such as erasure codes, to store original data as shards across multiple volumes, accompanied by sparse indices that facilitate efficient retrieval and regeneration of data, even in the event of shard corruption or unavailability, by leveraging a generator matrix to generate redundancy coded shards.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If redundancy coding techniques are applied to ensure data availability and durability, then data reliability is improved, but storage overhead increases
Solution Approach 1:
The patent segments data into multiple shards and distributes them across different storage volumes. By applying redundancy coding at the shard level rather than treating data as a single unit, the system achieves improved reliability while optimizing storage overhead. Each shard can be independently stored and retrieved, allowing efficient use of redundancy codes.
Solution Approach 2:
The patent dynamically adjusts redundancy coding parameters based on data characteristics and storage conditions. By changing the degree of redundancy applied to different data types or storage scenarios, the system optimizes the balance between data reliability and storage overhead, avoiding uniform application of redundancy codes to all data.
2Reliability
If data is stored as shards across multiple volumes, then data durability is improved, but retrieval complexity increases
Solution Approach 1:
The patent pre-computes and stores metadata information about shard locations, relationships, and reconstruction requirements before data retrieval is needed. This preliminary organization of information allows the retrieval process to simply follow pre-determined paths and procedures, significantly reducing retrieval complexity despite the distributed shard architecture.
3Speed
If sparse indices are used to facilitate efficient retrieval, then data access speed is improved, but storage overhead increases
Solution Approach 1:
The patent applies sparse indexing selectively to specific data regions or access patterns rather than uniformly across all stored data. By identifying which data segments benefit most from rapid retrieval and applying indices only to those portions, the system achieves improved access speed for critical operations while minimizing the overall storage overhead consumed by index structures.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored.