Grid Encoded Data Storage with Non-Parity Redundancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern network computing and data storage systems face challenges in balancing data availability and integrity, particularly in distributed environments, where existing error correction and redundancy schemes can lead to degraded performance and customer experience due to delays in data retrieval.
Innovation Solution
The implementation of grid encoded data storage systems that use shard-based encoding techniques, including redundancy encoding schemes like parity encoding and Reed-Solomon codes, to distribute data across multiple storage devices and datacenters, ensuring data durability and availability while minimizing additional storage volume requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional error correcting and error tolerance schemes are used to ensure data integrity, then data reliability is improved, but data availability deteriorates due to delays in retrieving data from the data archive
Solution Approach 1:
The patent divides data into multiple shards and distributes them across different storage locations. This segmentation allows parallel retrieval operations and eliminates the single-point bottleneck of traditional error correction schemes, thereby improving data retrieval speed while maintaining data integrity through distributed redundancy.
Solution Approach 2:
The patent introduces a two-dimensional grid structure for data storage, where data can be retrieved along either row or column dimensions. This dimensional approach provides multiple retrieval paths, enabling fast data access without requiring full error correction processing, thus resolving the contradiction between reliability and productivity.
2Reliability
If redundancy encoding schemes like parity encoding are used to distribute data across multiple storage devices, then data availability is improved, but storage capacity utilization deteriorates due to increased storage volume requirements
Solution Approach 1:
The patent combines multiple redundancy encoding schemes (parity encoding and Reed-Solomon codes) within a unified grid structure. This merging allows the system to achieve enhanced data availability and fault tolerance while optimizing storage capacity utilization by efficiently sharing redundant data across the grid rather than duplicating it separately for each encoding scheme.
Solution Approach 2:
The patent dynamically adjusts the redundancy level and encoding parameters based on the specific requirements of different data shards and storage conditions. By changing parameters such as the number of parity shards or the degree of Reed-Solomon encoding, the system optimizes the balance between data availability and storage capacity utilization for different scenarios.
Data Source
AI summary
Techniques for encoding data storage systems using grid-encoded data storage systems with non-parity linear redundancy encoding schemes are described herein. A grid of shards with derived shards and data shards is generated that is indexed by a first index and a second index and is configured so that each shard is reproducible from other shards with the same first index and is also reproducible from other shards with the same second index. The grid of shards is further configured so that each data row of the grid of shards has at least two derived shards and at least twice as many data shards as derived shards.