Grid-Encoded Shard Layout for Variable Media Capacity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern network computing and data storage systems face challenges in balancing data availability and integrity, particularly in distributed environments, where existing error correction and redundancy schemes can lead to degraded performance and customer experience due to data retrieval delays.
Innovation Solution
The implementation of grid encoded data storage systems that use shard-based redundancy encoding techniques, such as parity encoding and Reed-Solomon codes, to distribute data across multiple storage devices and datacenters, ensuring data durability and availability while minimizing additional storage volume requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional error correcting and redundancy schemes are used to ensure data integrity, then data integrity is improved, but data availability deteriorates due to retrieval delays
Solution Approach 1:
The patent segments data into fixed-size blocks and organizes them into a grid structure with rows and columns. Each block is independently encodable, allowing parallel processing and efficient retrieval without requiring full-system scans, thus maintaining data integrity while improving retrieval speed.
Solution Approach 2:
The patent introduces a two-dimensional grid structure with rows and columns, adding spatial dimensions to data organization. This allows redundancy to be distributed across both dimensions, enabling efficient retrieval paths while maintaining integrity through multiple access routes and repair capabilities.
2Device complexity
If fixed-size blocks are used in grid encoding, then data organization is simplified, but storage efficiency deteriorates when media size varies
Solution Approach 1:
The patent implements dynamic block sizing where the first block can vary in size to match the actual media capacity, while subsequent blocks use a standard fixed size. This dynamic approach optimizes storage efficiency for variable media sizes while maintaining the simplicity of fixed-size organization for the majority of data blocks.
Solution Approach 2:
The patent changes the size parameter of the first block based on the actual media capacity detected during initialization. This parameter adjustment allows the grid structure to adapt to different media sizes, improving storage efficiency without complicating the overall fixed-size block organization scheme.
3Reliability
If entire grids are rebuilt when adding data, then data integrity is maintained, but system productivity deteriorates due to extensive rewriting
Solution Approach 1:
The patent implements incremental updates that modify only the specific rows and columns affected by new data additions, rather than rebuilding the entire grid. This partial action approach maintains data integrity through targeted repairs while dramatically improving update efficiency by avoiding unnecessary rewriting of unchanged data blocks.
4Ease of manufacture
If uniform grid structure is used, then implementation is simplified, but adaptability deteriorates when media sizes vary
Solution Approach 1:
The patent makes the grid structure dynamic by allowing the first row and column to adjust their sizes based on the actual media capacity, while maintaining a uniform fixed-size structure for the remaining blocks. This dynamic adaptation simplifies implementation through a mostly uniform structure while providing the versatility needed for different media sizes.
Data Source
AI summary
Techniques for exploiting variable media sizes to create new redundancy encoded data storage systems are described herein. A set of storage devices is selected based at least in part on each storage device having an available capacity and, using the set of storage devices, a set of shards for a redundancy encoded data storage system is generated such that each shard of the set of shards has a storage capacity corresponding to the minimum available capacity of the set of storage devices.