Redundancy-Coded Shard Placement for Faster Archive Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network computing and storage systems face challenges in optimizing data performance and integrity, particularly in efficiently storing and retrieving redundancy coded data across multiple volumes while maintaining availability and durability.
Innovation Solution
The implementation of redundancy coding techniques, such as erasure codes, that allow original data to be stored as shards across multiple volumes, with indices generated for efficient location and retrieval, enabling direct or indirect data retrieval based on operational parameters and using additional shards to optimize performance without expanding system parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If redundancy coded data is stored across multiple volumes to improve data integrity and availability, then data reliability is improved, but device complexity increases
Solution Approach 1:
The patent segments data into multiple shards distributed across different storage devices and volumes. Each shard is independently stored, and the system uses segmentation to divide the original data into N shards where any K shards can reconstruct the original data. This segmentation approach improves reliability by distributing data across multiple components while managing complexity through structured organization of the segmented data.
Solution Approach 2:
The patent implements nested redundancy by creating multiple layers of shard groups. Within each volume set, there are multiple groups of shards, and each group can independently reconstruct data. This nested structure allows the system to maintain data integrity through multiple hierarchical levels of redundancy, where inner groups provide primary redundancy and outer groups provide additional fault tolerance.
2Speed
If additional shards are used to optimize retrieval performance, then data retrieval speed is improved, but storage space requirements increase
Solution Approach 1:
The patent implements dynamic shard allocation where the number and distribution of shards can be adjusted based on retrieval performance requirements. The system can dynamically select which shards to access for data retrieval operations, optimizing read performance by parallelizing access across multiple shards. This dynamic approach allows performance tuning without permanently increasing storage requirements, as the additional shards are virtual divisions of existing storage capacity rather than physical additions.
3Reliability
If data is stored as shards across multiple volumes, then data availability is improved, but random access performance deteriorates
Solution Approach 1:
The patent applies local quality optimization by creating volume sets with specific characteristics tailored to different access patterns. Within each volume set, shards are organized and indexed to optimize local retrieval efficiency. The system maintains different quality characteristics in different parts of the storage system - some volumes are optimized for sequential access while others are optimized for random access, allowing the system to maintain high availability while minimizing random access penalties through localized optimization.
Solution Approach 2:
The patent implements preliminary indexing and metadata preparation to accelerate random access to sharded data. Before actual data retrieval operations, the system pre-computes and stores index information that maps data locations across shards and volumes. This preliminary action allows the system to quickly locate and access required shards without scanning entire volumes, thereby maintaining data availability across multiple volumes while minimizing the time penalty for random access operations.
Data Source
AI summary
Techniques described and suggested herein include systems and methods for optimizing performance characteristics for data archives stored on data storage systems using redundancy coding techniques by placing different processed portions of the data in disparate systems. For example, redundancy coded shards, which may include identity shards that contain unencoded original data of archives, may be stored on different types of systems to optimize for various retrieval use cases and implemented environments. Implementing systems may monitor various performance characteristics so as to adaptively account for changes to some or all of the monitored parameters.