Distributed Data Shard Storage for Concurrent Restoration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, the efficiency of data backup and restoration is hindered by the concurrent restoration limitations when multiple nodes fail, leading to potential data loss and reduced availability.
Innovation Solution
The method involves determining M data nodes for storing data shards, obtaining N replicas, and sharding them into X data shards, ensuring that each data shard has N replicas stored on N different nodes, minimizing the number of replicas on the same storage nodes, thereby reducing data loss and improving restoration efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data shards are stored on multiple storage nodes with redundant backups, then data availability is improved, but restoration efficiency deteriorates when multiple nodes fail concurrently
Solution Approach 1:
The patent segments data into multiple shards and distributes them across storage nodes. When a node fails, only the specific failed shard needs restoration rather than restoring entire datasets, enabling parallel restoration of different shards from different nodes and improving restoration efficiency while maintaining data availability
Solution Approach 2:
The patent changes the parameter of data distribution by storing different shards of the same data on different storage nodes rather than complete copies. This allows the system to restore specific shards in parallel from multiple nodes simultaneously, improving restoration efficiency without sacrificing data availability
2Quantity of substance
If primary and backup data are stored on the same storage nodes, then storage efficiency is improved, but data loss risk increases when nodes fail
Solution Approach 1:
The patent divides data into multiple shards and distributes them across different storage nodes. Each node stores only specific shards rather than complete primary and backup copies, improving storage efficiency while ensuring that failed shards can be restored from other nodes, thereby reducing data loss risk
3Productivity
If data is sharded across multiple nodes, then I/O capability is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal sharding strategy where the same data is divided into multiple shards and distributed across all storage nodes. This multi-functional approach allows any node to serve any shard request and enables parallel I/O operations, improving I/O capability while managing complexity through standardized procedures
Data Source
AI summary
This application relates to distributed storage, and in particular, to a distributed shard storage technology. In a method for storing data shards in a distributed storage system, M data nodes on which to-be-stored data will be stored are determined, N replicas of the to-be-stored data are obtained, and each of the N replicas is sharded into X data shards in a same sharding mode. Then the to-be-stored data is stored on the M storage nodes, that is, N replicas of each of the X data shards are respectively stored on N storage nodes, and a quantity of data shards whose data shard replicas are stored on same N storage nodes is P or P+1, where P is an integer quotient of X divided by CMN.