Distributed Storage Node Assignment for Data Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems face high risks of data loss due to single-point failures, such as hard disk or server failures, especially when the system scale is small, as multiple data blocks of the same data slice are often deployed on the same physical storage node, leading to unreliability.
Innovation Solution
A distributed storage method that splits data files into slices, which are further divided into data blocks and check blocks using redundancy algorithms, with a random algorithm determining distinct physical storage nodes for each block and check block to ensure they are stored on different nodes, following a preset sorting manner, thereby reducing the risk of data loss from single-point failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple data blocks of the same data slice are deployed on the same physical storage node, then storage efficiency is improved, but data reliability deteriorates due to single-point failures
Solution Approach 1:
The patent segments data into multiple slices, and each slice is further divided into multiple blocks. These blocks are then distributed across different physical storage nodes using a hash function, ensuring that no single node holds all blocks of any slice. This segmentation approach resolves the contradiction by maintaining storage efficiency through systematic distribution while improving reliability through geographic dispersion of data blocks.
Solution Approach 2:
The patent introduces a new dimension of distribution by using hash functions to map data blocks to storage nodes based on node identifiers. This creates a multi-dimensional storage architecture where data blocks are not only organized by slice but also distributed across the dimension of physical storage nodes, thereby achieving both efficient storage and enhanced fault tolerance.
2Adaptability or versatility
If data blocks are randomly distributed using DHT, then storage flexibility is improved, but data safety deteriorates due to potential co-location on same failure domain
Solution Approach 1:
The patent applies local quality by ensuring that within each local failure domain (such as a rack or server), data blocks of the same slice are not co-located. The hash function distributes blocks such that they reside on different physical nodes, giving each local domain a unique quality characteristic of having only partial slice data, thereby maintaining overall slice integrity even if one domain fails.
Solution Approach 2:
The system performs preliminary distribution of data blocks to storage nodes before any failure can occur. By pre-distributing blocks across multiple nodes using the hash function, the system ensures that no single node becomes a single point of failure, thus maintaining data safety while preserving storage flexibility.
Data Source
AI summary
A distributed data storage method, apparatus, and system. The method includes: splitting a data file to generate K data slices, splitting each data slice of the K data slices to generate M data blocks for each data slice, and performing check coding on the M data blocks by using a redundancy algorithm to generate N check blocks; determining, by using a random algorithm, a first physical storage node corresponding to one block of the M data blocks and the N check blocks, and determining at least M+1 different physical storage nodes based on the determined first physical storage node and according to a first rule-based sorting manner; and storing at least M+1 blocks of the M data blocks and the N check blocks onto the at least M+1 different storage nodes, where K, M, and N are integers.