Distributed Storage Data Distribution Using Modular Arithmetic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed shared storage systems face performance reduction and complexity when data is distributed using the same pattern or when adjacent blocks from a single file are stored on the same server, and they struggle to handle changes in storage devices, such as failures or additions, without compromising data distribution resilience.
Innovation Solution
A function using sequentially assigned segment identifiers, where f(k) = ak + b with 'a' being relatively prime to 'n', is employed to distribute data blocks among storage units, allowing for efficient redistribution and minimizing data movement during changes in the storage unit list, while maintaining a history of mappings to recreate the data distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If data is distributed using the same pattern or adjacent blocks from a single file are stored on the same server, then storage organization is simplified, but performance is reduced
Solution Approach 1:
The patent applies local quality by making each file's data blocks have unique distribution characteristics. Instead of using a uniform distribution pattern for all files, the system generates a distinct distribution pattern for each file using a seed value specific to that file. This ensures that adjacent blocks from different files are likely to be stored on different servers, optimizing performance while maintaining manageable complexity through the use of a standardized algorithmic approach.
2Reliability
If a random or pseudorandom sequence algorithm is applied iteratively to generate each value in the sequence, then a unique distribution pattern for each file is achieved, but computational overhead increases as n computations are required for the nth value
Solution Approach 1:
The patent applies preliminary action by pre-generating and storing the complete distribution sequence for each file before actual data storage operations. When a file is introduced to the system, its entire distribution sequence is computed and stored in advance. This allows subsequent data block placements to simply look up pre-computed values rather than performing iterative calculations at the time of storage, significantly reducing computational overhead during normal operations.
3Loss of time
If the sequence of storage unit assignments is stored for each file to avoid computations, then computation time is reduced, but storage space increases due to the need to store long sequences for each file
Solution Approach 1:
The patent extracts only the essential information needed to regenerate the distribution sequence - specifically, a small seed value - and stores this compact representation instead of the complete distribution sequence. The full sequence can be regenerated on-demand using the stored seed and the distribution algorithm. This approach maintains fast access performance while minimizing the additional storage space required, as the seed value occupies minimal space compared to storing entire sequences.
4Reliability
If data is redistributed when storage devices are added or removed, then system resilience is improved, but data movement increases
Solution Approach 1:
The patent applies dynamics by making the data distribution pattern adaptive to changes in the storage system configuration. When storage devices are added or removed, the system dynamically regenerates the distribution sequences using the updated set of storage unit identifiers. This dynamic adaptation allows the system to maintain optimal distribution properties and resilience while minimizing data movement, as only the metadata (distribution sequences) needs to be updated rather than physically redistributing all data.
Data Source
AI summary
Segments or blocks of a file can be distributed among an number n of storage units by using a function of sequentially assigned segment identifiers for each segment, where the function is reduced modulo n, so long as the function is not congruent to segment identifier, modulo n, for any given segment identifier. An example of such a function, where n is the number of storage units and k is a segment identifier, is f(k)=ak+b, where a is relatively prime to n. Such a function can be computed quickly for any given segment. As the list of storage units changes, data may be redistributed using a new mapping of segments to storage units. Any new mapping can be restricted so that segments only move to a new storage unit or from an old storage unit, and not from one existing storage unit to another. In this way, the amount of data to be moved is limited. A chain of the lists of available storage units, as that list changes over time, is maintained to permit the history of file mappings to be recreated.