Distributed Storage Data Redistribution via Access Pattern Sorting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face performance bottlenecks due to data hotspots and uneven data block sizes, leading to poor data throughput and increased retrieval times as frequently accessed data concentrates on a few devices and small data blocks accumulate on others, causing seek operation-related overhead.
Innovation Solution
A method to redistribute data blocks based on access patterns and storage sizes using a data management module that sorts and categorizes blocks, then redistributes them across storage devices using either a uniform or proportional distribution scheme to prevent concentration and improve overall data throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data files are allocated to storage devices based on available free space, then storage devices can be utilized efficiently, but data hot spots are created where newer data files concentrate on a small number of storage devices
Solution Approach 1:
The patent implements dynamic data redistribution by periodically monitoring data access patterns and automatically relocating data blocks from hot spots to storage devices with lower activity. This dynamic adjustment resolves the contradiction by maintaining efficient storage utilization while preventing performance degradation from data concentration.
Solution Approach 2:
The system employs feedback mechanisms by continuously monitoring data access patterns and using this information to trigger redistribution operations. When access patterns indicate hot spot formation, the system responds by redistributing data, thus maintaining both storage efficiency and retrieval performance.
2Productivity
If storage devices accumulate many small data blocks, then storage capacity is effectively utilized, but seek operation-related overhead increases and retrieval performance deteriorates
Solution Approach 1:
The patent changes the parameter of data block distribution by redistributing small data blocks across storage devices based on access patterns. This transformation converts the state from accumulated small blocks on few devices to evenly distributed small blocks across many devices, reducing seek overhead while maintaining capacity utilization.
3Speed
If a small number of storage devices handle the bulk of data operations, then data access speed for frequently accessed files may be improved, but performance bottlenecks occur and overall system throughput decreases
Solution Approach 1:
The patent segments the workload by distributing data blocks across multiple storage devices rather than concentrating them on a few high-performance devices. This segmentation prevents any single device from becoming a bottleneck, thereby maintaining both access speed and overall system throughput.
Data Source
AI summary
Accesses to a number of data blocks stored in a distributed storage are observed. Following observation of the accesses, the stored data blocks are redistributed. In one aspect, redistribution of the data blocks includes determining the access patterns for one or more of the data blocks based on the observed accesses, and determining the storage sizes for the one or more data blocks. Thereafter, based on the determined access patterns and determined storage sizes, the one or more data blocks are sorted. Subsequently, the one or more data blocks are redistributed or rebalanced across a number of storage devices of the distributed storage based on the sorting. In one aspect, the one or more data blocks are redistributed according to either a uniform distribution scheme or a proportional distribution scheme.