Distributed Storage Data Redistribution via Access Patterns
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage systems suffer from data hotspots where frequently accessed file systems' data becomes concentrated on a small number of storage devices, leading to performance bottlenecks and decreased data throughput over time.
Innovation Solution
Implement a method to redistribute data across multiple storage devices based on access patterns, using either a uniform or proportional distribution scheme to ensure that data is spread evenly or proportionally across available storage devices, preventing concentration on a few devices and improving overall data throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is allocated to storage devices based on available free space, then storage devices can be efficiently utilized, but data hot spots are created causing performance bottlenecks
Solution Approach 1:
The patent changes the allocation parameter from simple available space to a composite parameter that includes access pattern levels. By determining access pattern levels for each file system and using them to guide data allocation, the system transforms the single-parameter allocation into a multi-parameter decision process that prevents data hot spots while maintaining storage efficiency
Solution Approach 2:
The system performs preliminary determination of access pattern levels for each file system before allocating data. By pre-analyzing which file systems are frequently accessed and establishing their access pattern levels, the system can proactively distribute data to prevent hot spot formation rather than reactively addressing performance degradation after it occurs
2Ease of operation
If data of frequently accessed file systems is concentrated on few storage devices, then allocation is simple, but retrieval performance is bottlenecked
Solution Approach 1:
The patent introduces access pattern levels as an additional parameter in the data allocation decision process. Instead of simply allocating data to any available storage device, the system now considers the access pattern level of each file system and distributes data accordingly, transforming a simple allocation rule into an intelligent distribution strategy that maintains both simplicity and performance
Solution Approach 2:
The system segments the storage system into different categories based on access pattern levels. By identifying which file systems are frequently accessed versus less accessed, the system can apply different allocation strategies to different segments, ensuring that hot data is distributed across more devices to prevent bottlenecks
3Productivity
If data is distributed across many storage devices, then performance bottlenecks are reduced, but system complexity increases
Solution Approach 1:
The system implements self-service through automatic access pattern level determination and data redistribution. Rather than requiring manual intervention to balance data across storage devices, the system autonomously monitors access patterns, identifies hot spots, and redistributes data to maintain optimal performance, thereby managing complexity through automation
Data Source
AI summary
A distributed storage system maintains multiple logically independent file systems. Each file system includes a data set stored by a distributed storage of the distributed storage system. During operation, access pattern levels for the multiple logically independent file systems are determined. Thereafter, the data sets included in the multiple logically independent file systems are redistributed across multiple storage devices of the distributed storage. In one aspect, redistribution of a particular data set is based at least in part on the particular file system including the particular data set and on the determined access pattern levels for the multiple logically independent file systems. In one implementation, redistribution is performed according to a uniform redistribution scheme. In another implementation, redistribution is performed according to a proportional distribution scheme.