Traffic Manager for Distributed Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face challenges in managing partitions effectively, particularly in responding to data traffic spikes and recurring patterns, which can lead to resource strain and potential damage to the system.
Innovation Solution
The system employs a data traffic management service that uses statistical analysis and autocorrelation to identify data traffic spikes and cyclic patterns. This information is used to proactively configure partitions and allocate resources, anticipating future traffic demands.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If pre-allocated resources are used in distributed data storage systems, then system stability is maintained, but resource strain occurs during data traffic spikes
Solution Approach 1:
The system performs preliminary identification of volatile accounts using statistical analysis and autocorrelation to detect cyclic traffic patterns. By proactively identifying accounts likely to generate traffic spikes before they occur, the system can pre-allocate additional resources in advance, avoiding resource strain when spikes actually happen while maintaining system stability
2Adaptability or versatility
If reactive load balancing is implemented to respond to traffic spikes, then resource allocation flexibility is improved, but system damage occurs before the response can take effect
Solution Approach 1:
The system applies statistical analysis and autocorrelation to identify cyclic traffic patterns and volatile accounts before traffic spikes occur. This preliminary identification enables the system to proactively allocate additional resources in advance of anticipated spikes, eliminating the harmful delay inherent in reactive load balancing approaches and preventing system damage before it can occur
3Loss of time
If statistical analysis and autocorrelation are used to identify traffic patterns, then proactive resource allocation is enabled, but computational complexity increases
Solution Approach 1:
The system creates simplified representations of traffic patterns by calculating statistical metrics (mean, standard deviation, autocorrelation coefficients) that capture essential characteristics of cyclic behavior. These compressed statistical models enable proactive identification of volatile accounts and traffic spikes without requiring complex real-time analysis of complete traffic histories, thus reducing computational complexity while maintaining effective prediction capability
Data Source
Figure 1A
Figure 1B
Figure 1C~1E
AI summary
A traffic manager for a distributed data storage system includes an iterative spike identifier to identify N levels of traffic spikes in traffic data on an account basis in a distributed data storage system, where N is greater than zero. A traffic cycle identifier selectively identifies cyclic traffic spikes in at least one of the N levels of traffic spikes for each of a plurality of accounts using autocorrelation and peak detection. A partition manager communicates with the traffic cycle identifier and selectively partitions one of the plurality of accounts based on based on the autocorrelation and the peak detection.