Access Count Management for Distributed Storage Load Spikes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face performance degradation due to load spikes, where access is concentrated on specific data, making it difficult to detect and manage efficiently without consuming excessive resources, especially when using algorithms like the space-saving algorithm that limit resource usage but struggle with rapid increases in access frequency over time.
Innovation Solution
An apparatus that counts access times for each data unit, stores management information elements with access counts, and deletes the element with the minimum count when a predetermined number is reached, allowing for efficient detection of data causing spikes by categorizing elements into two groups for counting and history purposes, and regularly migrating and deleting old elements to maintain resource efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the space-saving algorithm is used to limit resource usage for counting access frequency, then resource consumption is controlled, but the ability to detect rapid increases in access frequency deteriorates over time
Solution Approach 1:
The management information elements are divided into two groups: first group elements store access counts from the current unit time period, while second group elements store historical access count data. This segmentation allows the system to maintain a limited number of first group elements for rapid detection while preserving historical trends in the second group, thus resolving the contradiction between resource limitation and detection capability.
Solution Approach 2:
The system performs preliminary migration of elements from the first group to the second group before the first group reaches its capacity limit. This preliminary action ensures that when new access patterns emerge, there is sufficient space in the first group to capture them immediately, maintaining the ability to detect rapid increases while managing resource usage through controlled migration.
2Measurement precision
If management information elements are retained indefinitely to track access history, then detection accuracy is improved, but resource consumption increases
Solution Approach 1:
The system implements periodic migration of management information elements from the first group to the second group at predetermined intervals (when the first group reaches its element limit). This periodic action maintains detection accuracy by preserving historical data in the second group while controlling resource consumption by systematically managing the lifecycle of elements in the first group through migration and deletion.
3Productivity
If the number of servers is increased to handle load variation, then system capacity and reliability are improved, but performance degradation during spikes persists because access remains concentrated on specific nodes
Solution Approach 1:
The system continuously monitors access counts stored in management information elements and uses this feedback to detect spikes in access frequency. When a spike is detected by comparing current access counts against historical data, the system can trigger load distribution mechanisms to redistribute access load across available servers, thereby maintaining performance reliability during spike conditions while preserving the benefits of having multiple servers for capacity and redundancy.
Data Source
AI summary
An apparatus counts, for each piece of data, an access count indicating a number of times of access to the each piece of data for every unit time so as to store a management information element including the access count and identification information identifying the each piece of data. The apparatus deletes, from the plurality of management information elements stored in a memory, a management information element that includes the access count having a value minimum among the plurality of management information elements, when a number of the plurality of management information elements reaches a predetermined number. The apparatus determines whether there is a piece of data that satisfies a condition related to rapid increase of access, based on the access counts included in the plurality of management information elements.