Distributed Storage Aggregation via Node-Level Task Assignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale distributed storage systems, the head or dedicated node often becomes a bottleneck for aggregating statistics, and bounded estimation methods, while reducing resource usage, are not suitable for precise usage statistics required in billing and other use cases.
Innovation Solution
Distributed storage systems are configured to assign aggregation tasks to multiple storage nodes, allowing each node to compute and update its own aggregation load, thereby distributing resource usage evenly across nodes, reducing the likelihood of bottlenecks and ensuring precise statistics aggregation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a single head node or dedicated computer aggregates statistics from all storage nodes, then the aggregation process is centralized and simple to manage, but the head node or dedicated computer becomes a bottleneck in large-scale distributed storage systems
Solution Approach 1:
The patent divides the centralized aggregation function into multiple distributed aggregation tasks assigned to different storage nodes. Each storage node executes local aggregation for specific data stores, transforming a single-point bottleneck into a distributed parallel processing system that maintains manageability while improving performance.
Solution Approach 2:
The patent introduces a new dimension of distribution by assigning aggregation responsibilities across multiple storage nodes rather than concentrating them at one head node. This dimensional shift from centralized to distributed architecture enables the system to scale horizontally while maintaining aggregation efficiency.
2Loss of energy
If bounded estimation is introduced to reduce the number of communications and amount of data to be aggregated, then resource usage is reduced, but the precision of usage statistics is compromised for billing and other critical use cases
Solution Approach 1:
The patent segments the aggregation process into local and global phases. Local aggregation at each storage node reduces communication overhead by preprocessing data, while global aggregation ensures precision is maintained for billing-critical statistics through complete data collection when needed.
Solution Approach 2:
The patent applies different aggregation strategies to different data stores based on their requirements. Critical data stores requiring precise billing information undergo complete aggregation, while less critical data stores can use estimation, optimizing the balance between precision and resource usage.
Data Source
AI summary
A distributed storage system comprises a plurality of storage nodes which includes multiple resources including multiple kinds of resources. Each of the multiple storage nodes among the plurality of storage nodes is configured to execute an aggregation task which has been assigned to the storage node among aggregation tasks, Each of the aggregation tasks is a task for aggregating statistics relating to a resource among the multiple resources which corresponds to the task, to the storage node to which the task has been assigned.