Resource Allocation Control for Distributed Data Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data lakes, manual calculation of hardware resource allocation for distributed data stores makes it difficult to achieve target performance, as insufficient allocation can lead to performance issues in some data stores while excessive allocation wastes resources in others, and changes such as node additions or program updates complicate accurate QoS control.
Innovation Solution
A resource allocation control device that determines the required hardware resources based on performance models, using a storage unit to store information about resource-performance correlations and a processor to set the appropriate resource allocation for software execution, ensuring optimal resource utilization across distributed data stores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual calculation is used to set hardware resource allocation for distributed data stores, then the administrator can control resource distribution, but it becomes difficult to achieve target performance accurately and maintain consistent performance across different data stores
Solution Approach 1:
The system enables self-service by allowing the distributed data store to automatically report its own performance metrics and resource consumption to the management node. The management node then automatically determines resource allocation based on this reported information, eliminating the need for manual calculation and achieving both accuracy and simplicity
Solution Approach 2:
The system implements feedback mechanisms where performance information flows from the distributed data store to the management node. The management node uses this feedback to continuously adjust resource allocation, ensuring target performance is achieved while adapting to changing conditions without manual intervention
2Reliability
If hardware resources are allocated based on manual calculation, then resource distribution can be controlled, but performance becomes unstable when conditions change such as node additions or program updates
Solution Approach 1:
The system transitions from static manual allocation to dynamic automated allocation. The management node continuously receives performance information and adjusts resource allocation in real-time, allowing the system to adapt to changing conditions such as node additions or program updates while maintaining stable performance
Solution Approach 2:
The system performs preliminary actions by establishing automated monitoring and adjustment mechanisms in advance. When conditions change, the system is already prepared to detect and respond to these changes automatically, maintaining performance stability without requiring manual recalculation
3Productivity
If CPU allocation is increased to handle operations that consume large amounts of CPU, then performance of CPU-intensive operations improves, but CPU resources are wasted when such operations are not actually executed
Solution Approach 1:
The system uses dynamic resource allocation where CPU allocation is adjusted based on actual operation types and performance needs. The management node receives performance information indicating which operations are being executed and allocates CPU resources accordingly, improving efficiency for CPU-intensive operations while avoiding waste when they are not needed
Solution Approach 2:
The system applies local quality by allocating resources specifically to where they are needed. Instead of uniformly allocating CPU resources to all data stores, the system determines and allocates CPU resources locally to specific data stores based on their actual operation types and performance requirements
Data Source
AI summary
In a management node that controls the amount of hardware resources of storage nodes to be allocated to the software of distributed data stores executed by storage nodes, the management node includes a disk device that stores a performance model indicating the correspondence relationship between the amount of hardware resources and the performance that can be implemented by the hardware of the resource amount, and a central processing unit (CPU) connected to the disk device, in which the CPU receives the target performance by distributed data stores, determines the hardware resource amount required to achieve the target performance based on the performance model, and sets to allocate hardware of the determined resource amount to the programs of the distributed data stores.