Distributed Database Data Balancing via Weighted Virtual Node Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed database systems face challenges in achieving data balance between physical nodes due to varying data amounts, leading to potential overload or service failure, with no effective method for scheduling physical nodes to implement data balance.
Innovation Solution
A data balancing method that calculates weight coefficients for each virtual node based on load ratios and weight values, determines standard shard counts for physical nodes, and adjusts virtual node mappings to balance data distribution by migrating virtual nodes from overloaded to underloaded nodes, ensuring that data distribution meets a specified criterion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If virtual nodes are balanced between all physical nodes by adjusting quantities of virtual nodes, then data distribution uniformity is improved, but device complexity increases and redundant resources are wasted
Solution Approach 1:
The patent changes the parameter of virtual node quantity to a fixed configuration and introduces weight coefficients as the adjustable parameter instead. Each physical node is assigned a weight coefficient based on its actual data amount, and virtual nodes are allocated proportionally according to these weights. This parameter transformation resolves the contradiction by maintaining data distribution uniformity through weight-based allocation rather than through complex dynamic adjustment of virtual node quantities.
Solution Approach 2:
The patent introduces weight coefficients as an intermediary element between physical nodes and virtual nodes. The weight coefficient serves as a mediator that translates the actual data amounts of physical nodes into proportional allocation ratios for virtual nodes. This intermediary mechanism simplifies the scheduling process by providing a straightforward mathematical relationship (proportional allocation based on weights) rather than requiring complex adjustment logic.
2Reliability
If virtual node quantity is increased to ensure service正常运行, then service reliability is improved, but device complexity and resource waste increase
Solution Approach 1:
The patent transforms the approach from dynamically adjusting virtual node quantities to using fixed virtual node configurations combined with weight-based allocation. The weight coefficient parameter captures the actual data distribution state, and the proportional allocation mechanism ensures service reliability by automatically adapting to data distribution changes without requiring complex configuration adjustments.
Solution Approach 2:
The patent implements a self-service mechanism where the system automatically adjusts virtual node allocation based on weight coefficients calculated from actual data amounts. The data balancing apparatus autonomously performs the allocation without requiring manual intervention to adjust virtual node quantities, thereby maintaining service reliability while reducing configuration complexity.
3Device complexity
If virtual node quantity is decreased to reduce resource waste, then device complexity is reduced, but service reliability deteriorates
Solution Approach 1:
The patent uses weight coefficients as dynamic parameters that reflect actual data distribution. By allocating virtual nodes proportionally based on these weights, the system maintains adequate resource allocation for each physical node without requiring excessive virtual nodes. This parameter-based approach ensures service reliability while avoiding resource waste through precise, demand-driven allocation.
Solution Approach 2:
The patent applies local quality by allocating virtual nodes according to the specific needs of each physical node as indicated by its weight coefficient. Each physical node receives a proportion of virtual nodes matching its actual data amount, ensuring that nodes with more data get more resources while nodes with less data receive fewer resources. This localized allocation optimizes resource utilization and maintains service reliability.
4Reliability
If physical nodes are deployed based on maximum data amount with redundant resources, then service reliability is improved, but device complexity and cost increase
Solution Approach 1:
The patent changes the deployment strategy from provisioning for maximum data amount to dynamic allocation based on actual data amounts represented by weight coefficients. The system calculates weight coefficients reflecting current data distribution and allocates virtual nodes proportionally, ensuring service reliability while consuming only the necessary resources rather than provisioning for peak scenarios.
Solution Approach 2:
The patent implements dynamic resource allocation where weight coefficients are continuously updated based on actual data amounts, and virtual node allocations are adjusted accordingly. This dynamic approach replaces static provisioning for maximum data amount with adaptive allocation that matches actual needs, thereby maintaining service reliability while reducing resource consumption.
5Quantity of substance
If virtual node quantity is optimized to reduce costs, then device complexity is reduced, but data distribution uniformity deteriorates
Solution Approach 1:
The patent uses weight coefficients as the key parameter to achieve both cost optimization and data distribution uniformity. By calculating weight coefficients from actual data amounts and allocating virtual nodes proportionally, the system ensures that each physical node receives an appropriate share of resources. This parameter-based proportional allocation simultaneously optimizes resource utilization and maintains uniform data distribution.
Solution Approach 2:
The patent implements a feedback mechanism where weight coefficients are calculated based on actual data amounts of physical nodes, and this feedback information drives the virtual node allocation decision. The system continuously monitors data distribution through weight coefficients and adjusts allocations proportionally, ensuring that resource optimization does not compromise data distribution uniformity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of the present invention disclose a data balancing method, including: obtaining load ratios of n weight factors and n weight values of each virtual node of m physical nodes in a distributed database system, where m and n are integers greater than 1; performing weighted averaging based on the load ratios of the n weight factors and the n weight values, to obtain a weight coefficient of each virtual node; obtaining a standard shard count based on a weight coefficient of a virtual node associated with each physical node; determining, based on respective corresponding standard shard counts of the m physical nodes, whether data distribution meets a data balancing criterion; and if no, performing data balancing processing on the m physical nodes. The embodiments of the present invention further disclose a data balancing apparatus. According to the present invention, data balancing can be performed on physical nodes in a distributed database system, to optimize resource configuration.