Adaptive Data Partitioning for Dynamic Workload Parallelism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data partitioning models require a fixed number of partitions, which is not optimal for all workloads, leading to low parallelism and performance due to insufficient or excessive partitions, especially in hierarchical models where the number of partitions at each level may be too low compared to available CPUs.
Innovation Solution
A method to dynamically adjust the number of partitions at runtime based on workload analysis, determining the number of coarse and fine mapping objects and setting the number of coarse and fine partitions to optimize parallelism, allowing for more efficient use of available CPUs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a fixed number of partitions is used in conventional data partitioning models, then the system structure is simple and easy to manage, but the parallelism and performance are insufficient when the workload exceeds the partition capacity
Solution Approach 1:
The patent implements dynamic partitioning where the number of partitions is not fixed but adapts based on workload characteristics. The system monitors workload parameters and adjusts the number of partitions at different hierarchical levels accordingly, transforming the static partitioning model into a dynamic one that can respond to changing system conditions and maximize parallelism utilization
Solution Approach 2:
The system changes the partitioning parameters (number of partitions at each level) based on observed workload characteristics. By monitoring workload intensity and distribution, the system adjusts partition counts to optimize parallelism, transforming the rigid parameter setup into an adaptive one that evolves with system demands
2Productivity
If the number of partitions is increased to handle larger workloads, then parallelism improves, but the system complexity and overhead increase
Solution Approach 1:
Rather than statically configuring partitions for maximum parallelism, the system dynamically adjusts partition counts based on actual workload needs. This allows the system to achieve high parallelism when necessary while reducing partition complexity during lower workload periods, avoiding the constant overhead of managing a large fixed number of partitions
3Productivity
If a hierarchical data partitioning model is used with fixed partitions at each level, then object protection and access control are simplified, but the system cannot fully utilize available CPUs when workload exceeds partition capacity
Solution Approach 1:
The patent extends dynamic partitioning to hierarchical data partitioning models, allowing the system to adjust the number of partitions at each hierarchical level based on workload characteristics. This enables the system to fully utilize available CPUs by creating additional partitions when workload demands exceed current partition capacity, while maintaining the hierarchical structure's object protection benefits
Solution Approach 2:
The system dynamically changes partitioning parameters at multiple hierarchical levels based on workload intensity and distribution patterns. By monitoring which hierarchical levels are becoming bottlenecks, the system adjusts partition counts at appropriate levels to maintain optimal CPU utilization across the entire hierarchical structure
4Productivity
If the number of partitions is kept low to simplify management, then partition overhead is reduced, but parallelism and system performance decrease when workload increases
Solution Approach 1:
The system implements feedback mechanisms that monitor workload characteristics and system performance metrics, using this information to dynamically adjust the number of partitions. This feedback loop allows the system to automatically increase partition counts when performance degradation is detected due to insufficient parallelism, and reduce partitions when they are no longer needed, optimizing performance without permanent complexity
Data Source
AI summary
Methods, non-transitory computer readable media, and devices for dynamically changing a number of partitions at runtime in a hierarchical data partitioning model include determining a number of coarse mapping objects, determining a number of fine mapping objects, and setting a number of coarse partitions and a number of fine partitions based on the determined number of coarse mapping object and the determined number of fine mapping objects.