Unified Polling for Big Data Cluster Elastic Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for computing resource adjustment in big data clusters face inefficiencies due to long determination times for scaling rules, leading to poor performance in resource allocation, with existing solutions requiring multiple polling mechanisms for each cluster, resulting in resource wastage and inefficiency.
Innovation Solution
A data processing method that configures a cache space based on elastic scaling rules, obtains and updates indicator detection state data within a completion time range, and uses a centralized polling mechanism to determine elastic scaling decisions in real-time, avoiding the need for multiple polling mechanisms and improving decision-making efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple polling mechanisms are established for different indicator items of different clusters, then the computing resource adjustment can be determined according to preset rules, but the determination time becomes long, resulting in bad computing resource adjustment performance
Solution Approach 1:
The patent merges multiple polling mechanisms into a single unified polling mechanism that can handle multiple indicator items across different clusters simultaneously. Instead of having separate polling processes for each indicator-item-cluster combination, the system consolidates them into one polling cycle that collects all necessary data in a single pass, thereby reducing the overall determination time while maintaining reliable resource adjustment decisions.
Solution Approach 2:
The polling mechanism is designed to be universal and multi-functional, capable of polling multiple indicator items for multiple clusters within a single execution. This universal polling mechanism serves all clusters and all indicator types simultaneously, eliminating the need for separate specialized polling mechanisms and significantly reducing the time required for computing resource adjustment determination.
2Extent of automation
If computing resource adjustment processing is performed using preset rules for each cluster, then resource allocation can be automated, but the system wastes computing resources at idle time and becomes insufficient at busy times
Solution Approach 1:
The patent implements a feedback mechanism where the unified polling mechanism continuously monitors indicator items across clusters and uses this real-time data to dynamically adjust computing resources. The preset rules are executed based on actual observed conditions rather than static configurations, allowing the system to automatically scale resources up during busy periods and down during idle periods, thereby improving both automation and resource allocation efficiency.
Solution Approach 2:
The system transitions from static preset rules to dynamic resource adjustment by continuously polling indicator items and adapting resource allocation in real-time. The computing resource adjustment becomes dynamic, responding to actual cluster conditions rather than following fixed predetermined patterns, which enables the system to optimize resource utilization across varying load conditions.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
A data processing method and an apparatus, an electronic device, and a computer readable medium are provided. The method includes: after an elastic scaling rule for a target cluster is created, configuring a cache space corresponding to the target cluster according to the elastic scaling rule, obtaining indicator detection state data of the load type rule item within a completion time range according to a cluster identifier of the target cluster and an indicator item identifier carried by the load type rule item, storing the indicator detection state data to the cache space corresponding to the target cluster, keeping obtaining the indicator detection update result corresponding to the target cluster during the polling interval and updating a storage content in the cache space corresponding to the target cluster according to the indicator detection update result. Thus, by means of once completion plus continuous polling updating, the purpose of elastic scaling decision making for a cluster can be achieved.