Consumer Server Auto-Scaling via Aggregator Event Count Metrics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In near-real-time (NRT) data processing systems, the consumer side often becomes a bottleneck due to its inability to dynamically adjust to changing data processing demands, leading to latency and inefficient resource allocation, as it lacks direct metrics on data throughput and processing capacity, resulting in potential data loss and performance deterioration.
Innovation Solution
Implementing an automatic scaling mechanism for consumer servers that allows them to adjust their number based on real-time data availability from the aggregator server, using additional information such as event counts and processing thresholds to determine when to add or remove servers, ensuring optimal resource allocation and preventing data loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the consumer side uses a fixed number of servers, then device complexity is reduced and ease of operation is improved, but productivity deteriorates due to inability to handle varying data loads
Solution Approach 1:
The consumer side dynamically adjusts the number of consumer servers based on real-time metrics from the aggregator side. The system transitions from a static server configuration to a dynamic one where the number of consumers can automatically scale up or down according to the data processing demands, resolving the contradiction between fixed complexity and variable productivity
Solution Approach 2:
The consumer side implements a feedback mechanism by continuously monitoring metrics (such as data throughput, processing capacity, and queue depth) from the aggregator side and using this information to automatically adjust the number of consumer servers. This closed-loop control enables the system to adapt to changing workloads while maintaining manageable complexity through automated decision-making
2Adaptability or versatility
If the consumer side constantly monitors metrics and rebalances the system, then adaptability is improved, but device complexity increases due to error-prone manual intervention
Solution Approach 1:
The consumer side autonomously monitors its own performance metrics and automatically makes scaling decisions without requiring external intervention. The system self-regulates by comparing current workload against thresholds and automatically provisioning or de-provisioning consumer servers, eliminating the need for complex manual monitoring and reducing human error while maintaining high adaptability
Solution Approach 2:
The patent introduces an automated scaling controller as an intermediary component that sits between the aggregator side and the consumer side. This controller receives metrics from the aggregator, processes the information against predefined policies, and automatically adjusts consumer server allocation, thereby simplifying the overall system architecture by centralizing the complex monitoring and rebalancing logic in a dedicated automated component
3Reliability
If the aggregator sends all available events to consumers, then loss of substance is minimized, but productivity deteriorates due to consumer bottleneck
Solution Approach 1:
The system dynamically adjusts the rate at which the aggregator sends events to consumers based on consumer processing capacity. When consumers are overwhelmed, the aggregator automatically throttles the data flow to prevent bottlenecks and data loss, and when consumers have capacity, the aggregator increases the data flow rate, thereby maintaining both reliability and productivity through adaptive rate control
Solution Approach 2:
The aggregator side implements feedback by monitoring consumer performance metrics and using this information to regulate the data flow rate. When consumers show signs of being overwhelmed (increased latency, queue depth), the aggregator reduces the event transmission rate to prevent data loss, and when consumers are processing efficiently, the aggregator increases the rate to maximize throughput, thus resolving the contradiction between complete data delivery and processing speed
Data Source
AI summary
A system and method for automatically scaling consumer servers in a data processing system. To build an automatic scaling system, the present disclosure allows consumers to obtain additional information, e.g., the number of events that await to be read from an aggregator when receiving an event from the aggregator. This additionally obtained number provides a direct gauge for the data processing system to determine when the consumers are over-provisioned, i.e., when the number of events left to be read is close to zero, as well as when the consumers are under-provisioned, e.g., when the number of events left to be read continues to increase. As a result, the consumers can be automatically scaled to handle the dynamic data processing demand while providing optimal resource allocation.