Shadow Aggregator for Time Series Metric Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for monitoring and aggregating metrics in large, distributed business applications face challenges in processing millions of metrics per minute in real-time and upgrading aggregation components without significant data loss.
Innovation Solution
A distributed computing system that processes hierarchical metrics in real-time, utilizing a data-verification mechanism to upgrade aggregation components by introducing a shadow aggregator that verifies data against active aggregators, allowing seamless transitions from shadow to active state with minimal data loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the aggregator is upgraded by bringing down or restarting the system, then the aggregation portion can be updated to improve functionality, but data loss occurs and system availability deteriorates
Solution Approach 1:
A shadow aggregator is introduced and pre-configured with the new aggregation logic before the upgrade. The shadow aggregator runs in parallel with the active aggregator, allowing the system to prepare for the upgrade without interrupting service. This preliminary setup enables seamless transition when the upgrade is deployed.
Solution Approach 2:
The shadow aggregator acts as an intermediary component between the collectors and the active aggregator. It receives copies of metric data, processes it through the new aggregation logic, and allows verification of the new aggregation portion's correctness before full deployment, thus maintaining system availability during upgrades.
2Adaptability or versatility
If the system is brought down or restarted for aggregation portion upgrade, then the aggregation functionality can be improved, but data loss occurs
Solution Approach 1:
The system maintains continuous metric collection and aggregation operations during the upgrade process. The shadow aggregator processes metric data continuously in parallel with the active aggregator, ensuring that no data is lost during the transition. Both aggregators operate simultaneously, and the system can switch between them without interruption.
Solution Approach 2:
The shadow aggregator is pre-configured with the new aggregation logic before the upgrade is activated. This allows the system to verify the new aggregation portion's correctness by comparing its output with the active aggregator's output, ensuring data integrity and preventing data loss during the upgrade transition.
3Reliability
If a shadow aggregator is introduced to enable upgrades, then system availability is maintained, but device complexity increases
Solution Approach 1:
The shadow aggregator is essentially a copy of the active aggregator structure, which simplifies the implementation. By using the same architectural pattern for both aggregators, the system minimizes the complexity increase while maintaining reliability. The shadow aggregator uses identical interfaces and data structures, making it a straightforward addition to the existing system.
Solution Approach 2:
The aggregator design is made universal so that the same component can serve as either an active or shadow aggregator. This multi-functionality reduces overall system complexity by using a standardized, reusable architecture rather than creating separate specialized components for each role.
4Measurement precision
If verification is performed between shadow and active aggregator, then data accuracy is improved, but processing time increases
Solution Approach 1:
The verification process implements a feedback mechanism where the shadow aggregator's output is continuously compared with the active aggregator's output. This feedback loop allows for real-time validation of the new aggregation logic's accuracy without requiring complete reprocessing of data, thus maintaining data precision while minimizing time loss.
Solution Approach 2:
The verification process performs partial validation by comparing only critical metric aggregates rather than verifying every single data point. This selective verification approach maintains high data accuracy for the most important metrics while reducing the overall verification time and processing overhead.
Data Source
AI summary
A time series system is updated using a data-verification system. The aggregation system may include one or more aggregators. When an upgrade is appropriate, a shadow aggregator may be added to the set of active aggregators. Metrics are provided from one or more collectors to an active aggregator. The shadow aggregator may receive the metrics intended for a particular aggregator, process the metric, and then pass the metric to the intended aggregator for processing. After a period of time, the shadow aggregator data is verified against the intended aggregator data. If the shadow aggregator data is verified, the shadow aggregator becomes an active aggregator and processes data as normal.