Metrics Aggregation Service Partitioning for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional computing resource monitoring services face significant latency issues during high-impact operational events, making it challenging to rapidly obtain and consume data for troubleshooting and issue resolution in distributed systems.
Innovation Solution
A computing resource monitoring service that aggregates measurements using a front-end server, partitioner sub-systems, and aggregator sub-systems, transforming measurements into a binary serialization format, and storing them in memory for a specified retention period, enabling rapid data retrieval and minimal data loss through logical partitioning and asynchronous processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If conventional monitoring services are used to measure and diagnose computing resources, then data can be obtained for troubleshooting, but significant latency occurs between when data is obtained and when it is available for consumption
Solution Approach 1:
The system segments the monitoring service into multiple independent components: a client for data collection, a partitioner for data distribution, and multiple aggregators for parallel processing. This segmentation enables asynchronous data aggregation across multiple nodes, significantly reducing latency and improving data availability speed during high-impact operational events.
Solution Approach 2:
The system performs preliminary actions by pre-configuring multiple aggregators and partitioners before operational events occur. When events happen, data is immediately distributed to pre-positioned aggregators that begin processing in parallel, eliminating the need for sequential processing and reducing the time between data collection and availability.
2Quantity of substance
If monitoring services aggregate data from distributed systems, then comprehensive data is obtained, but the complexity of the system grows with the size of the distributed system
Solution Approach 1:
The monitoring service is divided into multiple independent aggregators that each handle specific portions of the distributed system. Each aggregator operates independently with similar interfaces, allowing the system to scale by simply adding more aggregators rather than increasing the complexity of individual components.
Solution Approach 2:
The partitioner acts as an intermediary component that distributes data from the client to multiple aggregators. This intermediary layer abstracts the complexity of data distribution from individual aggregators, allowing each aggregator to focus on processing data without needing to understand the overall system architecture or handle complex routing logic.
3Speed
If data is stored in memory for retention period, then rapid data retrieval is enabled, but data loss may occur during high-impact operational events
Solution Approach 1:
The system implements beforehand cushioning by maintaining retention periods in memory before data loss events occur. When high-impact operational events cause data loss, the retention mechanism ensures that recently aggregated data is preserved in memory long enough for retrieval, cushioning against the effects of data loss during critical events.
Solution Approach 2:
The system maintains continuous aggregation and storage of data in memory across operational events. The retention period ensures that data aggregation continues uninterrupted even during high-impact events, maintaining the useful action of data collection and storage while providing a buffer against complete data loss.
Data Source
AI summary
A web server computer system receives a plurality of measurements for a metric from one or more computing resources associated with the web server computer system. Each measurement includes metadata for the measurement, which specifies attributes of the measurement. The web server computer system determines, for each measurement and based at least in part on the metadata, a fully qualified metric identifier for the measurement. The web server computer system uses the fully qualified metric identifier to partition the plurality of measurements into various partitions. Once completed, the web server computer system transmits a request to one or more aggregator sub-systems of a computing resource monitoring service to store the plurality of measurements.