In-Memory Datastore Partitioning for Low-Latency Metric Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional computing resource monitoring services face significant latency issues during high-impact operational events due to the complexity of monitoring distributed systems, leading to delays in data availability and potential data loss, especially as system size increases.
Innovation Solution
A computing resource monitoring service architecture that includes a front-end server, partitioner sub-systems, and aggregator sub-systems, which transform measurements into a binary serialization format, partition based on fully qualified metric identifiers and timestamps, and store in-memory datastores for real-time aggregation and retrieval, ensuring minimal latency and data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If conventional monitoring techniques are used in distributed systems, then system monitoring capability is provided, but significant latency occurs and data availability is delayed
Solution Approach 1:
The monitoring service is divided into multiple independent components: a front-end server for data collection, partitioner sub-systems for data distribution, and aggregator sub-systems for data aggregation. This segmentation allows each component to process data independently, reducing overall system latency while maintaining monitoring capability across distributed systems.
Solution Approach 2:
The patent introduces an intermediary data structure (measurement data model) that standardizes how monitoring data is collected, partitioned, and aggregated. This intermediary layer abstracts the complexity of distributed system monitoring, enabling fast data processing without increasing operational complexity for users.
2Productivity
If monitoring data is collected rapidly, then real-time issue detection is improved, but data loss occurs during high-impact operational events
Solution Approach 1:
The system performs preliminary actions by pre-defining partition keys and aggregation policies before high-impact events occur. When events happen, the pre-configured partitioner sub-systems immediately distribute data to appropriate aggregators, and pre-established redundancy mechanisms ensure no data is lost during rapid ingestion periods.
Solution Approach 2:
The patent implements redundancy mechanisms and buffer structures that cushion against data loss during high-impact events. Multiple aggregator sub-systems maintain redundant copies of aggregated data, and the system continues ingesting data even when some components are overwhelmed, preventing information loss while maintaining high ingestion rates.
3Quantity of substance
If distributed system size increases, then computing resource capacity is improved, but monitoring complexity and latency increase
Solution Approach 1:
As the distributed system grows, the monitoring system segments data into smaller partitions managed by independent aggregator sub-systems. Each partition can be processed autonomously, allowing the monitoring system to scale with computing resource capacity without proportionally increasing overall complexity. The front-end server simply distributes data to available partitioner sub-systems.
4Loss of time
If conventional monitoring architecture is used, then system monitoring is provided, but significant latency occurs during high-impact operational events
Solution Approach 1:
The monitoring architecture dynamically adapts to high-impact operational events by allowing flexible data routing through the partitioner sub-systems to available aggregator sub-systems. During events, the system can dynamically adjust data collection and aggregation rates, prioritizing critical metrics while maintaining reliability through redundant data paths and continuous ingestion capabilities.
Data Source
AI summary
A computing resource monitoring service receives a plurality of measurements for a metric associated with an auto-scale group. Each measurement is associated with metadata for the measurement, which specifies attributes for the measurement. The computing resource monitoring service determines, for each measurement and based at least in part on the metadata, a fully qualified metric identifier for the measurement. The service partitions the plurality of measurements into a plurality of logical partitions associated with one or more in-memory datastores. The service transmits the measurements from the plurality of logical partitions to the one or more datastores for storage of the measurements. These measurements are provided to one or more computing resource managers for the auto-scale group to enable automatic scaling of computing resources of the group based at least in part on the measurements.