Adaptive Event Aggregation for API Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing monitoring and analytics systems for application networks face challenges in efficiently handling massive numbers of API calls, leading to high computational resource usage and inefficiencies in event reporting, particularly due to the need for statistical sampling and offline processing.
Innovation Solution
The implementation of lightweight agents that monitor network/API communications, performing inline real-time learning to determine relevant dimensions for lossless event reporting, and aggregating/collapsing dimensions as needed, reducing the burden on resources and enabling efficient event aggregation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If statistical sampling is used to handle massive API calls, then computational resource usage is reduced, but measurement precision and reliability of monitoring data deteriorate
Solution Approach 1:
The system dynamically adjusts aggregation behavior based on learned patterns of API call volumes and characteristics. Aggregation factors are not fixed but adapt over time based on observed data, allowing the system to maintain precision when needed while reducing resource usage when patterns are predictable
Solution Approach 2:
The system changes aggregation parameters (aggregation factors, time windows, dimensionality) based on learned characteristics of the API traffic. By modifying these parameters dynamically, the system optimizes the balance between computational efficiency and monitoring accuracy
2Loss of information
If all API call events are captured and stored for offline processing, then complete event reporting is achieved, but computational and memory resources are overwhelmed
Solution Approach 1:
The system segments the massive API call event stream into manageable chunks using learned aggregation factors. Events are grouped by dimensions (service, operation, status code, etc.) and aggregated in real-time, transforming an unmanageable continuous stream into discrete, processable units
Solution Approach 2:
The system performs preliminary aggregation and filtering of events before they reach the storage and analysis layers. By pre-processing events according to learned patterns, the system reduces the volume of data that needs to be stored and processed offline, while preserving all necessary information for complete event reporting
3Productivity
If traditional monitoring systems process massive API call volumes, then comprehensive analytics are provided, but system complexity and operational difficulty increase
Solution Approach 1:
The system automatically learns and adapts to the characteristics of API traffic patterns without requiring manual configuration or intervention. The learning algorithms automatically determine aggregation strategies, dimensionality reduction approaches, and resource allocation, making the system self-optimizing and reducing operational complexity
Solution Approach 2:
The system incorporates feedback loops where monitoring results and performance metrics are continuously analyzed to refine aggregation strategies. This feedback mechanism allows the system to automatically adjust its complexity level based on the actual data characteristics and resource availability
Data Source
AI summary
An application network is monitored using a plurality of agents. Adaptive event aggregation is performed to determine retaining values for an aggregation dimension. A report of the application network is generated based on the aggregation dimension.