Adaptive Network Telemetry Sampling via Stream Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional uniform sampling techniques for network telemetry data collection are inadequate as they may miss unusual events during non-sampled intervals, lead to resource exhaustion with high sampling rates, and result in undersampling of important events.
Innovation Solution
The integration of non-uniform sampling with stream processing, which selectively collects data based on network conditions, allowing for adaptive data collection, in-depth analysis, real-time alerting, and automated responses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If uniform sampling is used for data collection, then implementation is simple, but unusual events may be missed during non-sampled intervals
Solution Approach 1:
The patent applies dynamics by transitioning from static uniform sampling to dynamic non-uniform sampling. The sampling rate is dynamically adjusted based on network conditions and event priority, allowing the system to collect data more frequently during critical periods and less frequently during normal operation, thereby resolving the contradiction between implementation simplicity and event detection reliability
Solution Approach 2:
The patent changes the sampling rate parameter from a fixed uniform value to a variable non-uniform value. By modifying the sampling rate parameter based on detected event types and network conditions, the system achieves both ease of implementation through a straightforward sampling framework and improved reliability through adaptive event detection
2Measurement precision
If high sampling rate is used, then event detection accuracy improves, but resource exhaustion occurs
Solution Approach 1:
The patent applies local quality by differentiating sampling rates for different types of network events. Instead of using a uniform high sampling rate for all events, the system uses higher sampling rates only for critical events (such as security incidents or outages) and lower sampling rates for routine events, thereby improving event detection accuracy for important events while reducing overall resource consumption
Solution Approach 2:
The patent applies partial action by selectively applying high sampling rates only when and where needed. Rather than continuously using maximum sampling rate, the system uses partial sampling (higher rate) only for critical event detection, and standard or reduced sampling for non-critical events, achieving adequate measurement precision for important events while avoiding resource exhaustion
3Ease of manufacture
If uniform sampling is used, then data collection is straightforward, but important events are undersampled
Solution Approach 1:
The patent changes the sampling rate parameter dynamically based on event priority and type. By modifying the sampling rate parameter in response to detected event characteristics, the system maintains data collection simplicity through a unified sampling framework while ensuring important events receive adequate sampling through parameter adaptation
Data Source
AI summary
In some implementations, the method may include providing one or more collectors which periodically request memory utilization from a device. In addition, the method may include receiving, by a stream processor, memory utilization from the one or more collectors. The method may include monitoring, by the stream processor, if an average memory utilization evaluated over a predetermined time crosses a predetermined threshold (for example 5 minutes). Moreover, the method may include sending data downstream to a data sink for persistence. Also, the method may include sending a new sampling strategy to the one or more collectors, if the average memory utilization evaluated over the predetermined time crosses the predetermined threshold.


