Network Device Flow Telemetry Triggered by Dropped Packets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing telemetry techniques, such as Postcard Telemetry, randomly sample packets regardless of flow, leading to a lack of targeted sampling for flows experiencing packet drops.
Innovation Solution
A network device dynamically generates a sample policy in response to detected dropped packets, matching on the 5-tuple of the dropped packet to target the specific flow, and distributes this policy to other network devices to ensure all packets in the targeted flow are sampled.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If random sampling is used for telemetry, then the sampling process is simple and uniform, but flows experiencing packet drops are not reliably sampled
Solution Approach 1:
The system implements feedback by monitoring packet delivery status and using drop detection information to dynamically adjust sampling behavior. When packet drops are detected for a particular flow, the sampling rate for that flow is increased to ensure reliable telemetry collection, resolving the contradiction between simple random sampling and reliable sampling of problematic flows.
Solution Approach 2:
The sampling mechanism transitions from static random sampling to dynamic adaptive sampling. The system continuously adjusts sampling parameters based on real-time network conditions and flow performance metrics, allowing it to prioritize sampling of flows experiencing drops while maintaining simple random sampling for normal flows.
2Loss of information
If all packets are sampled to ensure complete visibility, then packet drop detection is comprehensive, but the amount of telemetry data becomes unmanageably large
Solution Approach 1:
The system applies different sampling qualities to different flows based on their performance characteristics. Normal flows receive minimal random sampling, while flows experiencing packet drops receive intensified sampling. This local differentiation ensures complete visibility for problematic flows without generating excessive telemetry data from healthy flows.
Solution Approach 2:
The system dynamically changes sampling parameters (sampling rate, selection criteria) based on flow performance metrics. When a flow exhibits packet drops, the sampling parameter for that flow is increased; when flows are healthy, sampling parameters remain low. This adaptive parameter adjustment balances information completeness with data volume management.
Data Source
AI summary
A flow that experiences packet drops is targeted for explicit sampling based on a dropped packet. A sample policy is created that matches on the dropped packet; for example, the match criteria can be based on the 5-tuple of the dropped packet. The sample policy is programmed in the network device that dropped the packet. The sample policy is distributed to and programmed in network devices that are upstream and downstream of the dropping device. Packets in the flow can then be explicitly sampled to capture the flow as it passes through the network. The sample policy can be updated to remove rules directed to flows that had exhibited drops but have not experienced subsequent drops after a user-configurable period of time.