Probabilistic Data Structures for IoT Telemetry Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As the Internet of Things (IoT) expands, networks face challenges in efficiently classifying and managing the growing number and variety of IoT devices, leading to overwhelming volumes of telemetry data that can overload resources and impact data traffic, especially when device types are unknown or unprofiled.
Innovation Solution
Implementing a probabilistic data structure within network elements to selectively send telemetry data to a device classification service only for flows without pre-existing entries for both source and destination devices, using mechanisms like Bloom filters or Cuckoo filters to control and downsample telemetry reporting, thereby reducing resource usage and optimizing data transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If telemetry data is collected for all IoT devices, then device classification accuracy is improved, but network resource consumption increases
Solution Approach 1:
The patent implements partial action by selectively collecting telemetry data only for devices that meet specific criteria (newly discovered devices or devices with unknown types), rather than collecting data for all devices. This is achieved through checking whether device identifiers exist in the probabilistic data structure before initiating telemetry collection, thereby reducing overall data volume while maintaining classification accuracy for unknown devices
2Reliability
If telemetry data is sent for all traffic flows, then device classification completeness is improved, but network overhead increases
Solution Approach 1:
The patent applies preliminary action by pre-populating the probabilistic data structure with known device identifiers before telemetry collection begins. This allows the system to proactively filter out traffic from known devices and only collect telemetry for unknown devices, ensuring classification completeness for new devices while avoiding redundant data collection for already-known devices
3Productivity
If probabilistic data structures are used for telemetry filtering, then data transmission efficiency is improved, but device classification precision may be reduced
Solution Approach 1:
The patent employs parameter changes by adjusting the false positive rate parameter of the probabilistic data structure to optimize the balance between filtering efficiency and classification precision. By carefully tuning this parameter, the system achieves high data transmission efficiency while maintaining sufficient precision for device classification
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
In embodiments, a network element in a network maintains a probabilistic data structure indicative of devices in the network for which telemetry data is not to be sent to a device classification service. The network element detects a traffic flow sent from a source device to a destination device. The network element determines whether the probabilistic data structure includes entries for both the source and destination devices of the traffic flow. The network element sends flow telemetry data regarding the traffic flow to the device classification service, based on a determination that the probabilistic data structure does not include entries for both the source and destination of the traffic flow.