Network Telemetry Data Segmentation for High Throughput Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network performance data communication systems face delays in data usability due to metadata being sent infrequently, leading to reduced performance optimization and high resource utilization, with low data collection rates and inefficient resource usage.
Innovation Solution
A method and system that collect network performance data, divide it into metadata and performance data portions, and associate them with unique keys for efficient storage and querying, allowing for real-time reporting and aggregation based on context tags, enabling rapid data processing and reduced resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If network performance data and metadata are sent at varying time scales with metadata sent infrequently, then data collection resource requirements are reduced, but data usability is delayed and performance optimization is reduced
Solution Approach 1:
The patent segments network performance data into two distinct portions: metadata portion and performance data portion. The metadata portion contains contextual information (device identifiers, location, time, etc.) while the performance data portion contains actual performance metrics. This segmentation allows each portion to be processed and stored independently, enabling the system to use performance data immediately upon receipt while metadata is processed asynchronously, thus reducing data usability delay without requiring frequent metadata transmissions.
Solution Approach 2:
The system performs preliminary actions by pre-processing and storing the metadata portion separately as it arrives, rather than waiting for complete data sets. The metadata is validated, normalized, and stored in a metadata database immediately upon receipt, so that when performance data arrives, the matching metadata is already prepared and available for immediate association and usability, eliminating delays.
2Measurement precision
If data collection systems cache data until resolution is completed, then data accuracy is improved, but system resource consumption increases and data collection rates decrease
Solution Approach 1:
The patent divides incoming network performance data into metadata portion and performance data portion, which are then processed and stored separately. The metadata portion is cached and validated independently in a metadata database, while the performance data portion can be processed and made usable immediately. This segmentation eliminates the need to hold entire data sets in cache until all resolution operations complete, thereby maintaining data accuracy through proper metadata association while significantly improving data collection rates.
Solution Approach 2:
The system extracts the metadata portion from the complete network performance data and handles it separately through validation, normalization, and caching operations. By taking out the metadata processing from the main data collection flow, the system ensures data accuracy requirements are met for metadata while allowing performance data to be collected and made usable at higher rates without being blocked by metadata resolution operations.
3Loss of energy
If metadata is retrieved intentionally rather than sent frequently, then communication overhead is reduced, but data usability is delayed
Solution Approach 1:
The system performs preliminary action by proactively sending the metadata portion with each network performance data transmission, rather than requiring intentional retrieval later. The metadata is packaged and transmitted alongside the performance data, so that when the performance data arrives, the matching metadata is already available at the receiving end, eliminating retrieval delays while avoiding the need for frequent separate metadata communication sessions.
Solution Approach 2:
The patent merges the transmission of metadata portion and performance data portion into a single communication event. Instead of separate communications for metadata and performance data, both portions are combined in one data structure and transmitted together, reducing communication overhead by eliminating redundant protocol handshakes and reducing the number of separate communication sessions required.
Data Source
AI summary
An example method includes collecting network performance data from a number of network devices, where the network performance data includes a unique key; dividing the network performance data into a metadata portion and a performance data portion; and associating each of the metadata portion and the performance data portion with a corresponding unique key. The method further includes storing the metadata portion at a first location, storing the performance data portion at a second location, and providing a performance report in response to a request for a performance query, where providing the performance report includes matching at least a portion of the metadata portion with a corresponding portion of the performance data portion.