Network Traffic Record Distribution via Hash-Based Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems face challenges in analyzing and managing large volumes of network traffic records, particularly in calculating unique counts of network entities, which are essential for network operations, due to insufficient data sampling and lack of scalable distribution and analytical query functionality.
Innovation Solution
A method involving a distribution server that generates hashes from network traffic records to identify sessions and distribute them to worker computing devices, allowing for efficient transmission and analysis, including ranking IP addresses and providing network security assessments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If network traffic data is sampled instead of collected completely, then data volume is reduced, but analysis accuracy and completeness deteriorate
Solution Approach 1:
The system segments network traffic data into discrete records with specific fields (source IP, destination IP, source port, destination port, protocol, bytes, packets, start time, last update). This segmentation allows complete records to be distributed across multiple computing devices while maintaining full analytical capability without requiring complete data collection at a single location.
Solution Approach 2:
The patent introduces a new dimensional approach by creating hash values from record fields and using these hashes as distribution keys. This transforms the data management problem from a single-dimension storage model to a multi-dimensional model where data can be simultaneously stored, distributed, and queried across multiple devices based on hash-based routing, enabling complete data analysis without centralization.
2Productivity
If multiple computing devices are deployed to analyze network traffic, then processing capacity increases, but data distribution and query functionality become more complex
Solution Approach 1:
The system introduces hash values as an intermediary mechanism between network traffic records and computing devices. Instead of implementing complex peer-to-peer data distribution logic, the patent uses hash-based routing where records are distributed to computing devices based on hash values derived from record fields. This intermediary simplifies the distribution architecture while enabling scalable processing across multiple devices.
Solution Approach 2:
The patent transforms data distribution from a topology-based approach to a parameter-based approach. By changing the distribution key from device identifiers or network topology information to hash values derived from record content fields, the system enables simple, scalable distribution logic that doesn't depend on the number or arrangement of computing devices, thereby reducing operational complexity.
3Loss of information
If complete network traffic data is captured, then analysis completeness improves, but data storage and processing requirements increase
Solution Approach 1:
The system segments complete network traffic data into standardized records with fixed fields, allowing the data to be divided and distributed across multiple computing devices. Each device stores a portion of the complete dataset, reducing individual storage requirements while maintaining collective data completeness for full analysis capability.
Solution Approach 2:
The patent implements a distributed copying model where complete network traffic records are replicated across multiple computing devices based on hash-based routing. Each device holds copies of relevant records needed for its processing role, enabling complete data analysis without requiring any single device to store the entire dataset, thereby reducing overall storage requirements while maintaining analysis completeness.
Data Source
AI summary
A method includes transmitting, by a distribution server, to each of a plurality of worker computers, a request for an enumeration of Internet Protocol (IP) addresses ranked according to a criterion. The method includes receiving, by the distribution computer, from a first of the plurality of worker computers, a first partial enumeration of the requested IP addresses ranked according to the criterion, the first partial enumeration stored in a hash table. The method includes receiving, by the distribution computer, from a second of the plurality of worker computers, a second partial enumeration of the requested IP addresses ranked according to the criterion, the second partial enumeration stored in a hash table. The method includes generating, by the distribution computer, a combined enumeration including the first partial enumeration and the second partial enumeration, the combined enumeration ranked according to the criterion. The distribution computer deduplicates the combined enumeration.