Datacenter Network Flow Queues for Low-Latency Threat Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for managing network flow information in datacenters expend substantial processing resources, leading to inefficiencies and latencies that compromise reaction times to threats and network performance.
Innovation Solution
A system and method that integrates a network flow collection device with modules for monitoring, compression, encapsulation, and serialization, coupled with a message brokering module operating under an rdKafka framework, to optimize resource usage and maintain network performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional techniques are used to collect and analyze network flow information, then network security monitoring and performance assessment are achieved, but processing resources are substantially expended leading to inefficiencies and latencies
Solution Approach 1:
The patent segments network flow information processing into multiple independent components: flow collection agents on individual hosts, centralized collection servers, and distributed analysis modules. This segmentation allows parallel processing of different flow streams, reducing overall processing latency and improving resource utilization efficiency while maintaining comprehensive security monitoring coverage.
2Measurement precision
If comprehensive network flow information is collected for threat detection, then security monitoring accuracy is improved, but processing latencies increase compromising reaction times
Solution Approach 1:
The patent implements preliminary action by pre-configuring flow collection templates, filtering rules, and analysis parameters before network traffic arrives. Collection agents are pre-deployed on hosts with configured monitoring parameters, enabling immediate processing of network flows without setup delays. This pre-preparation maintains high detection accuracy while minimizing processing latencies.
Solution Approach 2:
The patent introduces intermediary components including buffer queues between flow collection and analysis stages, and intermediate processing servers that perform initial filtering and aggregation. These intermediaries decouple the collection and analysis processes, allowing continuous collection without blocking analysis operations, thereby maintaining both accuracy and fast reaction times.
3Measurement precision
If detailed network flow analysis is performed to assess performance, then network performance monitoring is improved, but overall network efficiency deteriorates due to resource consumption
Solution Approach 1:
The patent applies partial action by implementing selective monitoring that focuses analysis resources on specific flow types, protocols, or suspicious patterns rather than uniformly analyzing all network traffic. Collection agents filter flows based on pre-defined criteria, performing detailed analysis only on relevant traffic. This approach maintains accurate performance monitoring for critical flows while significantly reducing overall processing resource consumption.
Data Source
AI summary
The disclosed systems and methods are directed to providing a configuration for the management of network flow information. The disclosed configuration provides a network flow collection device to receive/process network flow information packets provided by network routing devices, a compression module to compress the received network flow information packet stream, an encapsulation/serialization module configured to encapsulate/serialize the compressed packet data stream of network flow information into a consolidated network flow information data queue, a message brokering module to receive the consolidated network flow information data queue, segregate the consolidated information data queues into individual identifiable network flow information data queues and store for access by a client access module.