Datacenter Network Flow Queuing With Compression and Message Brokering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for managing network flow information in datacenters expend significant processing resources, leading to inefficiencies and latencies that compromise reaction times to threats and network performance.
Innovation Solution
A system and method that includes a network flow collection device with modules for monitoring, compression, encapsulation, and serialization, coupled with a message brokering module operating under the rdKafka utility framework, to optimize resource usage and maintain network performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional techniques are used to collect and analyze network flow information, then network security monitoring and performance assessment are achieved, but processing resources are excessively consumed leading to inefficiencies and latencies
Solution Approach 1:
The patent extracts and separates the network flow information processing from the main datacenter processing infrastructure. By using dedicated network flow collection devices and message brokering modules, the system isolates the resource-intensive flow analysis functions, preventing them from consuming valuable datacenter processing resources while maintaining comprehensive security monitoring and performance assessment capabilities
Solution Approach 2:
The patent introduces intermediary components including network flow collection devices, message brokering modules, and client access modules that act as mediators between network routing devices and the analysis systems. These intermediaries handle the collection, consolidation, and initial processing of network flow information, reducing the processing burden on core datacenter systems while ensuring reliable security monitoring
2Reliability
If conventional techniques process network flow information in real-time, then threat detection capability is maintained, but network latencies increase compromising reaction times
Solution Approach 1:
The patent implements preliminary action by having network flow collection devices continuously collect, consolidate, and pre-process network flow information before threats materialize. The message brokering modules maintain ready queues of processed flow data, so when security events occur, the analysis systems can immediately query pre-processed information rather than waiting for real-time processing, significantly reducing reaction times while maintaining threat detection capability
Solution Approach 2:
The system employs periodic action through continuous background collection and periodic consolidation of network flow information. The message brokering modules periodically update and maintain queues of processed flow data, enabling the system to provide up-to-date network state information without requiring continuous real-time processing, thus reducing latencies while preserving threat detection effectiveness
3Reliability
If more processing resources are allocated to network flow analysis, then threat monitoring and performance assessment improve, but overall network traffic performance deteriorates
Solution Approach 1:
The patent segments the network flow processing function into separate dedicated devices and modules independent from the datacenter processing infrastructure. Network flow collection devices, message brokering modules, and client access modules form a segregated processing chain that handles flow analysis separately from core datacenter workloads, allowing improved threat monitoring and performance assessment without consuming datacenter processing resources and thus maintaining network traffic performance
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The disclosed systems and methods are directed to providing a configuration for the management of network flow information. The disclosed configuration provides a network flow collection device to receive/process network flow information packets provided by network routing devices, a compression module to compress the received network flow information packet stream, an encapsulation/serialization module configured to encapsulate/serialize the compressed packet data stream of network flow information into a consolidated network flow information data queue, a message brokering module to receive the consolidated network flow information data queue, segregate the consolidated information data queues into individual identifiable network flow information data queues and store for access by a client access module.