Intelligent Network Recorder Load Balancing for Wire-Speed Capture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-speed networks pose challenges in capturing and analyzing network data packets due to high transmission rates, making it difficult to detect and respond to network attacks effectively, especially in virtual, hybrid, or cloud architectures.
Innovation Solution
The implementation of high-speed intelligent network recorders (HSINRs) that utilize an array of intelligent hard drives and hash tags for load balancing, allowing for minimal latency in packet capture and storage across various network loads and interfaces, enabling efficient analysis of network flows.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional network security monitoring is used, then network administrators can detect and respond to intrusions, but it becomes increasingly challenging to capture and analyze network behavior at 10 GbE or higher transmission rates
Solution Approach 1:
The system segments network traffic capture and analysis across multiple distributed intelligent network recorder nodes, each capable of handling portions of the 10 GbE or higher traffic flow. This segmentation allows the system to maintain wire-speed capture capability while distributing the analytical workload to prevent any single point from becoming a bottleneck.
Solution Approach 2:
The patent introduces intelligent network recorder nodes as intermediary devices between the high-speed network traffic and the analysis tools. These recorders capture packets at wire speed, perform initial filtering and metadata extraction, and present processed data to analysis systems, thereby mediating between the high-speed network and the analytical processes.
2Reliability
If data packet communication is stored for incident analysis, then administrators can determine what data was compromised and how the attack occurred, but storage capacity and retrieval time become constraints
Solution Approach 1:
The intelligent network recorder nodes perform preliminary actions by capturing and storing packet data in real-time as it flows through the network, rather than attempting to retrieve data after an incident is detected. The system pre-processes packets by extracting metadata, categorizing traffic flows, and organizing stored data with indexes, so that when an incident occurs, analysis can begin immediately with pre-organized data rather than searching through raw captured traffic.
3Loss of time
If high-speed packet capture is implemented, then minimal latency in packet capture and storage is achieved, but system complexity increases with array of intelligent hard drives and load balancing mechanisms
Solution Approach 1:
The system employs dynamic load balancing that adapts to changing network conditions and traffic patterns. The load balancer continuously monitors the state of intelligent network recorder nodes and dynamically redistributes traffic flows to maintain optimal performance. This dynamic behavior allows the system to handle varying loads efficiently without requiring over-provisioning for peak conditions, thereby managing complexity while maintaining high-speed capture capability.
Data Source
AI summary
A high speed intelligent network recorder for recording a plurality of flows of network data packets into and out of a computer network over a relevant data time window is disclosed. The high speed intelligent network recorder includes a printed circuit board; a high speed network switching device mounted to the printed circuit board; and an X column by Y row array of a plurality of intelligent hard drives with micro-computers mounted to the printed circuit board and coupled in parallel with the high speed network switching device.A method for network recording is disclosed. In one embodiment, the method includes the following: receiving a plurality of incoming packets, wherein each incoming packet belongs to a conversation flow; forming a capture stream of packet records for the incoming packets; and performing intelligent load balancing on the capture stream of packet records, the load balancing including reading the metadata for each packet record, determining a packet record is part of either a hot flow or a cold flow, selecting a destination node for each packet record based on the flow hash, and steering the packet record to one of a plurality of encapsulation buffers based on the destination node, wherein a cold flow tends to be maintained in a flow coherency at a node. The method may further include operations that include querying and back-testing in order to enable distributed analytics by using low cost, low band width nodes.