Hash Tag Load Balancing for Netflow Record Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional load balancing methods for network traffic in data centers, particularly at high speeds like 10 GbE, are inadequate for managing and analyzing Ethernet packets, especially in virtual, hybrid, or cloud architectures, as they struggle to distribute netflow records effectively across multiple collectors.
Innovation Solution
The implementation of hash tag load balancing using netflow generator appliances (NGAs) with data acquisition and generation (DAG) cards, which generate hash tags for packets and distribute them across multiple netflow collectors, ensuring even load balancing and minimizing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional load balancing methods are used for network traffic in data centers, then client distribution across servers can be achieved, but effective distribution and analysis of Ethernet packets at high speeds (10 GbE) cannot be accomplished
Solution Approach 1:
The patent segments the load balancing function by introducing hash tags that divide incoming Ethernet packet flows into distinct groups. Each hash tag corresponds to a specific netflow collector, enabling parallel processing of packet streams across multiple collectors simultaneously. This segmentation allows the system to handle high-speed traffic by distributing workloads effectively rather than processing packets sequentially through a single analyzer.
Solution Approach 2:
The patent introduces hash tags as an intermediary mechanism between incoming Ethernet packets and netflow collectors. These hash tags act as mediators that determine routing decisions, enabling the system to efficiently distribute packet analysis tasks across multiple collectors based on hash tag values derived from packet characteristics. This intermediary layer solves the contradiction by providing a scalable distribution mechanism that works at high speeds.
2Productivity
If netflow records are distributed across multiple collectors, then analysis capacity increases, but effective distribution and load balancing cannot be achieved with traditional methods
Solution Approach 1:
The patent changes the parameter used for load balancing from traditional IP address-based hashing to a more sophisticated hash tag mechanism that incorporates multiple packet characteristics. This parameter change enables better distribution of netflow records across collectors while maintaining computational efficiency. The hash tag parameter is derived from packet metadata and used to determine collector assignment, providing both improved productivity and manageable complexity.
Solution Approach 2:
The system implements self-service load balancing where each netflow record carries its own hash tag that automatically determines its destination collector. This eliminates the need for complex centralized scheduling or dynamic load assessment mechanisms. The hash tag essentially encodes the routing decision within the record itself, allowing collectors to independently process their assigned records without coordination overhead, thus achieving high productivity with relatively simple device architecture.
3Reliability
If hash tag load balancing is implemented, then even distribution of netflow records across collectors is achieved, but additional processing steps are required
Solution Approach 1:
The patent applies preliminary action by calculating and attaching hash tags to netflow records during the initial packet capture and classification phase, before the records need to be distributed to collectors. This pre-computation of hash tags ensures that the distribution decision is made early in the processing pipeline, avoiding the need for complex real-time load assessment at the collector level. The additional processing step is performed once during record creation, enabling reliable even distribution without ongoing complexity.
Data Source
AI summary
A netflow generator appliance is provided. The netflow generator appliance is configured to perform operations, including receiving a plurality of netflow records at a netflow generator appliance, each netflow record including at least a hash tag; converting a first hash tag of the first netflow record into a first percentage based on a load balancing function; based on the first percentage, storing the first netflow record in a first queue of data that are scheduled to be sent to a first netflow collector; converting a second hash tag of a second netflow record into a second percentage based on the load balancing function, wherein the second percentage differs from the first percentage; and based on the second percentage, storing the second netflow record in a second queue of data that are scheduled to be sent to a second netflow collector.