Flow-Based Data Traffic Storage Using Hash Grouping and SSD Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data traffic storage and search systems, such as Netflow, face inefficiencies in indexing and searching due to packet-based storage, high costs, and lack of versatility, particularly when dealing with high-speed data traffic, which is exacerbated by the explosive growth in data traffic volumes.
Innovation Solution
An apparatus and method for storing and searching data traffic based on flow units composed of packets, utilizing a packet storage unit, flow generation unit, and metadata generation unit to group packets using hash functions, store flows on hard disks, and index data on a solid state drive (SSD), enabling efficient data processing and search operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data traffic is stored on a packet basis using conventional systems like Netflow, then packet-level processing is achieved, but indexing and searching efficiency is low and high-performance hardware is required
Solution Approach 1:
The patent segments data traffic into flow records that group related packets together, rather than storing individual packets. This segmentation at the flow level reduces the number of records to index and search, improving efficiency while reducing hardware requirements
Solution Approach 2:
The patent extracts essential flow information (five-tuple: source IP, destination IP, source port, destination port, protocol) from packets to create flow records. This extraction eliminates redundant packet-level data while preserving identifying characteristics, enabling efficient indexing and searching
2Reliability
If data traffic is stored on a packet basis, then complete packet data is preserved, but processing time increases due to explosive growth in data traffic volumes
Solution Approach 1:
The patent extracts only the necessary flow identification information from packets while discarding redundant packet-level data. This extraction maintains data completeness for flow analysis purposes while dramatically reducing the volume of data to process
Solution Approach 2:
The patent performs preliminary processing by aggregating packets into flow records before storage. This preliminary action consolidates data reduction and flow identification work done upfront, allowing faster processing later without compromising data completeness for analysis purposes
3Measurement precision
If conventional packet-based storage systems are used, then packet-level detail is maintained, but costs increase and application versatility is limited
Solution Approach 1:
The patent creates a universal flow record structure that can serve multiple application needs (traffic analysis, security monitoring, quality of service, billing) through a single standardized format. This multi-functionality eliminates the need for separate packet processing systems for different applications
Solution Approach 2:
The patent changes the fundamental parameter of data organization from packet-level to flow-level representation. This parameter change enables the same storage system to support diverse applications by adjusting query parameters rather than requiring different storage mechanisms
Data Source
AI summary
An apparatus and method for storing data traffic on a flow basis. The apparatus for storing data traffic on a flow basis includes a packet storage unit, a flow generation unit, and a metadata generation unit. The packet storage unit receives packets corresponding to data traffic, and temporarily stores the packets using queues. The flow generation unit generates flows by grouping the packets by means of a hash function using information about each of the packets as input, and to store the flows. The metadata generation unit generates metadata and index data corresponding to each of the flows, and stores the metadata and the index data.