Flow Log Compression via Tuple Symbol Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of network traffic flows and flow logs generated by cloud applications poses a challenge in processing, transporting, and storing these logs efficiently, as current methods require significant resources and can lead to high costs due to the large scale and resource intensity of flow logging.
Innovation Solution
The method involves creating compressed flow logs by identifying primary and secondary tuple definitions within flow log entries, associating symbols with these tuples, and including only necessary field values and timestamp offsets in the compressed logs, which are then transmitted to flow log consumers, thereby reducing storage and transmission requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If flow logs are stored in full detail for each network packet and event, then complete network traffic analysis is achieved, but storage requirements and processing resources increase significantly
Solution Approach 1:
The patent extracts only the most essential flow log fields (primary tuple fields such as source/destination IP addresses, ports, and protocol) while omitting less critical detailed information. This selective extraction maintains the ability to perform network traffic analysis while dramatically reducing storage requirements by storing only the extracted essential fields rather than complete packet details.
Solution Approach 2:
The patent segments flow log data into hierarchical levels: primary tuple fields that are always stored, secondary tuple fields that provide additional context when needed, and full packet details that are processed but not stored in compressed format. This segmentation allows the system to maintain analysis completeness for essential parameters while reducing overall storage burden.
2Loss of information
If all flow log fields are transmitted to central resources, then complete logging information is available, but transmission bandwidth and processing resources are consumed
Solution Approach 1:
The patent extracts and transmits only primary tuple fields (essential flow identification information) to central resources, while secondary tuple fields and detailed packet information are processed locally or omitted from transmission. This extraction approach ensures that central resources receive the most critical logging information needed for security analysis and monitoring while minimizing bandwidth consumption and processing overhead.
3Quantity of substance
If compression algorithms are applied to flow log data, then storage and transmission efficiency is improved, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary extraction of primary tuple fields at the source network appliance before transmission, organizing data into a compressed format upfront. This preliminary action reduces the amount of data that needs to be processed during transmission and storage operations, thereby reducing overall processing time despite the initial compression effort.
Solution Approach 2:
The patent changes the parameter representation of flow log data by converting detailed packet information into condensed tuple symbols and references. Instead of storing full packet payloads and detailed metadata, the system represents flows using compact tuple identifiers and references to secondary fields, dramatically reducing data volume while maintaining queryability through the symbolic representation system.
4Duration of action of stationary object
If detailed flow log entries are retained for extended periods, then historical analysis capability is maintained, but storage costs and resource requirements increase
Solution Approach 1:
The patent extracts and retains only the essential primary tuple fields that are necessary for historical security analysis and threat detection, while omitting detailed packet payloads and transient metadata from long-term storage. This extraction enables organizations to maintain compressed flow logs for extended periods (months or years) at fractionally reduced storage costs, as the extracted tuple data is sufficiently compact for mass retention.
Data Source
AI summary
A compressed flow log can be created from flow log entries. Flow log entries include field values for flow log fields. The entries can be compressed based on tuples (e.g., primary tuple, secondary tuple, tertiary tuple, etc.) specified by tuple definitions that specify the flow log fields in tuple content values. A first primary tuple content value can be based on a primary tuple definition and the field values of a first flow log entry included in flow log entries. A first primary tuple symbol can be associated with the first primary tuple content value. The first flow log entry can be used to create a first compressed flow log entry that includes the first primary tuple symbol. The first compressed flow log entry can be included in the compressed flow log, and the compressed flow log can be transmitted to a flow log consumer.