Network Metadata Processing System for Real-Time Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network monitoring systems face challenges in analyzing and storing massive amounts of network metadata in real-time due to its high volume, diverse formats, and complexity, leading to difficulties in correlation and storage, which is referred to as the 'Big Data' problem.
Innovation Solution
A method and system that processes network metadata by validating, filtering, aggregating, and de-duplicating incoming packets, converting them into a common format, and applying policies to reduce data volume, allowing for real-time analysis and correlation, while also supporting multiple data formats such as NetFlow, syslog, and IPFIX, and enabling near-real-time processing and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If network metadata is collected from multiple sources in real-time, then the volume of information available for analysis increases, but the complexity of processing and storing the data increases
Solution Approach 1:
The patent segments the network metadata processing system into distinct functional modules: collectors that gather data from multiple sources, normalizers that standardize formats, analyzers that process the data, and exporters that deliver results. This segmentation allows each component to handle specific tasks independently, reducing overall system complexity while maintaining the ability to process large volumes of metadata from diverse sources.
Solution Approach 2:
The patent introduces intermediary components including a central repository that acts as a buffer between collectors and analyzers, and normalizers that serve as mediators to convert diverse metadata formats into a standardized structure. These intermediaries decouple the complexity of data collection from data analysis, allowing the system to handle high-volume heterogeneous data without proportionally increasing processing complexity.
2Speed
If network metadata is processed at high speed to enable real-time analysis, then the timeliness of security insights improves, but the processing requirements and resource consumption increase
Solution Approach 1:
The patent applies preliminary action through the normalizer component that pre-processes and standardizes metadata formats before they reach the analyzer. By performing format normalization and validation in advance, the system reduces the computational burden during real-time analysis, enabling faster processing without proportionally increasing resource consumption during critical analysis phases.
Solution Approach 2:
The patent implements local quality by having different processing stages with specialized capabilities: collectors optimize for data gathering, normalizers for format standardization, and analyzers for security event detection. Each component is optimized for its specific function, allowing the system to achieve high processing speeds while consuming resources efficiently by avoiding unnecessary processing operations at each stage.
3Adaptability or versatility
If diverse network metadata formats from different device types are integrated, then the comprehensiveness of network monitoring improves, but the difficulty of correlation and integration increases
Solution Approach 1:
The patent implements universality through the normalizer component that can handle multiple metadata formats (NetFlow, syslog, IPFIX, and other network device protocols) and convert them into a standardized internal representation. This universal interface allows the system to integrate comprehensive data from diverse network devices without requiring separate processing logic for each device type, thereby reducing integration complexity while maintaining broad format compatibility.
4Quantity of substance
If all network metadata is stored for later analysis and compliance, then the completeness of historical data improves, but the storage requirements increase
Solution Approach 1:
The patent extracts only the essential and relevant features from raw network metadata during the normalization and analysis stages, storing processed information in the central repository rather than raw data. By extracting key security-relevant attributes and discarding redundant information, the system maintains complete historical data for compliance while significantly reducing storage requirements compared to retaining all original metadata.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method and system for processing network metadata is described. Network metadata may be processed by dynamically instantiated executable software modules which make policy- based decisions about the character of the network metadata and about presentation of the network metadata to consumers of the information carried by the network metadata. The network metadata may be type classified and each subclass within a type may be mapped to a definition by a unique fingerprint value. The fingerprint value may be used for matching the network metadata subclasses against relevant policies and transformation rules. For template- based network metadata such as NetFlow v9, an embodiment of the invention can constantly monitor network traffic for unknown templates, capture template definitions, and informs administrators about templates for which custom policies and conversion rules do not exist. Conversion modules can efficiently convert selected types and/or subclasses of network metadata into alternative metadata formats.