Multi-Cloud Network Flow Analysis Across Varying Log Schemas
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multi-cloud environments, network flow logs captured by Cloud Service Providers (CSPs) have varying schemas, making it challenging to derive meaningful threat intelligence due to differences in attributes, and existing systems lack effective methods to process and correlate these logs across multiple CSPs.
Innovation Solution
A cloud-based system processes network flow logs from multiple CSPs, enriching critical fields, transforming and aggregating them, and identifying threats, while providing integration with ITSM and SIEM tools, and offering a GUI for visualization, with features like reprocessing corrupt or incomplete logs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If network flow logs are captured from multiple CSPs with varying schemas, then comprehensive threat intelligence coverage is improved, but data processing complexity increases
Solution Approach 1:
The patent introduces a standardized data schema as an intermediary layer between multiple CSP-specific log formats and the threat intelligence processing system. This mediator schema abstracts the variations in CSP log formats (AWS VPC Flow Logs, Azure NSG Flow Logs, GCP VPC Flow Logs) into a unified structure, allowing comprehensive multi-CSP coverage without proportionally increasing processing complexity.
Solution Approach 2:
The system dynamically adjusts processing parameters based on the source CSP and log format. By changing parameters such as field mapping configurations, schema versions, and processing rules according to the specific CSP being processed, the system handles diverse log formats efficiently without requiring completely separate processing pipelines for each provider.
2Speed
If network flow logs are processed in real-time for threat detection, then threat detection speed is improved, but processing resource consumption increases
Solution Approach 1:
The patent implements periodic batch processing of network flow logs combined with real-time streaming for critical threat indicators. By processing logs in periodic batches at optimized intervals while maintaining real-time detection capabilities for high-priority threats, the system achieves fast threat detection without continuously consuming maximum processing resources.
Solution Approach 2:
The system applies partial processing to log entries based on their threat relevance. Critical fields such as source/destination IPs, ports, and protocol information are processed in full real-time, while less critical metadata is processed selectively or in delayed batches. This partial action approach maintains detection speed for important threats while reducing overall resource consumption.
3Measurement precision
If all network flow log fields are processed and enriched, then threat intelligence quality is improved, but processing time increases
Solution Approach 1:
The patent applies different processing depths to different fields within the network flow logs based on their threat intelligence value. Critical security fields such as source IP, destination IP, port, protocol, and action (allow/deny) receive full enrichment and validation, while less critical fields receive minimal or selective processing. This local quality approach ensures high threat intelligence quality for security-critical attributes without processing all fields at maximum depth.
Solution Approach 2:
The system performs preliminary filtering and basic enrichment of network flow logs before full processing. By pre-identifying and prioritizing logs containing threat indicators (such as denied connections, unusual ports, or known malicious IPs) and performing basic field validation beforehand, the system ensures that full enrichment is applied only to high-priority logs, reducing overall processing time while maintaining intelligence quality for critical threats.
Data Source
AI summary
The present disclosure includes systems and methods for multi-cloud network analysis and threat intelligence correlation. In various embodiments, systems are adapted to perform steps of retrieving network flow logs associated with one or more Virtual Private Cloud (VPC) networks; processing the network flow logs to identify new files and enriching critical fields from the network flow logs; transforming and aggregating the network flow logs for further processing; and identifying threats associated with the one or more VPC networks.