Real-Time Threat Detection in High Volume Network Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network security systems are limited in detecting novel or non-signature based threats due to their narrow view and reliance on deterministic procedures, and they struggle with analyzing large, unwieldy data logs from protection devices like firewalls and proxies, making it impractical to identify threats in real time.
Innovation Solution
A high-speed threat detection system that includes a data analysis unit with a relational database engine to analyze network data in real time, package it into time-based slices, identify patterns, and output relevant patterns to operators, using methods like rate-detection and rare value detection to reduce data volumes and identify potential threats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If protection devices (firewall, proxies) log data streams to detect threats, then threat detection capability is improved, but data log volume becomes excessively large and unwieldy for real time analysis
Solution Approach 1:
The patent extracts only the relevant threat information from the massive data logs generated by protection devices. Instead of analyzing the entire log volume, the system extracts and focuses on specific patterns and anomalies that indicate threats, thereby reducing the effective data volume for analysis while maintaining detection capability.
Solution Approach 2:
The patent introduces an intermediary data processing layer between the protection devices and the analysis system. This intermediary layer pre-processes and filters the raw log data, transforming it into a more manageable format that highlights only the critical threat indicators, thus bridging the gap between high data volume and real-time analysis requirements.
2Measurement precision
If known devices analyze data flow using signatures of known attack vectors, then deterministic threat identification is improved, but ability to detect novel or non-signature based threats deteriorates
Solution Approach 1:
The patent employs dynamic analysis techniques that adapt to evolving threat patterns. Instead of relying on static signatures, the system dynamically learns from network behavior patterns and can identify anomalies that indicate novel threats. This dynamic approach allows the system to detect previously unknown attack vectors by recognizing deviations from normal behavior patterns.
Solution Approach 2:
The patent changes the detection parameters from fixed signatures to statistical patterns and behavioral anomalies. By analyzing changes in network traffic parameters over time and identifying deviations from established baselines, the system can detect novel threats that do not match any known signature, thereby enhancing both precision and adaptability.
3Reliability
If network security systems log all data streams for analysis, then comprehensive threat coverage is improved, but processing time and system complexity increase
Solution Approach 1:
The patent segments the data stream analysis into multiple stages: initial filtering of obviously malicious traffic, pattern recognition in remaining data, and detailed analysis only of suspected threats. This segmentation allows comprehensive threat coverage while reducing processing time by avoiding exhaustive analysis of all data streams.
Solution Approach 2:
The patent performs preliminary actions by pre-processing and filtering data streams before they enter the main analysis queue. By identifying and removing clearly benign or already-handled traffic in advance, the system reduces the volume of data requiring full analysis, thereby decreasing processing time while maintaining comprehensive coverage through the preliminary filtering stage.
Data Source
AI summary
A high speed detection system and method capable of generating audits of investigable patterns from log data using techniques for grouping and filtering the data so as to create vectors of patterns which can be then further analyzed by applying conditional filters to conclude that a threat may be active has been created to solve at least the above discussed problems.