Probabilistic Sketches for Scalable Network Traffic Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions for identifying security risks on computer networks rely on text-based log analysis and centralized logging systems, which become inefficient as data size grows, and lack effective processing of network packet captures for real-time threat detection.
Innovation Solution
A computer-readable data model indexed by multiple sets of keys representing different time intervals, allowing for dynamic resolution and processing of time-indexed data, including sketches that approximate network traffic for real-time security risk analysis, using Map/Reduce jobs and distributed databases for scalable processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If text-based log analysis and centralized logging systems are used, then security risk identification is enabled, but processing efficiency deteriorates as data size grows
Solution Approach 1:
The patent segments the centralized logging system into distributed components across multiple servers. Each server maintains local sketches of network traffic data, eliminating the need to centralize all logs in one system. This segmentation allows parallel processing of security analysis while maintaining comprehensive coverage, thus improving processing efficiency without sacrificing security risk identification capability.
Solution Approach 2:
The patent introduces a new dimensional approach by using probabilistic sketches (hyperloglog, count-min sketches) to represent network traffic data. Instead of storing complete log entries, the system uses compact probabilistic data structures that enable security analysis in a transformed dimension, achieving both efficient processing and reliable threat detection.
2Productivity
If logs are abstracted and aggregated further, then data processability is improved, but data resolution is lost
Solution Approach 1:
The patent changes the parameter representation of log data by using probabilistic sketches instead of complete log entries. These sketches maintain essential statistical properties (cardinality, frequency distributions) while using significantly less space, enabling efficient processing without complete loss of data resolution. The sketches can be updated incrementally and support various query types.
Solution Approach 2:
The patent creates simplified copies of the original log data in the form of probabilistic sketches. These sketches are updated incrementally as new logs arrive, maintaining a compressed representation that preserves key characteristics of the data stream, enabling processability while retaining useful resolution for security analysis.
3Reliability
If point solutions for PCAP processing are used, then network packet capture analysis is enabled, but scalability is limited
Solution Approach 1:
The patent creates a universal system that can process multiple types of network data (PCAPs, logs, flow data) using the same probabilistic sketching infrastructure. The distributed architecture allows the system to scale from small to large deployments, and the sketches support multiple analysis functions (threat detection, traffic characterization, anomaly detection), making the solution adaptable and versatile.
Data Source
AI summary
The systems and methods described herein relate to storing values, such as sketches, that represent time indexed data related to network traffic. The model may be indexed by multiple sets of keys and time range values. It is an advantage that the index is essentially repeated for different time intervals. Utilization of the sketches avoids prohibitively large amounts of network traffic from overwhelming the monitoring computing device(s). Further, the probabilistic representative accuracy of the sketches is dynamically configurable. The time indexed data may represent data traffic on a computer network. The time indexed data may be data packets sent on the network. In one or more embodiments the time indexed data may be sketches that represent and/or approximate the data packets. Further, it is contemplated herein that the accuracy of the sketches' representation may be dynamically configurable.