Lossy Counters for Network Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cybersecurity frameworks in distributed computing environments provide insufficient information for analysts to determine the severity of malicious activity, as they lack contextual data, leading to inefficient analysis and resource overload when dealing with vast amounts of network data.
Innovation Solution
The implementation of lossy counters, which use a count sketch to efficiently track the most common network data items by discarding less frequent ones, allowing for the preservation of prevalent data while minimizing computational resources and storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If security frameworks monitor and collect all network data for analysis, then completeness of security information is improved, but processing and storage resource consumption increases
Solution Approach 1:
The patent extracts only the most frequent network data items from the complete dataset using lossy counters. By taking out and retaining only the top-k frequent items, the system achieves a balance between information completeness and resource efficiency, discarding less frequent items that consume processing resources without contributing significantly to security analysis
Solution Approach 2:
The patent changes the parameter of data representation from exact counts to approximate counts within bounded error. By allowing counts to be within a factor of two of the true value, the system reduces storage and processing requirements while maintaining sufficient accuracy for security anomaly detection
2Use of energy by moving object
If lossy counters discard less frequent network data items, then resource consumption is reduced, but information completeness deteriorates
Solution Approach 1:
The patent applies local quality by treating different network data items differently based on their frequency. Frequent items are tracked with higher precision and retained in the count sketch, while less frequent items are discarded. This localized differentiation optimizes resource allocation by focusing computational effort on the most relevant data
3Measurement precision
If analysts manually analyze all cybersecurity data, then analysis accuracy is improved, but analysis time increases
Solution Approach 1:
The patent performs preliminary action by pre-processing network data to identify and rank frequent items before analyst review. The lossy counter systematically identifies top-k frequent network data items in advance, so that when analysts receive the data, it is already organized and filtered to the most relevant items, reducing their analysis time while maintaining accuracy
Data Source
AI summary
A lossy counter counts distinct network data items. The lossy counter includes a count sketch bounded by a predetermined value to limit the number of distinct network data items included in the count sketch. The count sketch may include counts for a set of distinct network data items. The lossy counter has an associated time interval, and the first set of distinct network data items and the second set of distinct data items include timestamps that coincide with the time interval associated with the lossy counter.