Extended Bloom Filter Log Lightweighting for Duplicate Threat Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log processing methods face delays in data processing due to significant memory or database size requirements for duplicate removal, especially with high cardinality data, leading to performance degradation and loss of information, which affects threat detection accuracy.
Innovation Solution
An electronic apparatus utilizing an extended Bloom filter to compress and filter duplicate logs by extracting item-specific data, applying hash functions, and updating duplication counts, enabling efficient log processing and threat detection with reduced computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If memory cache or database is used to remove duplicates, then duplicate removal is achieved, but memory size increases and performance degradation occurs
Solution Approach 1:
The patent uses Bloom filter data structures to create a compact copy representation of log data for duplicate detection. Instead of storing actual log entries in memory, the system stores hash values in Bloom filters, achieving duplicate removal with minimal memory consumption while maintaining detection accuracy
Solution Approach 2:
The patent transforms the memory storage problem by changing the representation parameter from storing complete log data to storing only hash values in Bloom filter structures. This parameter change reduces memory requirements from O(n) to O(k) where k is the Bloom filter size, independent of the number of logs processed
2Productivity
If memory cache is used for duplicate removal, then processing speed improves, but memory size increases
Solution Approach 1:
The system creates a lightweight copy mechanism using Bloom filters that can be quickly queried for duplicate detection without requiring large memory allocations. The hash-based copying approach enables fast O(1) average time complexity for duplicate checks while using minimal memory space
Solution Approach 2:
The patent extracts only the essential identifying features of log data (hash values of specific fields) and stores them in Bloom filters, leaving out the complete log data. This extraction approach enables fast duplicate detection with minimal memory usage by storing only the necessary identification information
3Reliability
If duplicate checks are performed, then duplicate removal is achieved, but information on filtered logs is lost
Solution Approach 1:
The patent segments log data into specific fields (source IP, destination IP, port, protocol) and applies hash functions to these segmented components. This segmentation allows the system to detect duplicates based on key identifying fields while preserving the complete original log data for threat analysis, preventing information loss
Solution Approach 2:
The Bloom filter acts as an intermediary structure that stores only hash values for duplicate detection purposes. The actual log data is preserved separately and only marked for filtering when the Bloom filter indicates a duplicate. This intermediary approach ensures duplicate detection accuracy while maintaining access to complete log information for security analysis
4Reliability
If large volume of log data is processed, then comprehensive threat detection is achieved, but computational resources increase
Solution Approach 1:
The system performs preliminary duplicate detection using Bloom filters before applying computationally intensive rule-based threat detection. By pre-filtering duplicates with the lightweight Bloom filter structure, the system reduces the volume of data requiring expensive computational analysis, thereby lowering overall computational resource consumption while maintaining comprehensive threat detection capability
Data Source
AI summary
An operation method of an electronic apparatus is disclosed. The operation method according to the present disclosure comprises extracting item-specific data that matches a plurality of preconfigured items from within log data, applying the extracted item-specific data to a hash function corresponding to each item to obtain an item-specific hash value and checking for duplication of the log data based on whether the item-specific hash value is duplicated, updating a duplication count based on the checked duplication status, and detecting a threat based on the identified item-specific data and the updated duplication count.