Log File Compression Using Relevance Scoring and Hierarchical Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression methods for log files, particularly for textual data, are inefficient due to their reliance on lossless compression, which limits the achievable compression ratio, making it difficult to transmit large volumes of data over limited bandwidth channels without excessive storage requirements.
Innovation Solution
A method employing two-level hierarchy clustering to encode unique values from log files, assigning security relevance scores, and selecting a subset of encoded values for lossy compression, which reduces entropy and allows controlled compression size adaptation based on system needs, combined with anomaly detection using machine learning algorithms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If lossless compression methods are used for log files, then information integrity is maintained, but compression ratio is limited
Solution Approach 1:
The patent changes the fundamental parameter of compression by transitioning from lossless to lossy compression methods. It introduces a relevance scoring system that assigns importance values to different log entries, allowing the system to selectively compress or discard less important information while maintaining critical data integrity. This parameter change enables achieving higher compression ratios by accepting controlled information loss in non-critical log entries.
Solution Approach 2:
The patent applies different compression strategies to different parts of the log file based on their relevance scores. High-relevance log entries are preserved with minimal compression to maintain integrity, while low-relevance entries undergo aggressive lossy compression or are discarded entirely. This local quality approach allows the system to optimize compression ratios for specific portions of the data without compromising overall information integrity.
2Loss of information
If larger volumes of log data are transmitted, then more complete information is provided, but bandwidth consumption increases
Solution Approach 1:
The patent implements partial action by selectively transmitting only the most relevant log entries rather than compressing and transmitting all log data. The relevance scoring system identifies and prioritizes critical log entries for transmission, while less important entries are either compressed more aggressively or discarded. This partial transmission approach reduces bandwidth consumption while maintaining sufficient information completeness for monitoring purposes.
3Measurement precision
If all log entries are retained for analysis, then detection accuracy is improved, but storage requirements increase
Solution Approach 1:
The patent extracts and retains only the most relevant log entries for storage and analysis based on their relevance scores. By applying the relevance scoring system, the system identifies and extracts critical log entries that are most useful for anomaly detection, while discarding or heavily compressing less important entries. This extraction approach maintains detection accuracy by preserving the most informative data while significantly reducing storage requirements.
Data Source
AI summary
A method and apparatus for log files of data compression are disclosed. The method comprising: classifying each of a plurality of lines in a plurality of the log files of data with at least two levels hierarchy clustering comprising identifying a plurality of strings repeated in the plurality of lines of the plurality of log files of data. Creating a table matching each of the plurality of strings to a unique value. Creating a vector encoding the unique value matched to each of the plurality of strings using the table. Assigning each of the encoded unique values in the vector, a security relevance score according to the classification of the plurality of lines; and selecting a subset of the encoded unique values such that the encoded unique values in the vector are filtered according to the security relevance score of each unique value.