Log Pattern Matching via Tokenization and Database Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large volumes of diverse log data from multiple sources in complex computer networks is challenging due to the complexity of formats and the need for efficient data processing to identify meaningful patterns, which often results in prolonged execution times and resource-intensive processes.
Innovation Solution
A system that tokenizes data streams, stores tokenized objects in a token database, and computes patterns using a longest subsequence match algorithm, while managing the size of pattern and token databases by deleting older or less popular patterns to maintain efficiency and effectiveness, allowing for real-time pattern matching and data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the system processes and stores all tokenized objects and patterns from multiple log sources, then the completeness and accuracy of pattern matching is improved, but the execution time and resource consumption increase significantly
Solution Approach 1:
The patent extracts and stores only the essential pattern information in a pattern database, separating the pattern matching function from the raw log data. This allows the system to maintain high pattern matching accuracy while reducing the time required to process new log entries, as the system only needs to compare against stored patterns rather than analyze all raw data each time.
Solution Approach 2:
The system performs preliminary tokenization and pattern extraction during log processing, pre-computing patterns and storing them in the pattern database before they are needed for matching. This preliminary action enables faster real-time pattern matching operations, as the heavy computational work of pattern recognition has already been completed in advance.
2Adaptability or versatility
If the pattern database grows to include all possible patterns from diverse log formats, then the versatility and coverage of log analysis is improved, but the system complexity and resource requirements increase
Solution Approach 1:
The patent implements a universal tokenization framework that can handle multiple log formats through a common pattern structure. The system uses generic token types and pattern templates that can represent diverse log formats without requiring separate processing logic for each format, thereby maintaining versatility while reducing system complexity.
Solution Approach 2:
The system manages pattern database size by dynamically adjusting parameters such as pattern retention criteria, minimum pattern frequency thresholds, and database size limits. This allows the system to maintain high log format coverage by retaining essential patterns while automatically removing or consolidating redundant patterns, thus controlling system complexity.
3Loss of information
If the system retains all historical patterns in the pattern database, then the completeness of historical analysis is improved, but the resource consumption and processing overhead increase
Solution Approach 1:
The patent implements a pattern retention policy that selectively discards historical patterns based on criteria such as age, frequency, and relevance. The system recovers essential pattern information by maintaining pattern templates and metadata that can be regenerated or reconstructed when needed, allowing the system to reduce resource consumption by removing redundant historical patterns while preserving analytically valuable information.
Data Source
AI summary
A data processing method includes receiving a stream of digital data with a plurality of objects and, in response to receiving an object, tokenizing the object to create a tokenized object, and storing the tokenized object in a token database. The method further includes comparing the tokenized object to a plurality of other tokenized objects stored in the token database, computing a pattern associated with the tokenized object, storing the pattern in a pattern database, and managing a size of the pattern database by identifying, a subset of patterns that are eligible for deletion from the pattern database based on an age of each pattern, ranking each pattern of the subset based on a quality and a popularity metric, identifying, based on the ranking and from the subset, a second pattern and deleting the second pattern from the pattern database to produce an updated database.