Semantic Filtering for Event Log Data Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Service providers face challenges in efficiently filtering large volumes of event log data, leading to unnecessary resource usage due to extraneous data, and traditional static filters are not scalable or effective in identifying relevant data.
Innovation Solution
The implementation of semantic filtering, which involves retrieving training data, performing vectorization and tokenization, computing semantic scores, transforming scores using a modulation function, and fitting a filtering function to reduce data processed by the system, allowing for adaptive and accurate filtering of incoming data streams.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If traditional static filters are used to filter event log data, then device complexity is reduced, but measurement precision and filtering accuracy deteriorate
Solution Approach 1:
The patent transforms static filter parameters into dynamic semantic parameters. The filtering system uses vectorization to convert text data into numerical vectors, and dynamically adjusts filtering thresholds based on semantic similarity scores calculated from training data. This allows the system to adapt filtering parameters automatically rather than using fixed static thresholds.
Solution Approach 2:
The patent replaces traditional mechanical keyword-matching filters with a semantic-based filtering mechanism. Instead of using rigid rule-based systems, the invention employs vector space models and semantic similarity calculations to determine data relevance, enabling more accurate and flexible filtering without requiring complex manual configuration.
2Measurement precision
If all event log data is processed without filtering, then measurement precision is maintained, but productivity and resource efficiency deteriorate
Solution Approach 1:
The patent implements preliminary filtering action by training the semantic filtering model on historical data before actual data processing. The system pre-calculates semantic vectors and establishes filtering thresholds in advance, so that during runtime, data can be quickly filtered using pre-computed parameters rather than requiring full analysis of every data point.
Solution Approach 2:
The patent extracts only the most relevant features from event log data through vectorization and semantic analysis. Instead of processing entire data records, the system extracts key semantic vectors and compares them against training data to identify relevant entries, significantly reducing the amount of data that requires detailed processing while maintaining accuracy.
3Measurement precision
If semantic filtering with vectorization and tokenization is implemented, then measurement precision improves, but device complexity and computational resources increase
Solution Approach 1:
The patent segments the filtering process into distinct modular components: tokenization module, vectorization module, training module, and filtering module. Each component performs a specific function and can be independently optimized or replaced. This modular architecture manages complexity by breaking down the complex semantic filtering task into manageable, well-defined stages.
4Measurement precision
If semantic filtering is trained on large volumes of data, then measurement precision improves, but productivity and training time increase
Solution Approach 1:
The patent applies partial action by using a representative subset of training data rather than processing all available data during model training. The system identifies and uses key training samples that capture the essential semantic patterns, achieving effective filtering accuracy without the computational overhead of training on the complete dataset.
Data Source
AI summary
An approach is provided for semantic filtering to filter an input data stream of a data system to reduce an amount of data processed by the data system. The approach, for example, involves retrieving training data comprising a plurality of training data items responsive to a plurality of training queries conducted in a data system. The approach also involves performing a vectorization and a tokenization of the training data to generate a plurality of vectors comprising a plurality of tokens extracted from the plurality of training data items. The approach also involves computing a plurality of reference semantic scores for the plurality of vectors based on a semantic distance between the plurality of tokens. The approach also involves transforming the plurality of reference semantic scores into a plurality of transformed reference semantic scores based on a modulation function. The approach also involves fitting a filtering function to the plurality of transformed reference semantic scores. The approach also involves providing the filtering function to filter an input data stream of the data system to reduce an amount of data processed by the data system.