Semantic Event Log Deduplication via Representative Records
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing event logging systems face challenges with excessively large log sizes due to redundancy, leading to high storage and transmission costs, as they often log the same events multiple times with different permutations, resulting in wasteful data processing and storage inefficiencies.
Innovation Solution
A semantic deduplication process that uses a two-step analysis to categorize event records and compute a semantic match score, incrementing a count for similar records and omitting redundant ones, thereby reducing log volume without losing essential information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all event records are logged completely without deduplication, then data integrity is maintained, but log size becomes excessively large leading to high storage and transmission costs
Solution Approach 1:
The patent extracts only the essential distinguishing features of event records by comparing them against representative records. Instead of storing complete duplicate records, it extracts and retains only the unique characteristics that differentiate similar events, thereby reducing log size while preserving data integrity.
Solution Approach 2:
The patent inverts the traditional approach by not starting with complete records and filtering them, but rather by maintaining representative records and only adding new records when they fundamentally differ from existing ones. This inversion allows for more efficient storage by preventing redundancy at the source.
2Reliability
If redundant event records are logged with different permutations, then comprehensive event coverage is achieved, but data processing efficiency decreases due to wasteful redundancy
Solution Approach 1:
The patent applies preliminary anti-action by proactively comparing incoming event records against existing representative records before storing them. This preliminary comparison prevents redundant records from entering the log in the first place, thereby maintaining comprehensive event coverage while eliminating processing inefficiencies caused by redundancy.
Solution Approach 2:
The patent changes the parameters used for record comparison by evaluating multiple attributes and characteristics of event records. By adjusting and comparing multiple parameters simultaneously, the system can accurately identify redundant records across different permutations while maintaining comprehensive coverage of unique events.
3Reliability
If complete records are stored without optimization, then full data recovery is possible, but storage costs increase prohibitively
Solution Approach 1:
The patent uses copying by maintaining representative records that capture the essential characteristics of event classes. Instead of storing complete copies of every variant, it creates and maintains representative copies that can represent multiple similar events, enabling data recovery of essential information while significantly reducing storage volume.
4Quantity of substance
If semantic deduplication is implemented, then log volume is reduced, but system complexity increases due to similarity computation
Solution Approach 1:
The patent applies segmentation by breaking down the event record comparison process into discrete, manageable steps: extracting key attributes, comparing against representative records, and determining similarity. This segmentation of the deduplication process reduces system complexity by making the computation more modular and manageable while still achieving significant log volume reduction.
Data Source
AI summary
Operations include: determining that the first event record does not meet similarity criteria in relation to any of a plurality of representative records in an event log; adding a new representative record, to the plurality of representative records, that is based on the first event record; determining that the second event record meets the similarity criteria in relation to a first representative record of the plurality of representative records; incrementing the numerical value, associated with the first representative record in the event log, that indicates the number of event records that are represented by the first representative record; generating log data that (a) comprises the incremented numerical value associated with the first representative record and (b) does not include the second event record; and updating the event log based on the log data to generate an updated event log that does not include the second event record.