Semantic Deduplication for Event Log Volume Reduction

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing event logging systems face challenges with excessively large log sizes, redundancy in event logging, and unpredictable log volumes, leading to inefficient storage and processing.

Innovation Solution

A semantic deduplication process that dynamically adjusts the level of similarity for identifying duplicates in real-time, based on event stream characteristics such as event rate, processing time, and available resources, using machine learning models to predict the number of representative events.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If lossless logging is implemented to retain all data, then data integrity is maintained, but storage volume and processing costs increase excessively

Engineering Contradiction:

Improvedata integrityVSAvoidlog volume

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent extracts and removes redundant duplicate events from the event log, keeping only representative events that capture the essential information. This extraction principle reduces log volume by eliminating repeated occurrences of the same event type while maintaining data integrity for analysis purposes.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent discards redundant duplicate events that provide no additional analytical value, while recovering and preserving the essential event patterns through representative event selection. This allows the system to discard unnecessary data while recovering the core information needed for security monitoring and analysis.

Inventive Principle:

Principle #34Discarding and recovering

2Productivity

If log volume is reduced through deduplication, then storage and processing efficiency improve, but data loss may occur

Engineering Contradiction:

Improveprocessing efficiencyVSAvoiddata loss

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

The patent employs feedback mechanisms where the system continuously monitors event stream characteristics, deduplication performance, and resource availability. Based on this feedback, it dynamically adjusts the deduplication level to optimize the balance between reducing log volume and maintaining data integrity, preventing excessive data loss while improving processing efficiency.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent changes the parameter of deduplication level from a fixed value to a dynamic parameter that adjusts based on event stream characteristics and resource availability. By modifying this parameter in real-time, the system optimizes the trade-off between processing efficiency and data retention, ensuring that data loss is minimized while still achieving volume reduction.

Inventive Principle:

Principle #35Parameter changes

3Quantity of substance

If high deduplication level is applied to reduce redundancy, then storage optimization improves, but event processing time increases

Engineering Contradiction:

Improvelog sizeVSAvoidprocessing time

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent makes the deduplication level dynamic rather than static, allowing it to adjust in real-time based on event stream characteristics and system resource availability. This dynamic approach enables the system to apply higher deduplication levels when resources are abundant and lower levels when processing speed is critical, optimizing the balance between log size reduction and processing time.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent implements periodic reassessment of deduplication parameters based on changing event stream characteristics and resource availability. Rather than applying a constant deduplication level, the system periodically adjusts the deduplication intensity, allowing it to optimize storage optimization at different time intervals while managing processing time constraints.

Inventive Principle:

Principle #19Periodic action

4Device complexity

If fixed deduplication strategy is used, then system complexity is reduced, but adaptability to varying event volumes decreases

Engineering Contradiction:

Improvesystem complexityVSAvoidadaptability to event volume

Core Design Contradiction:

Device complexityVSAdaptability or versatility

Solution Approach 1:

The patent enables the deduplication system to self-adjust and self-optimize by automatically monitoring event stream characteristics and resource availability, then autonomously adjusting the deduplication level without external intervention. This self-service capability allows the system to adapt to varying event volumes dynamically while maintaining relatively simple operational complexity, as the adjustment logic is embedded in the automated control mechanism.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS12204509B1Auto-scaling for semantic deduplication of event logs

Publication Date: 2025.01.21 ORACLE INT CORP

AI summary

Techniques for autoscaling a deduplication process include receiving a first plurality of event records in an event stream during a first period of time and determining real-time characteristics of the event stream based on the event records received during the first period of time. Based on the real-time characteristics of the event stream, a level of similarity for a deduplication process is selected to apply for future event records received in the event stream. The deduplication process may also be enabled or disabled based on the real-time characteristics. A second plurality of event records is received in the event stream during a second period of time. If deduplication is enabled, the second plurality of event records are deduplicated based on the level of similarity.