Anomaly Detection in Data Logs via Pattern Recognition

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional methods fail to effectively detect and alert production support teams to persistent data anomalies, leading to reduced application availability and end-user dissatisfaction, as they rely on predefined error thresholds that are often breached only after prolonged issues arise.

Innovation Solution

A system and method that utilize a processor to receive, trim, and compare data logs with previous logs, identify anomalies using trained models, and automatically trigger alerts based on pattern recognition, thereby bypassing the need for pre-defined thresholds and enhancing detection of persistent errors.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If predefined error thresholds are used to trigger alerts, then the system is simple to implement, but persistent anomalies are not detected until thresholds are breached after prolonged issues

Engineering Contradiction:

Improveease of alert triggeringVSAvoidanomaly detection reliability

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The system performs preliminary actions by continuously comparing current data logs with historical logs and identifying anomalies before they breach predefined thresholds. The trained model proactively detects patterns and triggers alerts in advance, preventing the wait-and-react approach of traditional threshold-based systems.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements feedback mechanisms where the trained model continuously learns from historical data log comparisons and adjusts its anomaly detection capabilities. This feedback loop enables the system to improve its detection accuracy over time while maintaining proactive monitoring without requiring manual threshold adjustments.

Inventive Principle:

Principle #23Feedback

2Use of energy by moving object

If traditional threshold-based alerting is used, then the system requires minimal processing resources, but application availability deteriorates due to delayed anomaly detection

Engineering Contradiction:

Improveprocessing resource consumptionVSAvoidapplication availability

Core Design Contradiction:

Use of energy by moving objectVSReliability

Solution Approach 1:

The system performs self-service by automatically using its own processed data and trained models to detect anomalies and generate alerts without requiring external intervention. The trained model autonomously compares current logs with historical data, identifies patterns, and triggers alerts independently, reducing the need for manual monitoring while improving detection timing.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system changes the parameter of anomaly detection from static predefined thresholds to dynamic pattern recognition based on trained models. This parameter transformation enables the system to adapt to changing data patterns and detect anomalies earlier, improving application availability while managing processing resources through efficient machine learning inference.

Inventive Principle:

Principle #35Parameter changes

3Ease of manufacture

If predefined thresholds are set by developers, then the system is easy to configure, but it fails to detect persistent anomalies that do not breach the threshold

Engineering Contradiction:

Improvesystem configuration easeVSAvoidanomaly detection precision

Core Design Contradiction:

Ease of manufactureVSMeasurement precision

Solution Approach 1:

The system replaces the mechanical threshold-based detection mechanism with an intelligent trained model that uses pattern recognition and machine learning. This substitution eliminates the need for manual threshold configuration while significantly improving anomaly detection precision, as the model can identify subtle patterns and persistent issues that fixed thresholds miss.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system transforms the configuration parameter from static developer-defined thresholds to dynamic trained model parameters learned from historical data. This parameter change maintains ease of configuration (developers simply train the model once) while dramatically improving measurement precision through automated, data-driven anomaly detection that adapts to changing system behavior.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS20240370328A1Method and system for triggering alerts on identification of an anomaly in data logs

Publication Date: 2024.11.07 JPMORGAN CHASE BANK NA

AI summary

A method and system for automatically triggering alerts for at least one anomaly are disclosed. The method includes receiving a plurality of data logs associated with a plurality of applications. Next, the method includes trimming the plurality of data logs into a set of data logs and comparing the set of data logs with each of a plurality of previously stored set of data logs. Next, the method includes identifying a set of new events associated with the at least one anomaly. Next, the method includes analyzing the set of new events to identify a pattern associated with the at least one anomaly and displaying, via a display, the pattern associated with the at least one anomaly to at least one entity. Thereafter, the method includes automatically triggering the alerts for the at least one anomaly based on the identification of the pattern associated with the at least one anomaly.