Anomaly Detection Using Correlation Coefficients for Cloud Data Flows

Solution Overview

Problem

Existing anomaly detection methods in cloud systems face increased processing loads and calculation times due to high data flow communication amounts, making it difficult to efficiently detect data anomalies.

Innovation Solution

Anomaly detection apparatus that classifies data flows based on similarity in time series changes, calculates correlation coefficients at normal and specific times, and determines anomalies by comparing these coefficients to a threshold, reducing the number of combinations for correlation analysis and optimizing discretization widths for each flow.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If correlation analysis is performed on all data flow pairs to detect anomalies, then detection accuracy is improved, but processing load and calculation time increase significantly

Engineering Contradiction:

Improveanomaly detection accuracyVSAvoidprocessing speed

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent segments data flows into groups based on their time series characteristics and correlation patterns. Instead of performing correlation analysis on all possible data flow pairs, the system divides them into manageable segments (groups) and performs analysis within each group, significantly reducing the total number of comparisons while maintaining detection accuracy for anomalies that affect correlated flows.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary classification of data flows into groups based on their time series characteristics before conducting correlation analysis. This preliminary action organizes the data in advance, allowing the system to efficiently identify which groups are likely to contain anomalies without performing exhaustive analysis on all data flows, thus improving processing speed while maintaining detection capability.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If correlation analysis is performed on all data flow pairs, then comprehensive anomaly detection is achieved, but computation resources increase

Engineering Contradiction:

Improveanomaly detection completenessVSAvoidcomputation resource consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent segments data flows into groups based on their time series characteristics and correlation patterns. Instead of performing correlation analysis on all possible data flow pairs, the system divides them into manageable segments (groups) and performs analysis within each group, significantly reducing the total number of comparisons while maintaining detection accuracy for anomalies that affect correlated flows.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies local quality by performing correlation analysis selectively on specific groups of data flows that exhibit similar characteristics, rather than uniformly analyzing all data flows. This allows the system to concentrate computational resources on groups that are more likely to contain anomalies, reducing overall computation resource consumption while maintaining reliable detection.

Inventive Principle:

Principle #3Local quality

3Productivity

If discrete time points are used for correlation calculation, then processing speed improves, but detection capability for prolonged anomalies decreases

Engineering Contradiction:

Improveprocessing speedVSAvoidprolonged anomaly detection capability

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent implements periodic action by performing correlation analysis at multiple discrete time points throughout the monitoring period. Instead of continuous analysis, the system periodically samples data flows at predetermined intervals, which maintains processing speed while ensuring that prolonged anomalies are detected through multiple measurement opportunities across different time periods.

Inventive Principle:

Principle #19Periodic action

Solution Approach 2:

The patent performs preliminary classification of data flows into groups based on their time series characteristics before conducting correlation analysis. This preliminary action organizes the data in advance, allowing the system to efficiently identify which groups are likely to contain anomalies without performing exhaustive analysis on all data flows, thus improving processing speed while maintaining detection capability.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10673721B2Anomaly detection apparatus, anomaly detection system, and anomaly detection method using correlation coefficients

Publication Date: 2020.06.02 HITACHI VANTARA LTD

AI summary

An anomaly detection apparatus for detecting data flow anomalies classes a plurality of data flows on the basis of similarity in time series changes in the data amounts of the data flows; calculates a correlation coefficient at a normal time and a correlation coefficient at a certain timing between at least two data flows belonging to the same class; and determines that at least one of the at least two data flows is anomalous when a difference between the correlation coefficient at the normal time and the correlation coefficient at the certain timing is greater than a predetermined threshold.