Anomaly Detection via Correlation Matrix Residuals
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional anomaly detection tools are impractical for high-dimensional data due to the complexity and size of transaction datasets, often focusing on select attributes, which limits their effectiveness in identifying interaction anomalies.
Innovation Solution
Generating attribute correlation matrices for interaction datasets across different time periods, computing residuals between these matrices, and using machine learning algorithms like the Isolation Forest to detect interaction anomalies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If conventional anomaly detection tools are used on high-dimensional transaction datasets, then the data size and complexity make the tools impractical, but focusing on select attributes reduces complexity at the cost of limiting effectiveness in identifying interaction anomalies
Solution Approach 1:
The patent extracts the correlation structure from the high-dimensional transaction data by computing correlation matrices between attributes. This extraction reduces the data to a more manageable form that captures essential relationships while reducing dimensionality and complexity for anomaly detection algorithms to process.
Solution Approach 2:
The patent transforms the original transaction data parameters into correlation matrix parameters. By changing the representation from raw transaction records to correlation coefficients between attributes, the data is reformulated into a format that reduces complexity while preserving the information needed for effective anomaly detection.
2Productivity
If the correlation matrix approach is used to reduce data complexity, then anomaly detection becomes feasible on high-dimensional data, but the computational process requires generating and comparing multiple correlation matrices
Solution Approach 1:
The patent performs preliminary computation of correlation matrices for different time periods before actual anomaly detection. By pre-computing and storing these correlation matrices, the system prepares the data in advance in a format that is ready for efficient anomaly detection, reducing the computational burden during the actual detection process.
Solution Approach 2:
The patent creates copies of correlation matrices for different time periods (first correlation matrix for first time period, second correlation matrix for second time period). These copies allow the system to compare correlation structures across time without reprocessing the original high-dimensional data, enabling efficient anomaly detection through comparison of the pre-computed matrix representations.
Data Source
AI summary
A method is disclosed. The method comprises generating by a processing network computer, a first attribute correlation matrix comprising correlations between attributes of a first interaction dataset, wherein the first interaction dataset comprises interaction data of a plurality of interactions conducted over a first time period. The processing network computer may generate a second attribute correlation matrix, similar to the first attribute correlation matrix, comprising interaction data conducted over a second time period. The method then comprises identifying sets of attributes from the first attribute correlation matrix and the second attribute correlation matrix. After identifying sets of attributes, the processing network computer may compute residuals between the first attribute correlation matrix and the second attribute correlation matrix. The processing network computer may then determine a number of interaction anomalies in the first interaction dataset using the residuals.