Anomaly Detection Using Semantic Models and Parallel Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data mining methods face challenges in efficiently detecting anomalies in large datasets, requiring significant computational resources and time, and often identify irrelevant anomalies that lack real-world significance.
Innovation Solution
A computer-implemented method and system that uses a semantic model to form multi-dimensional data instances from permutations and combinations of data entities, analyzing these instances to detect statistically significant deviations and presenting the results through a graphical user interface, allowing users to navigate and select anomalies relevant to real-world problems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing data mining methods are used to detect anomalies in large datasets, then anomaly detection capability is provided, but computational resources and time consumption increase significantly
Solution Approach 1:
The patent segments the large dataset into multiple partitions and processes them in parallel using multiple computing nodes. The anomaly detection system divides the data processing task into smaller sub-tasks that can be executed concurrently, reducing overall processing time while maintaining detection accuracy.
Solution Approach 2:
The patent introduces a semantic model dimension that organizes data entities into hierarchical structures with defined relationships. By adding this semantic dimension, the system can navigate and filter data more efficiently, focusing computational resources on relevant data portions rather than processing entire datasets.
2Reliability
If existing data mining methods are used to detect anomalies in large datasets, then anomaly detection capability is provided, but computational resources increase significantly
Solution Approach 1:
The system segments data processing across multiple computing nodes, distributing computational load. Each node processes a portion of the data independently, reducing the computational resources required at any single point and enabling parallel processing that improves overall efficiency.
Solution Approach 2:
The patent implements preliminary data filtering and preprocessing steps that reduce the volume of data requiring full anomaly detection processing. By pre-processing data to identify and eliminate obviously normal records, the system reduces computational resource consumption while maintaining detection accuracy for suspicious records.
3Reliability
If existing data mining methods are used for anomaly detection, then anomalies are identified, but irrelevant anomalies lacking real-world significance are detected
Solution Approach 1:
The patent introduces a semantic model as an intermediary layer between raw data and anomaly detection algorithms. This semantic model incorporates domain knowledge and defines meaningful relationships between data entities, allowing the system to evaluate anomalies in context and filter out those lacking real-world significance.
Solution Approach 2:
The system dynamically adjusts detection parameters and thresholds based on semantic relevance and contextual information from the semantic model. By changing detection parameters according to the semantic context of different data entities, the system can distinguish between statistically anomalous but irrelevant patterns and truly significant anomalies.
4Reliability
If complex data analysis is performed on large datasets, then comprehensive anomaly detection is achieved, but user navigation and interpretation become difficult
Solution Approach 1:
The semantic model serves as an intermediary that translates complex analytical results into interpretable formats for users. By organizing anomalies within the semantic framework of data entity relationships, the system presents results in a structured manner that is easier for users to navigate and understand.
Solution Approach 2:
The patent implements a hierarchical presentation structure where anomalies are nested within the context of their parent data entities and relationships. Users can navigate from high-level summaries down to detailed anomaly information in a structured, progressive manner, making complex results more manageable and interpretable.
Data Source
AI summary
According to an aspect of some embodiments of the present invention there is provided a computer implemented method for detecting at least one anomaly in a dataset, comprising: managing a dataset including a plurality of data entities each including at least one value; receiving a semantic model that defines associations between two or more data entities; forming a plurality of multi dimensional data instances, each multi dimensional data instance formed from at least one of a permutation and a combination of a set of data entities from the plurality of data entities according to the semantic model; analyzing the multi dimensional data instances to detect at least one anomalous value, the anomalous value representing a statistically significant deviation according to a deviation requirement, of one or more values from a set of values of the multi dimensional data instances; and providing the detected at least one anomalous value.