Knowledge Graph Anomaly Detection for Unstructured Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning algorithms struggle to effectively extract patterns and detect anomalies from unstructured data sources like real-time tweets, making it difficult to identify abnormal activities in multi-INT fusion scenarios.
Innovation Solution
A method and system for pattern discovery and real-time anomaly detection using a knowledge graph (KG) are developed, which construct a local KG from unstructured data, apply statistical relational learning to predict hidden relations, and identify normalcy patterns, then compare incoming data against these patterns using semantic distance to detect abnormalities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional machine learning algorithms are used to extract patterns from unstructured data, then the system can process structured data effectively, but it fails to extract patterns from unstructured data like real-time tweets
Solution Approach 1:
The patent introduces a knowledge graph as an intermediary structure between unstructured data and pattern extraction algorithms. The knowledge graph transforms unstructured text data into structured semantic representations with entities, relations, and SPO triples, enabling reliable pattern extraction from previously unprocessable data sources while maintaining accuracy through semantic validation.
Solution Approach 2:
The system changes the representation parameters of unstructured data by transforming text into knowledge graph structures with specific properties (entities, relations, SPO triples). This parameter transformation allows conventional pattern extraction algorithms to effectively process unstructured data by operating on the structured knowledge graph representation instead of raw text.
2Productivity
If conventional machine learning algorithms are applied to unstructured data, then processing speed may be maintained, but the ability to identify abnormal activities is significantly reduced
Solution Approach 1:
The system performs preliminary action by constructing the knowledge graph and discovering normalcy patterns in advance before real-time anomaly detection. Normalcy patterns representing typical SPO triples and relations are pre-established from historical data, enabling the system to quickly compare incoming data against these predefined patterns for rapid and accurate anomaly identification.
Solution Approach 2:
The system creates a semantic copy of normal data patterns in the form of a knowledge graph with typical SPO triples and relations. This copied representation of normalcy serves as a reference model against which real-time data can be compared, enabling accurate anomaly detection by identifying deviations from the copied normal patterns.
3Adaptability or versatility
If a knowledge graph approach is used to extract patterns from unstructured data, then pattern extraction capability is improved, but system complexity increases
Solution Approach 1:
The patent segments the complex knowledge graph system into distinct functional modules: data ingestion module, knowledge graph construction module, pattern discovery module, and anomaly detection module. Each module handles a specific aspect of processing, reducing overall system complexity by breaking down the monolithic architecture into manageable, independent components with clear interfaces.
Solution Approach 2:
The knowledge graph itself serves as an intermediary layer that simplifies the interface between unstructured data and analysis algorithms. By introducing this semantic representation layer, the system manages complexity by providing a standardized intermediate format that bridges the gap between diverse unstructured inputs and pattern extraction requirements.
4Loss of information
If statistical relational learning is applied to predict hidden relations in the knowledge graph, then pattern completeness is improved, but computational time increases
Solution Approach 1:
The system applies partial action by using statistical relational learning selectively to predict only the most significant hidden relations in the knowledge graph rather than attempting to discover all possible relations. This selective approach recovers important hidden information while limiting computational time by focusing resources on high-priority relation predictions.
Solution Approach 2:
The system changes computational parameters by adjusting the confidence thresholds and sampling strategies in statistical relational learning. By modifying these parameters, the system achieves a balance between discovering hidden relations and controlling computational time, allowing it to predict the most probable hidden relations without exhaustive computation.
Data Source
AI summary
A method for pattern discovery and real-time anomaly detection based on knowledge graph, comprising: based on a dataset including messages collected within a certain period, constructing a local knowledge graph (KG); applying a statistical relational learning (SRL) model to predict hidden relations between entities to obtain an updated local KG; from all SPO triples of the updated local KG, discovering a normalcy pattern that includes frequent entities, frequent relations, and frequent SPO triples; and in response to receiving streaming data from a message bus, extracting a plurality of entities, a plurality of relations, and a plurality of SPO triples, from the streaming data for comparison with the normalcy pattern using semantic distance, thereby determining whether there is an abnormal entity, relation, or SPO triple in the streaming data.