Cognitive Data Outlier Pre-Check via Lineage Backtracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In industrial manufacturing and production, identifying anomalies in massive amounts of production data is challenging due to obscured abnormal data and numerous intermediate links, making it difficult to detect anomalies in real-time production data, intermediate process data, and root data.
Innovation Solution
A method for cognitive data outlier pre-checking based on data lineage, using error inverse propagation to backtrack outliers through data links, select significant data nodes and links, and generate potential anomaly indications, continuously optimizing models with feedback data for improved anomaly detection efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional anomaly detection methods are used on massive production data, then comprehensive data coverage is achieved, but detection efficiency deteriorates due to obscured abnormal data and numerous intermediate links
Solution Approach 1:
Instead of forward-propagating anomalies from raw data through numerous intermediate processing steps to final outputs (which obscures anomalies), the patent inverts the approach by backward-propagating outlier information from final data nodes through the data lineage graph to source data nodes. This reverse traversal allows direct identification of anomaly sources without being obscured by intermediate processing steps.
Solution Approach 2:
The patent introduces a data lineage graph as an intermediary structure that models relationships between data nodes and processing steps. This graph serves as a mediator that enables efficient backward propagation of outlier information, allowing the system to trace anomalies through complex data pipelines without directly processing the massive raw data itself.
2Reliability
If all data nodes are checked for outliers, then comprehensive anomaly detection is achieved, but computational complexity increases due to massive data volumes
Solution Approach 1:
The patent extracts only the essential outlier information from data nodes and propagates this extracted signal backward through the data lineage graph. Instead of analyzing all raw data, the system extracts outlier metrics and propagates them, significantly reducing computational complexity while maintaining detection completeness.
Solution Approach 2:
The system implements feedback by using detected outliers to guide the backward propagation process. Outlier information from final nodes feeds back through the data lineage to identify source nodes, creating a focused detection mechanism that adapts to actual anomaly patterns rather than uniformly processing all data.
3Measurement precision
If backward propagation is performed through all data links, then complete anomaly tracing is achieved, but processing time increases due to numerous intermediate links
Solution Approach 1:
The patent segments the data processing pipeline into discrete data nodes and data links within a data lineage graph. This segmentation allows the system to perform backward propagation through structured, manageable units rather than treating the entire data pipeline as a monolithic structure, improving both precision and efficiency.
Solution Approach 2:
The system performs partial backward propagation by focusing computational resources only on paths where outliers are actually detected, rather than uniformly processing all data links. This selective approach reduces processing time while maintaining complete anomaly tracing for relevant paths.
Data Source
AI summary
Methods, computer program products, and/or systems are provided that perform the following operations: obtaining pre-check data associated with specified data nodes; calculating outliers for each specified data node, wherein the outliers are calculated based on a unit of the pre-check data associated with each specified data node; backtracking the calculated outliers for each specified data node through an associated generating data link; selecting one or more data nodes associated with a set of largest outliers; selecting one or more data links associated with the set of largest outliers; and generating potential anomaly indications based on the one or more data nodes selected and the one or more data links selected.