Historical Key-Field Exception Detection With Cause-Field Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing dataset management systems struggle to accurately and efficiently detect and handle data exceptions, which hinder analysis tasks, requiring manual intervention by administrators.
Innovation Solution
A method and apparatus for determining data exceptions by identifying a key field in a dataset, analyzing its historical state for exceptions, and automatically identifying cause fields using automation scripts and dimension decomposition, with adjustable thresholds and alert mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual detection and handling of data exceptions is performed by administrators, then data exceptions can be detected and handled, but the process is time-consuming and inefficient
Solution Approach 1:
The system performs self-diagnosis by automatically detecting data exceptions through automated scripts that analyze data sources, compare data changes against predefined rules, and identify root causes without requiring administrator intervention. The system serves itself by generating alerts and determining causality relationships automatically.
Solution Approach 2:
The system performs preliminary actions by pre-configuring data change rules, thresholds, and alert mechanisms before exceptions occur. Automated scripts are prepared in advance to monitor data sources, and when exceptions occur, the system immediately executes pre-programmed detection and analysis procedures, eliminating the need for manual intervention.
2Measurement precision
If manual detection of data exceptions is performed, then exceptions can be identified, but the accuracy and effectiveness are insufficient
Solution Approach 1:
The system segments the complex task of exception detection into distinct components: data collection from multiple sources, change detection against predefined rules, causality analysis through dimension decomposition, and alert generation. Each component is handled by specialized automated scripts, improving detection accuracy while reducing overall management complexity through systematic organization.
Solution Approach 2:
The system implements feedback mechanisms by continuously monitoring data changes, comparing them against predefined rules, and automatically generating alerts when exceptions are detected. The system provides feedback loops where detection results trigger automated responses, and the entire process is continuously refined based on detected patterns and root cause analysis.
Data Source
AI summary
There are provided a method, apparatus, device and medium for determining a data exception is provided. In the method, a key field in a dataset is determined, the dataset comprising a plurality of data sources, and respective data sources of the plurality of data sources respectively comprising at least one field. A historical state of the key field in a historical time period is obtained. In response to determining that the historical state indicates that a data change in the key field satisfies an exception condition, it is determined that the key field has a data exception, the data exception indicating that the key field has an exception in the historical time period. At least one cause field associated with the data exception is determined in the dataset, a data exception of the at least one cause field resulting in a data exception of the key field.