Data Quality Evaluation Nodes for Automated Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data processing systems lack the ability to automatically detect and address data anomalies and quality issues, leading to inaccurate or incomplete output generation, which is not immediately apparent to operators.
Innovation Solution
A data quality evaluation system that includes data quality evaluators to automatically detect errors, anomalies, and other issues in data received and processed by the system, generating data quality results and scorecards to facilitate timely investigation and correction of underlying problems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data processing systems are used without automated quality evaluation, then device complexity is reduced, but data quality and reliability deteriorate due to undetected anomalies
Solution Approach 1:
The patent implements preliminary data quality evaluation by inserting evaluation nodes before data processing operations. These nodes proactively detect anomalies, schema violations, and quality issues in advance, preventing defective data from propagating through the system and compromising output reliability.
Solution Approach 2:
The patent introduces intermediary data quality evaluation components that act as mediators between data sources and processing systems. These evaluators analyze data characteristics, generate quality scores, and provide feedback without requiring fundamental changes to the core processing architecture, thus improving reliability with minimal complexity increase.
2Measurement precision
If automated data quality evaluation is implemented, then detection precision of data issues is improved, but loss of time for data processing increases due to additional evaluation steps
Solution Approach 1:
The patent applies partial evaluation by focusing quality assessment on critical data fields and high-risk processing stages rather than uniformly evaluating all data. This selective approach maintains high detection precision for important anomalies while minimizing time overhead by skipping less critical evaluation steps.
Solution Approach 2:
The system performs preliminary quick checks using lightweight evaluation rules that rapidly identify obvious quality issues. Only data failing these preliminary checks proceeds to more time-consuming detailed analysis, thereby maintaining high detection precision while reducing overall processing time through early filtering.
3Manufacturing precision
If comprehensive data quality evaluation is performed, then manufacturing precision of data output is improved, but productivity decreases due to extended processing time
Solution Approach 1:
The patent implements differentiated evaluation strategies where critical data streams receive comprehensive quality assessment to ensure high output accuracy, while non-critical streams receive streamlined evaluation. This partial application of full evaluation maintains manufacturing precision for important outputs while preserving overall productivity through selective assessment.
Solution Approach 2:
The evaluation system is segmented into multiple independent evaluation nodes that can operate in parallel on different data streams or processing stages. This segmentation allows comprehensive quality checks to be performed on critical paths without creating sequential bottlenecks, thereby maintaining output accuracy while improving throughput through concurrent processing.
Data Source
AI summary
A data quality evaluation system can automatically detect one or more types of data anomalies or other data quality issues associated with a data processing system that may impact the quality of output generated by the data processing system. For example, the data quality evaluation system can detect data errors associated with data, detect when data is not received by the data processing system in compliance with defined schedules, detect when elements of the data processing system may be mishandling data, and/or detect when patterns of data does not correspond with historical patterns or validation data. By automatically detecting such data quality issues, technical issues or problems causing the data quality issues can be investigated and corrected.