Data Quality Delta Analysis for Database Resource Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Controlling data quality in large databases is resource-consuming and inefficient, as existing methods require processing all data for quality analysis, which is time-consuming and I/O intensive, especially when only a small portion of data has modifications.
Innovation Solution
Implementing a method for data quality delta analysis that focuses on tracking and analyzing only modified records, using delta rules to determine a data quality score when the number of modified records exceeds a predefined threshold, thereby optimizing resource usage and reducing unnecessary processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full data quality analysis is performed on the entire dataset, then measurement precision of data quality is improved, but use of energy and processing resources deteriorates
Solution Approach 1:
The patent divides the dataset into two segments: modified records (delta data) and unmodified records. Only the modified records segment is subjected to quality analysis, while unmodified records are excluded from processing. This segmentation allows the system to maintain measurement precision for quality assessment while significantly reducing processing resource consumption by analyzing only the relevant portion of the dataset.
Solution Approach 2:
The patent extracts and isolates the modified records from the complete dataset using delta data mechanisms. By taking out only the changed portions and applying quality rules specifically to these extracted records, the system achieves accurate quality measurement without the overhead of processing the entire dataset, thus resolving the contradiction between precision and resource usage.
2Reliability
If full data quality analysis is performed on the entire dataset, then reliability of quality control is improved, but productivity deteriorates
Solution Approach 1:
The patent segments the quality analysis process to focus exclusively on modified records. This segmentation maintains reliability by ensuring that all changes are captured and analyzed according to quality rules, while simultaneously improving productivity by eliminating unnecessary processing of unchanged data, thus achieving both reliable quality control and faster processing speeds.
Solution Approach 2:
The patent applies partial action by performing quality analysis only on the necessary portion of the data (modified records) rather than the complete dataset. This partial analysis approach maintains sufficient reliability for quality control purposes while dramatically improving productivity by reducing the scope of processing to only what is essential.
3Measurement precision
If data quality rules are applied to all records, then measurement precision of quality changes is improved, but loss of time deteriorates
Solution Approach 1:
The patent implements preliminary action by first identifying and marking modified records before applying quality rules. This preliminary step of delta detection allows the system to precisely target only the records that have changed, ensuring measurement precision for quality changes while minimizing the time required for analysis by excluding unmodified records from the rule application process.
Data Source
AI summary
The present disclosure relates to a method for data quality delta analysis on a dataset. The method provides a set of data quality rules for the dataset. At least one delta rule of a set of data quality rules is defined as relevant for delta analysis of at least part of the dataset, the delta rule being a delta analysis quality rule. Data changes on the dataset are tracked. In response to determining that a number of modified records of the at least part of the dataset is higher than a predefined insert modification threshold, a data quality score may be determined for said modified records using the delta rule.