Data Quality Rule Selection and Ordering for Faster Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data processing systems apply all data quality (DQ) rules indiscriminately on large datasets, leading to inefficient use of computer resources and prolonged processing times without ensuring data quality improvement, as rules with zero detection rates consume resources without adding value.
Innovation Solution
An automated framework selects and orders a subset of DQ rules based on their likelihood to detect data deficiencies, minimizing resource usage and processing time by prioritizing rules that are most effective in identifying data issues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data quality rules are applied indiscriminately on large datasets, then comprehensive data quality checking is achieved, but processing time is prolonged and computer resources are inefficiently used
Solution Approach 1:
The patent segments the complete set of data quality rules into multiple subsets based on their detection rates and characteristics. Instead of applying all rules uniformly to the entire dataset, the system divides rules into groups (e.g., high-priority rules with above-threshold detection rates versus lower-priority rules) and applies them in a staged manner. This segmentation allows comprehensive quality checking while reducing overall processing time by avoiding unnecessary application of all rules to all data.
Solution Approach 2:
The patent implements partial action by applying only a subset of data quality rules that meet predetermined thresholds (e.g., detection rate thresholds) to the dataset. The system identifies and applies only those rules that are likely to detect actual data quality issues, rather than applying every available rule. This partial application of rules maintains effective quality checking while significantly reducing processing time and resource consumption.
2Reliability
If all data quality rules are applied indiscriminately on large datasets, then comprehensive data quality checking is achieved, but computer resource utilization becomes inefficient
Solution Approach 1:
The patent segments the data quality rule set into prioritized groups based on detection rate analysis. High-priority rules with detection rates above predetermined thresholds are separated from lower-priority rules. This segmentation enables the system to apply only the necessary subset of rules, optimizing computer resource utilization while maintaining comprehensive quality checking for critical data issues.
Solution Approach 2:
The patent changes the parameter of rule application by introducing detection rate thresholds as a filtering criterion. Rules are evaluated based on their detection rate performance, and only rules meeting the threshold parameter are applied to the dataset. This parameter-based filtering optimizes resource utilization by eliminating application of rules that are unlikely to detect quality issues.
3Adaptability or versatility
If data quality rules with zero detection rates are applied, then complete rule coverage is achieved, but resources are consumed without adding value
Solution Approach 1:
The patent extracts and removes data quality rules with zero or below-threshold detection rates from the application set. By analyzing detection rates and identifying rules that do not contribute to quality detection, the system extracts only the valuable subset of rules for application. This eliminates waste of computer resources on rules that provide no value while maintaining complete coverage of all detectable quality issues.
Solution Approach 2:
The patent discards rules with zero detection rates from the active rule set. The system evaluates each rule's performance, discards those that do not detect quality deficiencies, and recovers computational resources that would have been wasted on ineffective rule application. This discarding process maintains rule coverage completeness for all meaningful quality checks while eliminating resource consumption on useless rules.
4Productivity
If a subset of data quality rules is selected based on detection rates, then processing efficiency is improved, but rule coverage may be reduced
Solution Approach 1:
The patent implements partial action with a twist: it applies a subset of rules that are sufficient for effective quality checking. By selecting rules with detection rates above predetermined thresholds, the system achieves the necessary level of quality assurance without applying all possible rules. The threshold is calibrated to maintain completeness of quality checking while maximizing processing efficiency.
Solution Approach 2:
The patent uses parameter changes (detection rate thresholds) to balance efficiency and completeness. By adjusting the threshold parameter, the system can control the subset of rules applied. The threshold is set to ensure that all rules capable of detecting quality issues are included, while excluding only those with genuinely insufficient detection rates. This parameter-based selection maintains reliability while improving productivity.
Data Source
AI summary
Selecting and ordering the execution of data quality rules includes generating a snapshot of a table-formatted dataset. The snapshot comprises a reduced number of rows of the dataset such that each column variation of the dataset is included in the snapshot. A predetermined collection of data quality (DQ) rules is executed on the snapshot. One or more performance statistics is determined for each of the DQ rules. The performance statistics indicate a likelihood that a DQ rule determines a data quality deficiency. Based on the performance statistics, a subset of the DQ rules is generated. Each DQ rule of the subset is selected based on the likelihood that the DQ rule selected detects a quality deficiency. An ordered subset of selected DQ rules is generated by ordering the application of each of the subset of DQ rules selected. The ordering specifies a sequence for executing each selected DQ rule.