Data Quality Rule Ranking for Network-Constrained Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large scale data processing introduces the potential for low quality data, and the execution of data quality rules is burdensome on network capabilities, with inefficient selection of rules straining the network and lacking standardization for high-quality data.
Innovation Solution
A system and method for determining data quality during data processing that uses machine learning to rank data quality rules based on metrics such as execution validity, time, and processing power, generating rankings to optimize data quality rule execution on networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data quality rules are executed on large scale data processing, then data quality is improved, but network capabilities are burdened
Solution Approach 1:
The system performs preliminary actions by pre-calculating and storing data quality rule rankings before actual data processing occurs. The ranking system evaluates multiple data quality rules across different metrics (execution validity, time, processing power) in advance, creating a prioritized list that can be quickly applied during data processing without requiring real-time network computation, thus improving data quality while minimizing network burden
Solution Approach 2:
The patent replaces the mechanical network-based data quality rule execution system with an intelligent ranking system that uses machine learning models. Instead of relying on network capabilities to evaluate and execute rules in real-time, the system substitutes this with pre-computed rankings stored locally, eliminating the need for continuous network communication during data quality assessment and reducing network strain
2Reliability
If multiple data quality rules are executed to ensure high-quality data, then data quality is improved, but processing time increases
Solution Approach 1:
The system performs preliminary evaluation of multiple data quality rules across various metrics (execution validity, time, processing power) and stores the results as pre-computed rankings. During actual data processing, instead of evaluating multiple rules sequentially which would consume time, the system simply applies the pre-determined top-ranked rules, significantly reducing processing time while maintaining data quality standards
Solution Approach 2:
The patent changes the parameter of rule selection from dynamic real-time evaluation to static pre-computed rankings. By transforming the data quality rule selection process into a parameter-based ranking system that considers execution validity, time, and processing power, the system enables quick rule selection during data processing without requiring time-consuming real-time evaluations of multiple rules
Data Source
AI summary
Systems, computer program products, and methods for determining data quality during data processing are provided. The method includes receiving data quality rule information relating to a plurality of data quality rules executed on a network. The data quality rule information includes at least one data quality metric for each data quality rule. The method also includes comparing one or more of the at least one data quality metric for each of the plurality of data quality rules. The method further includes determining a data quality rule ranking for each of the plurality of data quality rules. The method still further includes receiving one or more processes to be executed on the network. The method also includes determining one or more executed data quality rules to be executed on data associated with the one or more processes based on the data quality rule ranking.