Plugin-Based Data Quality Rules for Processing Flows
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale data processing is challenging to monitor for data quality, requiring individual coding of applications and making it difficult to ensure data accuracy, completeness, consistency, timeliness, and uniqueness, especially with diverse data sources and complex processing flows.
Innovation Solution
A system and method for determining and maintaining data quality through automated data quality rules and interactive automated code generation, utilizing a flow designer interface with plugin portions and customizable data quality rules, enabling visual development without individual coding, and monitoring data feeds and quality in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If automated data quality rules are implemented, then data quality monitoring capability is improved, but system complexity increases
Solution Approach 1:
The system automatically generates data quality rules by analyzing the data flow execution plan and plugin metadata without requiring manual intervention. The processor autonomously determines validation requirements, data quality metrics, and monitoring rules based on the configured data processing flow, enabling self-service data quality assurance.
Solution Approach 2:
Data quality rules are determined in advance during the configuration phase, before actual data processing occurs. The system pre-analyzes the data flow execution plan and generates appropriate validation rules and quality metrics that will be applied during execution, eliminating the need for runtime rule creation.
2Adaptability or versatility
If custom data quality rules are allowed, then data quality customization is improved, but ease of operation deteriorates
Solution Approach 1:
The system provides dynamic data quality rule configuration where users can add, modify, or remove custom data quality rules based on specific needs. The interface adapts to user input and allows flexible customization of data quality metrics while maintaining an intuitive workflow that guides users through the configuration process.
3Measurement precision
If real-time monitoring is implemented, then data quality detection is improved, but use of energy increases
Solution Approach 1:
The system applies data quality rules selectively based on the data flow execution plan and identified validation requirements. Rather than monitoring all data uniformly, the system focuses computational resources on critical data points and validation rules that are most important for maintaining data quality, reducing unnecessary energy consumption.
Data Source
AI summary
Systems, computer program products, and methods for determining and maintaining data quality in data processing is provided. The method includes determining one or more plugins used in a flow execution. The flow execution performs at least one transformation on one or more data sets. The method also includes determining one or more automatic data quality rules based on at least one of the one or more plugins using in the flow execution. The method further includes determining one or more custom data quality rules based on a custom data quality rule input received from an end-point device. The custom data quality rule includes one or more data statistics to be monitored during the flow execution. The method still further includes causing execution of each of the automatic data quality rules and the one or more custom data quality rules on data in the flow execution.