Plugin-Based Data Quality Monitoring in Processing Flows
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale data processing is challenging to monitor for data quality, requiring individual coding of processes and making it difficult for non-technical users to automate and maintain data integrity, especially with varying data sources and formats.
Innovation Solution
A system and method for determining and maintaining data quality through a flow designer interface with plugin portions, enabling interactive automated code generation and modification, including data quality rules and customizable data feeds, allowing users to create and monitor data processing flows without manual coding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If individual applications are coded for process-specific tasks, then customization and functionality are improved, but device complexity and difficulty of operation increase
Solution Approach 1:
The system segments data quality monitoring into separate, reusable plugins that can be independently configured and combined. Each plugin handles a specific aspect of data quality (completeness, accuracy, consistency, etc.), allowing customization without requiring complex custom coding for each aspect.
Solution Approach 2:
The platform provides a universal data quality monitoring system that can handle multiple data sources, formats, and quality attributes through a single integrated framework. The same core system supports various data types and quality checks through configurable plugins rather than requiring separate applications for each function.
2Adaptability or versatility
If manual coding of data processing flows is required, then customization flexibility is improved, but ease of operation and productivity deteriorate
Solution Approach 1:
The system enables users to configure data quality monitoring and processing flows through self-service configuration interfaces without requiring programming expertise. Users can define quality rules, select plugins, and configure parameters through user-friendly interfaces that automatically generate the underlying processing logic.
Solution Approach 2:
The system provides pre-built plugins and templates that perform common data quality checks and transformations in advance. These pre-configured components eliminate the need for users to manually code standard data quality monitoring functions, improving productivity while maintaining customization through selective configuration.
3Reliability
If data quality monitoring is performed on large-scale data processing, then data quality assurance is improved, but measurement precision and detection difficulty increase
Solution Approach 1:
The monitoring system is segmented into specialized plugins, each designed to detect specific types of data quality issues (completeness, accuracy, consistency, validity). This segmentation makes complex data quality monitoring manageable by breaking it down into discrete, well-defined detection functions that can be independently optimized.
Solution Approach 2:
The system implements continuous feedback mechanisms that monitor data quality metrics in real-time and provide actionable insights. Quality measurements are fed back to users and system administrators, enabling timely detection and resolution of data quality issues without requiring complex manual analysis of large datasets.
Data Source
AI summary
Systems, computer program products, and methods for determining and maintaining data quality in data processing is provided. The method includes determining one or more plugins used in a flow execution. The flow execution performs at least one transformation on one or more data sets. The method also includes determining one or more automatic data quality rules based on at least one of the one or more plugins using in the flow execution. The method further includes determining one or more custom data quality rules based on a custom data quality rule input received from an end-point device. The custom data quality rule includes one or more data statistics to be monitored during the flow execution. The method still further includes causing execution of each of the automatic data quality rules and the one or more custom data quality rules on data in the flow execution.