Dataflow Control Architecture for Integrated Quality Checks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large organizations face challenges in detecting data anomalies, preventing data loss, and ensuring data quality due to the complexity of their IT infrastructure, which leads to difficulties in maintaining data integrity and accuracy across various databases and systems.
Innovation Solution
A dataflow control architecture that incorporates lineage information into data itself, using control values to track data flow and transformations, allowing for real-time monitoring and validation of data integrity, timeliness, and accuracy through a lineage server that aggregates and analyzes control values across multiple nodes in the IT infrastructure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual reconciliation and validation methods are used across multiple databases and systems, then data integrity can be maintained, but the time and computing resources required increase significantly
Solution Approach 1:
The patent applies preliminary action by embedding control values and lineage information into data at the source systems before data movement occurs. This allows automated tracking and validation throughout the data lifecycle, eliminating the need for time-consuming manual reconciliation processes while maintaining data integrity across complex IT infrastructures.
Solution Approach 2:
The patent implements feedback mechanisms through control values that continuously monitor and validate data flow across systems. The lineage server aggregates control values and provides real-time feedback on data quality metrics, enabling automated detection and correction of integrity issues without manual intervention.
2Loss of information
If control values are embedded in all data elements across multiple databases, then data lineage tracking is improved, but the complexity of data storage and processing increases
Solution Approach 1:
The patent merges control values directly with data elements in a unified storage structure. Instead of maintaining separate metadata repositories, the control values are embedded within the same storage framework as the actual data, simplifying the overall system architecture while enabling comprehensive lineage tracking across distributed databases.
3Measurement precision
If real-time monitoring of data flow is implemented across the IT infrastructure, then data quality issues are detected faster, but the computing resources required increase
Solution Approach 1:
The patent implements self-service through automated control value aggregation and validation performed by the lineage server. The system monitors its own data flow using embedded control values, eliminating the need for external manual auditing while maintaining continuous real-time detection of data quality issues with optimized resource utilization.
Data Source
AI summary
A system for performing an integrated data quality control is disclosed. The system determines a dataflow path for one or more input data elements. The, the system performs a lineage control check, a timeliness control check, and a variation control check on the dataflow path. If the dataflow path integrated scoring of the three controls is sufficient, the system determines that data related to the dataflow path is fit for use. If the dataflow path fails any one of the three checks, the system determines that data related to the dataflow path is not fit for use.