Data Quality Analysis Using Upstream Profile Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data quality analysis methods fail to effectively identify the root cause of data quality issues in downstream data sets, often due to poor quality source or reference data, and errors in transform elements, leading to inefficiencies in data processing and management.
Innovation Solution
A computer-implemented method that utilizes data lineage analysis to identify upstream data sets and transform elements contributing to data quality issues by comparing profiles to historical averages and applying user-specified or automatically generated rules to detect deviations and errors, providing a user interface for rule management and notification of potential issues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data quality analysis is performed on downstream datasets, then data quality issues can be detected, but the root cause in upstream datasets remains difficult to identify
Solution Approach 1:
The patent segments the data processing system into distinct upstream datasets, transform elements, and downstream datasets. By analyzing each component separately through profile comparison and rule application, the system identifies which specific segment caused the data quality issue, making root cause detection possible without overwhelming complexity.
Solution Approach 2:
The patent introduces an intermediary analysis layer that compares profiles of upstream datasets against reference profiles and applies validation rules between upstream and downstream datasets. This intermediary mechanism translates downstream quality issues into identifiable upstream problems, bridging the detection gap.
2Reliability
If comprehensive data lineage analysis is performed to identify all upstream datasets, then root cause identification improves, but processing time and system complexity increase
Solution Approach 1:
The patent performs preliminary actions by pre-comparing upstream dataset profiles against reference profiles and pre-applying validation rules before actual data processing occurs. This advance preparation identifies potential issues early, reducing the time needed during main processing and enabling faster root cause identification when problems arise.
Solution Approach 2:
The patent applies local quality analysis by focusing profile comparisons and rule validations on specific upstream datasets and transform elements that are most likely to cause issues, rather than uniformly analyzing everything. This targeted approach maintains high reliability for root cause identification while minimizing unnecessary processing time.
3Ease of operation
If automated rule generation is used to reduce manual configuration, then ease of operation improves, but measurement precision of data quality rules may deteriorate
Solution Approach 1:
The patent implements self-service through automated profile comparison and rule generation, where the system automatically analyzes upstream dataset profiles, compares them against historical reference profiles, and generates validation rules without manual intervention. This automation maintains measurement precision by using objective statistical comparisons while dramatically improving ease of operation.
Solution Approach 2:
The patent incorporates feedback mechanisms where the system continuously monitors data quality, compares current profiles against reference profiles, and adjusts validation rules based on observed deviations. This feedback loop ensures that automated rule generation remains accurate by adapting to actual data patterns while maintaining ease of operation through continuous self-improvement.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A computer-implemented method for analyzing an update to a transform element is provided, the method including: identifying a time of a recent update to a particular transform element of one or more transform elements, the recent update being an update within a threshold amount of time; in response to determining that the particular transform element was recently updated, reviewing update logs associated with the particular transform element to identify one or more inconsistencies between the update logs and a timing of the recent update stored in a data repository, wherein the presence of inconsistencies can be indicative of an error or a potential error associated with the particular transform element; and if any errors or potential errors associated with the particular transform element are identified, outputting an alert to indicate an identified error or potential error associated with the particular transform element.