Data Quality Remediation in Consolidated Data Lakes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data lakes face challenges in maintaining data quality due to inconsistent and unreliable data sources, leading to inaccurate conclusions when poor-quality data is processed.
Innovation Solution
A computer-implemented method that assesses dataset flow quality and health quality by determining age and comparing it to thresholds, initiating corrective actions to ensure data quality meets predetermined standards, and providing messages to user interfaces for datasets beyond these thresholds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is collected from multiple diverse sources in a data lake, then data volume and versatility are improved, but data quality and consistency deteriorate
Solution Approach 1:
The patent segments data quality assessment into multiple independent dimensions: flow quality (timeliness, freshness) and health quality (completeness, consistency, accuracy, validity). Each dimension is evaluated separately against predefined thresholds, allowing targeted remediation of specific quality issues without discarding entire datasets. This segmentation enables the system to handle diverse data sources effectively while maintaining high quality standards.
Solution Approach 2:
The patent implements a feedback mechanism where quality metrics are continuously monitored, compared against thresholds, and used to trigger automated remediation actions. When datasets fail quality thresholds, the system automatically sends notifications to data owners and requests corrected datasets, creating a closed-loop feedback system that ensures continuous quality improvement while maintaining data versatility.
2Reliability
If data age threshold is applied to ensure freshness, then data flow quality is improved, but data processing time increases
Solution Approach 1:
The patent performs preliminary assessment of data flow quality by calculating dataset age and comparing it against predefined thresholds before downstream processing begins. This preliminary action filters out stale data early in the pipeline, preventing time-consuming processing of irrelevant data and ensuring only fresh, high-quality data proceeds to analysis and reporting stages.
Solution Approach 2:
The system dynamically adjusts data processing parameters based on quality metrics. When datasets meet flow quality thresholds, the system modifies processing parameters to optimize for speed and efficiency. This parameter adaptation allows the system to maintain fast processing for high-quality data while ensuring quality standards are met, balancing processing time with data freshness requirements.
3Reliability
If comprehensive quality checks are performed on all data elements, then data health quality is improved, but system complexity increases
Solution Approach 1:
The patent divides comprehensive quality checks into four distinct health quality dimensions: completeness, consistency, accuracy, and validity. Each dimension is evaluated using specific, targeted checks rather than a single complex validation process. This segmentation simplifies the overall system by making quality checks modular, manageable, and easier to implement for each specific data type and source.
Solution Approach 2:
The system adjusts the scope and stringency of quality checks based on data type, source reliability, and business context. By dynamically modifying quality validation parameters, the system maintains high data health quality through comprehensive checks when needed while reducing complexity for less critical datasets, achieving a balance between thoroughness and system simplicity.
Data Source
AI summary
A computer remediates data quality issues in a consolidated data record includes elements from several datasets, comprising. The computer determines, based at least in part on dataset feed information, dataset flow quality value for first and second datasets for a consolidated data record resulting from combining a plurality of datasets. The datasets include associated metadata. The computer, responsive to flow quality value determinations, compares dataset flow quality values with a predetermined dataset flow quality threshold and conducts a correction action for datasets with dataset flow quality value beyond the threshold. The computer determines, based at least in part on said dataset element health information, dataset health quality values for the datasets. The computer, compares the health quality values with a predetermined dataset health quality threshold and conducts a dataset health quality correction action for datasets having a dataset health quality value beyond the dataset health quality threshold.