Metadata-Driven Data Quality Engine for Faster Rule Assessment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data quality assessment processes are costly, time-consuming, and unsustainable, requiring extensive resources and formal procedures that hinder timely and efficient data fitness evaluation.
Innovation Solution
A metadata-driven data quality framework and engine that automatically generates and executes data quality rules based on metadata, allowing for efficient and dynamic assessment of data quality, reducing manual intervention and enhancing data governance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a formal process of requirements gathering, design, development, and testing is used for data quality assessment, then measurement precision is improved, but loss of time increases
Solution Approach 1:
The patent applies preliminary action by pre-defining data quality rules, metrics, and assessment criteria before actual data assessment occurs. The system establishes a comprehensive data quality framework in advance, including predefined rules for completeness, accuracy, consistency, and timeliness. When assessment is needed, the pre-configured system can immediately execute without requiring time-consuming setup, thus maintaining high measurement precision while reducing assessment time.
2Reliability
If a formal data quality assessment process is implemented, then reliability is improved, but productivity is worsened
Solution Approach 1:
The patent implements self-service by enabling the data quality assessment system to automatically execute predefined rules, collect results, and generate reports without requiring manual intervention for each assessment. The system autonomously manages the entire assessment lifecycle including rule execution, data collection, analysis, and reporting. This automation maintains reliable quality assurance through consistent rule application while dramatically improving productivity by eliminating manual processing bottlenecks.
3Measurement precision
If extensive resources are allocated for data quality assessment, then measurement precision is improved, but loss of substance increases
Solution Approach 1:
The patent applies parameter changes by enabling dynamic adjustment of assessment parameters such as sampling rates, rule priority levels, and metric thresholds. The system can modify assessment intensity based on data criticality, historical quality trends, and resource availability. For high-priority data, comprehensive assessment with multiple metrics maintains high measurement precision. For lower-priority data, the system reduces assessment intensity, thereby maintaining adequate quality evaluation while significantly reducing resource consumption in terms of computing power, storage, and processing time.
Data Source
AI summary
An embodiment of the present invention is directed to a Metadata-Driven Data Quality Framework and Engine that enables dynamic generation of code for assessing data quality based on qualified metadata content. The Data Quality Framework may be directed to an enterprise scaled application that embodies Data Quality disciplines for good/optimal Data Governance. An embodiment of the present invention may be integrated into a Metadata Management process of an overall Data Governance Program so the data user's meaning and understanding may become part of the automated data quality process.