Automated Data Quality Rule Binding via Column-Variable Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data quality evaluation systems rely on manual processes, which are time-consuming and error-prone, and fail to automatically apply appropriate data quality rules due to lack of direct correlation between data variables and business context, leading to inefficiencies in data transformation and rule expression application.
Innovation Solution
A method and system that automatically determine appropriate data quality evaluations by building candidate binding combinations between dataset columns and rule expression variables, generating new bound rule expressions through data transformation, and storing them for evaluation, utilizing techniques like business classification, data classification, and name similarity to identify suitable rule expressions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual processes are used to apply data quality rules, then data quality validation can be performed, but the process is time-consuming and error-prone
Solution Approach 1:
The system performs self-service by automatically matching data columns to rule expression variables using name similarity algorithms and business context analysis, eliminating the need for manual configuration and reducing human error in rule application
Solution Approach 2:
The manual mechanical process of rule application is replaced with an automated computational system that uses algorithms to match columns to variables, transform data formats, and apply rules automatically based on business context
2Extent of automation
If conventional systems are used to apply data quality rules, then some validation can be performed, but appropriate rules cannot be automatically determined due to lack of direct correlation between variables and business context
Solution Approach 1:
Business context information acts as an intermediary that bridges the gap between data columns and rule variables. The system uses business context to infer meaningful correlations and guide the automatic matching process when direct name matches are not available
Solution Approach 2:
The system changes parameters by transforming data formats and applying data transformations to make data compatible with rule expressions. This includes converting data types, formatting values, and restructuring data to match expected patterns in rule variables
3Adaptability or versatility
If data transformation is required to apply rule expressions, then appropriate rules can be applied, but the process becomes more complex
Solution Approach 1:
The system performs preliminary actions by pre-defining data transformations and validation rules. When new data is encountered, the system can automatically apply pre-configured transformations based on the detected data type and format, reducing the complexity of ad-hoc transformation logic
Data Source
AI summary
Methods and systems for data quality evaluation are disclosed. A method includes: receiving, by a computing device, at least one data set and a list of rule expressions to bind; building, by the computing device, candidate binding combinations between columns of the at least one data set and variables of each rule expression in the list of rule expressions; building, by the computing device, a new bound rule expression candidate based on the candidate binding combinations; generating, by the computing device, a new bound rule expression based on the new bound rule expression candidate and a data transformation applied to at least one of the columns of the at least one data set; and storing, by the computing device, the new bound rule expression.