Data Quality Engine Rich Type Inference Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data quality checks in enterprises are difficult to automate, particularly in determining if business rules are satisfied, new values are permitted in categorical variables, syntax or format is correct, data has valid semantic interpretation, detecting data drift, and identifying noise in semantic attributes, due to limitations in schema attributes.
Innovation Solution
The implementation of a data quality engine that infers rich data types for each column based on format, column name heuristics, pattern matching, and consistency, applies data quality constraints, and identifies issues in production data using unit tests, with the ability to correct anomalies and notify data owners.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If common schema attributes are used for data quality checks, then the system is simple to operate, but automation capability is poor
Solution Approach 1:
The patent transforms basic schema attributes into enriched data types by adding semantic parameters and constraints. This enrichment enables automated inference of business rules and semantic constraints without increasing operational complexity, resolving the contradiction between simplicity and automation capability
Solution Approach 2:
The patent introduces an intermediary layer of rich data type inferences that sits between the simple schema attributes and the complex data quality validation logic. This intermediary automatically derives semantic constraints and business rules, enabling automation while keeping the user interface simple
2Extent of automation
If rich data type inferences are implemented, then automation capability improves, but device complexity increases
Solution Approach 1:
The patent segments the data quality validation system into modular components: schema attribute parsing, rich data type inference engine, constraint generation module, and validation execution module. This segmentation reduces overall system complexity by making each component independent and manageable while enabling sophisticated automation
3Productivity
If automated data quality checks are implemented, then productivity improves, but measurement precision of semantic constraints deteriorates
Solution Approach 1:
The patent performs preliminary action by inferring rich data types and generating semantic constraints before actual data validation occurs. This pre-computation of constraints enables both high-speed automated validation and high precision, as the constraints are derived once and reused efficiently during validation
Data Source
AI summary
Systems and methods for automated data quality semantic constraint identification using rich data type inferences are disclosed. In one embodiment, a method for automated data quality analysis may include: (1) receiving, by a data quality engine computer program, reference data from a data source, wherein the reference data comprises a plurality of columns; (2) inferring, by the data quality engine computer program, a rich data type for each of the plurality of columns, wherein the rich data type has a specific format, a content constraint, and/or a specific application; (3) applying, by the data quality engine computer program, a data quality constraint to each column based on the rich data type for the column; (4) updating, by the data quality engine computer program, the reference data with production data; and (5) identifying, by the data quality engine computer program, a data quality issue in the production data.