Data Quality Processing Using Semantic Annotations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data quality systems are cumbersome for users due to the need to define and store multiple configurations for each scenario, particularly in cloud-based solutions where different applications and devices have varying data schemas, locales, and data quality rules, making traditional on-premise configurations ineffective.
Innovation Solution
A computer-implemented method and system that uses semantic annotations to automatically map input and output formats and data quality annotations to apply rules, eliminating the need for separate configuration definitions, allowing for flexible and customizable data quality functions in a cloud-based environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional on-premise configuration methods are used for cloud-based data quality solutions, then each configuration contains detailed input mapping, output selection, and data quality rules, but this approach becomes cumbersome for users to use due to the need to define and store a configuration for each data quality scenario
Solution Approach 1:
The system enables self-service by allowing data to carry its own metadata (semantic annotations and data quality annotations) that automatically configure the data quality processing. The data quality function process automatically maps semantic annotations to input/output fields and applies data quality rules without requiring users to manually define configurations for each scenario.
Solution Approach 2:
The patent extracts configuration information from separate configuration files and embeds it directly into the data records as annotations. This separation removes the burden of configuration management from users, as the configuration is now intrinsic to the data itself rather than a separate entity that must be defined and stored.
2Adaptability or versatility
If separate configurations are defined for each data quality scenario with input mapping, output selection, and data quality rules, then data quality functions can be applied, but the system becomes complex to manage especially in cloud-based solutions with multiple applications and devices
Solution Approach 1:
The patent creates a universal configuration mechanism where a single data structure (the annotated record) serves multiple purposes: it carries semantic information for mapping, quality rules for validation, and formatting information for output. This universal approach works across multiple applications, devices, and data quality scenarios without requiring separate configuration systems.
Solution Approach 2:
The patent merges previously separate configuration elements (input mapping configuration, output selection configuration, and data quality rules configuration) into a unified annotation structure that travels with the data. This consolidation eliminates the need to manage multiple separate configuration files or systems.
3Reliability
If manual configuration definition is required for each data quality scenario, then specific data quality rules can be applied, but this increases user error and complexity
Solution Approach 1:
The patent implements preliminary action by pre-annotating data with semantic and data quality information at the source before the data quality function is invoked. This advance preparation eliminates the need for time-consuming manual configuration at runtime and ensures accuracy is established before processing begins.
Data Source
AI summary
A computer-implemented method for performing a data quality function includes receiving, at a data quality function process, at least one record from a source, where the record includes data, semantic annotations and data quality annotations associated with the data in the record. The semantic annotations are attributes that define the data including input format and output format and the data quality annotations are attributes that define data quality rules to be applied to the data. The data quality function process automatically maps the semantic annotations to data quality input fields and to data quality output fields. The data quality function process applies the data quality rules to the data using the data quality annotations to perform a data quality function. The data quality function process performs the data quality function on the data and outputs the data to a destination in a format defined by the data quality output fields.