AI Data Warehouse Quality Protocol Generation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data quality management systems face challenges in manually capturing and adapting to the complexities of heterogenous data sources and system architectures, making it difficult to define automated data quality protocols that are flexible and effective for business intelligence functions.

Innovation Solution

An AI-based system that uses attribute classification and hashing algorithms to quantify relationships between source and target data, generating customized data quality metrics and reports, and enabling flexible protocol design for individual system architectures.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If manual data quality management methods are used, then flexibility in addressing unique system architectures is improved, but productivity and automation extent are worsened

Engineering Contradiction:

Improveflexibility to address unique system architecturesVSAvoidautomation of data quality protocols

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The system performs self-service by automatically discovering data sources, extracting attributes, generating quality metrics, and validating data without manual intervention. The automated data quality engine discovers data sources autonomously, extracts attributes using classifiers, generates appropriate quality metrics based on discovered data characteristics, and executes validation protocols, enabling the system to serve itself rather than requiring manual configuration for each unique architecture.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent implements universality through a unified automated data quality engine that can handle multiple data source types (relational databases, NoSQL databases, data lakes, flat files), various data formats (structured, semi-structured, unstructured), and diverse quality metrics (completeness, accuracy, consistency, timeliness) through a single multi-functional platform, eliminating the need for separate manual processes for each data type or system architecture.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Productivity

If automated data quality protocols are implemented, then productivity is improved, but adaptability to heterogenous data sources and system architectures is worsened

Engineering Contradiction:

Improveautomation of data quality protocolsVSAvoidhandling of heterogenous data sources

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The system applies dynamics by making the data quality protocol configuration adaptive and changeable based on discovered data characteristics. The automated engine dynamically adjusts quality metrics, validation rules, and protocols based on the actual data sources, attributes, and relationships discovered during automated exploration, allowing the system to adapt to heterogeneous data sources while maintaining full automation rather than requiring fixed pre-configured protocols.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent utilizes parameter changes by automatically modifying data quality metric parameters, validation thresholds, and protocol configurations based on the characteristics of discovered data sources. The system changes parameters such as data completeness thresholds, accuracy metrics, consistency rules, and timeliness requirements according to the specific data types and system architectures encountered, enabling automated protocols to effectively handle diverse heterogeneous data sources.

Inventive Principle:

Principle #35Parameter changes

3Measurement precision

If data quality metrics are customized for individual system architectures, then measurement precision is improved, but device complexity increases

Engineering Contradiction:

Improvedata quality metricsVSAvoidsystem configuration

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The system applies preliminary action by pre-configuring a comprehensive library of data quality metrics, validation rules, and protocols that can be automatically selected and applied based on discovered data characteristics. Rather than requiring complex custom configuration for each system, the automated engine performs preliminary setup of quality metrics for various data types and system architectures, then automatically selects and applies the appropriate pre-prepared metrics, reducing configuration complexity while maintaining measurement precision.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent uses an intermediary approach by introducing an automated data quality engine that acts as a mediator between raw data sources and quality measurement. This intermediary engine automatically discovers data characteristics, selects appropriate quality metrics, and applies validation protocols, eliminating the need for users to directly configure complex quality measurement systems for each data source. The intermediary handles the complexity internally while providing simple automated quality assessment to users.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11586599B1Smart data warehouse protocols

Publication Date: 2023.02.21 BANK OF AMERICA CORP

AI summary

Systems, methods and apparatus are provided for AI-based generation of data warehouse quality protocols. An attribute classifier may quantify relationships between source data and target data from an enterprise data warehouse. A data quality engine may apply these relationships to identify specific data quality concerns and generate customized data quality metrics. A user interface may enable a user to enter parameters for the classification protocols and corresponding rule-based generation of data quality metrics.