Data Quality Analysis via Fingerprint Matching

Solution Overview

Problem

Current data quality analysis methods, such as IBM's CP4D WKC, face performance issues and resource wastage when analyzing large data volumes, often resulting in CPU bottlenecks and requiring repetitive tuning, which disrupts production systems.

Innovation Solution

The approach generates and matches fingerprints of data sources using configuration sets to quickly identify errors and validate content, allowing for efficient data quality analysis without exhaustive methods that create system bottlenecks, and automatically updates the repository with new fingerprints for improved performance.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If exhaustive data quality analysis methods are used, then measurement precision is improved, but productivity deteriorates due to system bottlenecks

Engineering Contradiction:

Improvedata quality analysis accuracyVSAvoiddata quality analysis speed

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The system performs preliminary actions by generating fingerprints of data sources and storing them in a repository before actual data quality analysis. When a new data source arrives, its fingerprint is compared against the repository to find matching configuration sets, allowing the system to skip exhaustive analysis for similar data sources and directly apply pre-determined quality rules, thus improving speed while maintaining accuracy

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates simplified copies (fingerprints) of data sources that capture essential characteristics without containing the full data. These fingerprints are stored in a repository and used for quick comparison. When a fingerprint match is found, the corresponding configuration set is copied and applied to the new data source, avoiding repeated exhaustive analysis while preserving measurement precision

Inventive Principle:

Principle #26Copying

2Measurement precision

If exhaustive data quality analysis methods are used, then data quality analysis accuracy is improved, but use of energy worsens due to CPU bottlenecks

Engineering Contradiction:

Improvedata quality analysis accuracyVSAvoidCPU resource consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary actions by generating fingerprints of data sources and storing them in a repository before actual data quality analysis. When a new data source arrives, its fingerprint is compared against the repository to find matching configuration sets, allowing the system to skip exhaustive analysis for similar data sources and directly apply pre-determined quality rules, thus improving speed while maintaining accuracy

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates simplified copies (fingerprints) of data sources that capture essential characteristics without containing the full data. These fingerprints are stored in a repository and used for quick comparison. When a fingerprint match is found, the corresponding configuration set is copied and applied to the new data source, avoiding repeated exhaustive analysis while preserving measurement precision

Inventive Principle:

Principle #26Copying

3Reliability

If repetitive tuning is performed to resolve performance issues, then reliability is improved, but loss of time worsens due to production system disruption

Engineering Contradiction:

Improvesystem performance stabilityVSAvoidproduction system downtime

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs self-service by automatically generating fingerprints, comparing them against the repository, and selecting appropriate configuration sets without requiring manual tuning. The repository is automatically updated with new fingerprints and their corresponding configuration sets, enabling the system to self-optimize and maintain reliability without disrupting production operations

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system implements feedback by continuously comparing data source fingerprints against the repository and using the results to automatically select or update configuration sets. This closed-loop approach allows the system to learn from past analyses and improve performance over time without requiring external intervention or production stoppage

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS20230385252A1Data quality analyze execution in data governance

Publication Date: 2023.11.30 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

An approach is provided that retrieves fingerprint configuration sets corresponding to a received data source and uses the configuration sets to generate fingerprints that correspond to the data source. These fingerprints are compared to a number of fingerprints that are stored in a repository. If a match is found, then the data quality configuration set is retrieved from the repository and used to perform a data quality analysis. On the other hand, if a match is not found, then one of the configuration sets is selected to perform the data quality analysis on the received data source and the repository is updated so that the selected fingerprint configuration set corresponds to the received data source.