Heterogeneous Data Quality Scoring via Pattern Templates

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data quality assessment technologies are inadequate for handling heterogeneous data sources, as they fail to efficiently integrate and assess the quality of mixed data types such as structured, semi-structured, quasi-structured, and unstructured data, often requiring costly and cumbersome root-cause analysis after bad results are discovered.

Innovation Solution

A method and system that determine the type of incremental heterogeneous data using pattern templates, select appropriate data quality tests, and calculate a score based on user-defined parameters to assess the quality of the data, enabling efficient quality assessment across diverse data sources.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If existing data quality monitoring techniques are used on structured data from relational databases, then data quality can be assessed by normalizing the data, but these techniques fail to handle heterogeneous data sources including unstructured, semi-structured, and quasi-structured data

Engineering Contradiction:

Improvedata type compatibilityVSAvoidquality assessment accuracy

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system dynamically changes assessment parameters based on data type identification. Different data types (structured, semi-structured, unstructured) trigger different quality metrics and assessment methods, allowing the system to adapt to heterogeneous data sources while maintaining reliable quality assessment for each specific data type

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent segments the data quality assessment process into distinct stages: data type identification, pattern template matching, and type-specific quality metric application. This segmentation allows the system to handle each data type appropriately while providing comprehensive coverage across all heterogeneous data sources

Inventive Principle:

Principle #1Segmentation

2Loss of energy

If root-cause analysis is performed only when bad results are discovered, then analysis cost is reduced, but the process becomes extremely expensive, cumbersome, or impossible given the volume and speed of data pushed into data lakes

Engineering Contradiction:

Improveanalysis costVSAvoidquality assessment delay

Core Design Contradiction:

Loss of energyVSLoss of time

Solution Approach 1:

The system performs preliminary data quality assessment continuously as data enters the data lake, rather than waiting for bad results to be discovered. By proactively applying pattern templates and quality metrics to incoming data streams, the system identifies quality issues early, reducing both the cost and time of remediation while preventing bad data from propagating through the system

Inventive Principle:

Principle #10Preliminary action

3Quantity of substance

If data acquisition is performed from various sources including live feeds and click stream data, then data completeness is improved, but integration of different types of data sources becomes troublesome and extremely error prone

Engineering Contradiction:

Improvedata volumeVSAvoidintegration complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent implements a universal pattern template framework that can process multiple data types (structured, semi-structured, unstructured) through a single integrated system. The pattern templates serve as a multi-functional interface that adapts to different data sources including databases, live feeds, and click stream data, simplifying integration while maintaining comprehensive data acquisition

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS10565173B2Method and system for assessing quality of incremental heterogeneous data

Publication Date: 2020.02.18 WIPRO LTD

AI summary

The present disclosure relates to a method and system for assessing quality of incremental heterogeneous data by a data quality assessing system. The data quality assessing system determines an incremental heterogeneous data from at least one data source, obtains details associated with the incremental heterogeneous data from the at least one data source, identifies type of the incremental heterogeneous data based on the details and pattern templates, selects one or more data quality tests from a plurality of data quality tests for the incremental heterogeneous data based on the identified type of the incremental heterogeneous data and determines a score for the incremental heterogeneous data based on the one or more data quality tests and user defined parameters to assess quality of heterogeneous incremental data.