Data Quality Rule Recommendations for Duplicate Dataset Control

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Large organizations face challenges with duplicate datasets leading to increased infrastructure costs and skewed data analysis due to inefficient database management, resulting in inaccurate conclusions.

Innovation Solution

A computing system that analyzes datasets for data quality characteristics, identifies patterns, and generates data quality rule recommendations, allowing for the implementation of rules based on user input to improve dataset management.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If duplicate datasets are stored to various databases and servers, then data availability for multiple teams is improved, but infrastructure costs increase

Engineering Contradiction:

Improvedata availabilityVSAvoidinfrastructure costs

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The patent implements a centralized data registry that consolidates metadata about datasets across multiple storage locations into a single accessible system. This allows multiple teams to access and share the same underlying datasets through the registry without requiring physical duplicates, thereby reducing infrastructure costs while maintaining data availability.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The data quality rules engine acts as an intermediary layer between data storage locations and data consumers. It evaluates data quality characteristics and enforces rules that prevent duplicate or low-quality data from being stored, thereby reducing the need for redundant storage infrastructure while ensuring data availability through quality control.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If duplicate datasets are stored, then data access for multiple projects is improved, but data analysis accuracy deteriorates

Engineering Contradiction:

Improvedata accessVSAvoiddata analysis accuracy

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The system implements feedback loops where data quality characteristics are continuously evaluated against defined rules. When duplicates or low-quality data are detected, the system provides feedback to prevent their storage or inclusion in analysis, thereby maintaining data analysis accuracy while allowing broad data access through the centralized registry.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent applies data quality rules in advance before data is stored or used in analysis. By evaluating data quality characteristics and enforcing rules preemptively, the system prevents duplicate or inaccurate data from entering the dataset, ensuring analysis accuracy is maintained while data remains accessible through the registry.

Inventive Principle:

Principle #10Preliminary action

3Reliability

If data quality analysis is performed on datasets, then data quality characteristics are improved, but processing time increases

Engineering Contradiction:

Improvedata qualityVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs partial data quality analysis by evaluating only the most critical data quality characteristics against prioritized rules rather than conducting exhaustive analysis on all possible attributes. This selective approach maintains data quality improvement while significantly reducing processing time compared to comprehensive analysis.

Inventive Principle:

Principle #16Partial or excessive action

4Reliability

If data quality rules are implemented, then data uniqueness is improved, but system complexity increases

Engineering Contradiction:

Improvedata uniquenessVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent segments the data quality management system into distinct modular components: a data registry for metadata storage, a rules engine for evaluation, and characteristic analyzers for specific data attributes. This segmentation allows data uniqueness to be enforced through configurable rules while keeping system complexity manageable through modular, independently maintainable components.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12579116B2Database and data structure management systems

Publication Date: 2026.03.17 TRUIST BANK

AI summary

Systems and methods access, from one or more data storage locations, a dataset; perform data analysis on the dataset to detect one or more data quality characteristics each corresponding to at least one data quality dimension including timeliness, uniqueness, accuracy, completeness, validity, or consistency; evaluate the one or more data quality characteristics present in the dataset to identify one or more common patterns; and generate one or more data quality rule recommendations based on the identified one or more common patterns.