Data Quality Rule Generation for Centralized Duplicate Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large organizations face challenges with duplicate datasets leading to increased infrastructure costs and skewed data analysis, necessitating improved database management systems to enhance data quality and reduce storage expenses.
Innovation Solution
A computing system that analyzes datasets for data quality characteristics, identifies patterns, and generates data quality rule recommendations, allowing users to implement rules based on input, applies these rules to new data, and determines whether to include or discard data based on confidence scores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If duplicate datasets are stored to various databases and servers, then data availability and access flexibility improve, but infrastructure costs increase and data accuracy deteriorates
Solution Approach 1:
The patent merges duplicate datasets into a single centralized dataset, eliminating redundant storage across multiple databases and servers. This consolidation maintains data accessibility while reducing infrastructure requirements and costs.
Solution Approach 2:
The centralized dataset serves as a universal data source for multiple teams and projects, replacing the need for separate duplicate datasets. This single dataset provides universal access while eliminating the infrastructure overhead of multiple storage locations.
2Adaptability or versatility
If duplicate datasets are stored to various databases and servers, then data availability improves, but data analysis accuracy deteriorates
Solution Approach 1:
By merging duplicate datasets into a single centralized dataset, the patent eliminates data inconsistency across multiple sources. This ensures that all teams analyze the same unified data, improving measurement precision and analysis accuracy while maintaining broad data availability.
3Reliability
If manual data quality rule creation is performed, then data quality control improves, but time consumption and operational complexity increase
Solution Approach 1:
The system performs self-service by automatically analyzing datasets and generating data quality rules without manual intervention. The automated analysis identifies data quality issues and creates appropriate rules, eliminating the time-consuming manual rule creation process while maintaining reliable data quality control.
Solution Approach 2:
The system performs preliminary analysis of datasets to proactively identify potential quality issues before they affect data usage. By pre-generating data quality rules based on this analysis, the system prevents future data quality problems without requiring time-consuming manual rule creation.
4Productivity
If automated data quality rule generation is implemented, then productivity improves, but system complexity increases
Solution Approach 1:
The patent replaces the mechanical process of manual data quality rule creation with an automated computational system. This substitution uses algorithms and data analysis techniques to generate rules automatically, significantly improving productivity while the modular system architecture keeps complexity manageable.
Data Source
AI summary
Systems and methods access, from one or more data storage locations, a dataset; perform data analysis on the dataset to detect one or more data quality characteristics each corresponding to at least one data quality dimension including timeliness, uniqueness, accuracy, completeness, validity, or consistency; evaluate the one or more data quality characteristics present in the dataset to identify one or more common patterns; and generate one or more data quality rule recommendations based on the identified one or more common patterns.