Database Data Cleaning via Deterministic Fixing Rules

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current data cleaning methods in databases often fail to accurately and automatically correct errors in attribute values, as they either introduce new errors or require significant user intervention, making them inefficient and unreliable.

Innovation Solution

The implementation of fixing rules that detect errors in semantically related attribute values and apply deterministic corrections, ensuring consistency across multiple rules to provide dependable and automatic data repair without heuristic methods or user interaction.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Difficulty of detecting and measuring

If integrity constraints are used to detect errors, then error detection capability is improved, but the ability to automatically repair errors deteriorates

Engineering Contradiction:

Improveerror detection capabilityVSAvoidautomatic repair capability

Core Design Contradiction:

Difficulty of detecting and measuringVSExtent of automation

Solution Approach 1:

The patent segments the data cleaning process into distinct components: error detection using integrity constraints, error classification into types (swap errors, insertion errors, deletion errors), and targeted repair strategies for each type. This segmentation allows automatic repair by matching detected errors to predefined repair rules.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent changes the parameter of error repair from heuristic-based random selection to deterministic selection based on error type classification. By categorizing errors into specific types with known correction patterns, the system transforms an automated but inaccurate process into an accurate and fully automated process.

Inventive Principle:

Principle #35Parameter changes

2Extent of automation

If heuristic-based repair methods are used, then automatic repair is improved, but reliability of error correction deteriorates

Engineering Contradiction:

Improveautomatic repairVSAvoiderror correction accuracy

Core Design Contradiction:

Extent of automationVSReliability

Solution Approach 1:

The patent changes the repair approach from heuristic-based to rule-based deterministic repair. By classifying errors into specific types (swap, insertion, deletion) and applying corresponding repair rules, the system achieves both high automation and high reliability, eliminating the trade-off between these two parameters.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent implements a feedback mechanism where the results of error detection are used to classify error types, which then inform the selection of appropriate repair rules. This closed-loop feedback ensures that repairs are systematically determined rather than randomly selected, improving reliability.

Inventive Principle:

Principle #23Feedback

3Measurement precision

If user-guided repair is used, then accuracy of error correction is improved, but productivity and efficiency deteriorate

Engineering Contradiction:

Improveerror correction accuracyVSAvoiddata cleaning efficiency

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent enables the data cleaning system to serve itself by automatically classifying errors and applying appropriate repair rules without user intervention. The system uses integrity constraint violations to self-diagnose error types and self-correct errors, achieving both accuracy and high productivity.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces the mechanical interaction between users and the system (manual verification and correction) with an automated information processing system that classifies errors and applies repairs algorithmically, thereby eliminating the productivity-accuracy trade-off.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

4Reliability

If master data with correct information is used, then reliability of repair is improved, but device complexity and cost increase

Engineering Contradiction:

Improverepair accuracyVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent makes the existing integrity constraints serve multiple functions: not only detecting errors but also classifying error types and guiding repairs. This multi-functionality eliminates the need for separate master data structures, maintaining reliability while reducing system complexity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent introduces error type classification as an intermediary step between error detection and repair. This intermediary structure enables deterministic repair rule selection without requiring external master data, achieving high reliability with minimal additional complexity.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS10545932B2Methods and systems for data cleaning

Publication Date: 2020.01.28 HAMAD BIN KHALIFA UNIVERSITY

AI summary

A method for cleaning data stored in a database, the method comprising providing a set of fixing rules. Each fixing rule incorporates a set of attribute values that capture an error in a plurality of semantically related attribute values, and a deterministic correction which is operable to replace one of the set of attribute values with a correct attribute value to correct the error. The method further comprises comparing at least two of the fixing rules with one another to check that the error correction carried out by one fixing rule is consistent with the error correction carried out by another fixing rule.