Cross-Column Data Error Detection Using Compatibility Indexes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for detecting data errors within a data set are often inaccurate and limited, as they rely on manually-defined rules or only consider values within the same input column, leading to challenges in identifying inconsistencies and incompatibilities that can corrupt downstream results.
Innovation Solution
An automated system analyzes an extensive corpus of existing data to generate a compatibility index, using generalization languages to identify patterns within target data, and leverages this index to detect errors by comparing patterns across columns, employing statistical measures like point-wise mutual information to determine compatibility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If conventional methods use regular expression patterns to detect inconsistent values, then detection can be performed locally based only on values in a given input column, but the detection accuracy deteriorates because such techniques make local decisions and are error-prone
Solution Approach 1:
The patent transitions from one-dimensional local column analysis to multi-dimensional analysis by incorporating data from multiple columns and tables. The system evaluates relationships across different dimensions (columns, tables, data types) to improve detection accuracy while maintaining operational simplicity through automated multi-dimensional context analysis.
2Measurement precision
If automated systems analyze extensive corpora of existing data to generate compatibility indexes, then detection accuracy improves through broader data context, but system complexity increases
Solution Approach 1:
The system performs preliminary actions by pre-computing compatibility indexes from extensive data corpora before actual error detection occurs. This advance preparation stores compatibility relationships in searchable structures, enabling accurate real-time detection without repeating complex analysis during operation, thus managing system complexity through upfront computation.
Solution Approach 2:
The compatibility index serves as an intermediary structure between the extensive data corpus and the error detection process. It mediates by pre-processing and organizing compatibility relationships into a compact, query-efficient format, reducing the complexity of direct corpus analysis while preserving detection accuracy.
3Device complexity
If manual rules are used for error detection, then system complexity remains low, but the ability to identify inconsistencies and incompatibilities deteriorates leading to corrupted downstream results
Solution Approach 1:
The patent replaces manual rule-based mechanical systems with automated machine learning models that learn compatibility patterns from data. This substitution transitions from static manual rules to dynamic automated analysis, significantly improving error detection capability while the system manages complexity through automated model training and inference processes.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, computer systems, computer-storage media, and graphical user interfaces are provided for facilitating data error detection, according to embodiments of the present invention. In one embodiment, a target data set having a plurality of values for which to identify incompatible data is obtained. A pattern for each of the plurality of values is generated using at least one generalization language. A pair of patterns that represent a pair of values is utilized to identify a compatibility indicator that corresponds with a pair of training patterns in a compatibility index that match the pair of patterns. The compatibility indicator indicates the pair of patterns are incompatible with one another based on a statistical analysis performed in association with a corpus of data external to the target data set. An indication that the values are incompatible with one another is provided.