Detecting Duplicate Tables in Data Lakes via Knowledge Graphs

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In commercial data lakes, duplicate tables are difficult to detect due to differences in syntactical schema, leading to wasted storage, maintenance difficulties, and inconsistent query results, as well as introducing bias in machine learning models.

Innovation Solution

A computer-implemented method and system that converts relational tables to knowledge graphs, maps nodes to sources, and applies graph matching to detect duplicate tables by assessing the semantic similarity between them.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If manual inspection is used to detect duplicate tables, then human judgment can identify semantic similarity, but the process is neither practical nor accurate for large-scale data lakes

Engineering Contradiction:

Improvesemantic similarity detection accuracyVSAvoidinspection efficiency

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent replaces manual inspection with an automated system that converts relational tables to knowledge graphs and applies graph matching algorithms. This substitution of mechanical/manual processes with automated computational methods enables both high accuracy in semantic similarity detection and scalability to large data volumes, resolving the contradiction between precision and productivity.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If syntactical schema comparison is used to identify duplicate tables, then the process is simple and fast, but it fails to detect tables with different schemas but same information content

Engineering Contradiction:

Improvedetection speedVSAvoidsemantic similarity detection accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent introduces knowledge graphs as an intermediary representation between relational tables and comparison operations. By converting tables to knowledge graphs that capture semantic relationships and information content independently of syntactical schema, the system enables accurate semantic similarity detection while maintaining computational efficiency through graph matching algorithms.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If traditional duplicate detection techniques are applied, then content-duplicates can be identified, but semantically equivalent tables with different schemas remain undetected

Engineering Contradiction:

Improveduplicate detection completenessVSAvoiddetection system complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent changes the representation parameters from syntactical schema to semantic knowledge graph structure. This parameter transformation enables the detection system to identify tables based on their information content and semantic relationships rather than their structural form, achieving comprehensive duplicate detection including semantically equivalent tables with different schemas.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS12326871B2Detecting duplicate tables in data lake databases

Publication Date: 2025.06.10 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Detecting duplicate tables by converting relational tables to knowledge graphs; and mapping nodes in the knowledge graphs to sources in the relational tables. The method may further include applying graph matching to the knowledge graphs; and assessing degree of matching between matched knowledge graphs.