Data Lake Table Clustering for Redundant Data Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data lakes contain large volumes of redundant data due to the lack of data relatedness information, making it difficult and computationally expensive to identify and remove, and existing technologies fail to consider content-level similarity between tables.
Innovation Solution
A data provenance system that clusters tabular data based on schema-level and statistical similarities, calculates similarity scores, and forms graphs to determine duplicate data, efficiently identifying and deleting redundant data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data lakes store huge volumes of data without provenance information, then data storage capacity is improved, but data quality and redundancy management deteriorate
Solution Approach 1:
The system performs preliminary actions by computing and storing provenance information (schema-level features, statistical features, embeddings) for each table when data is ingested into the data lake. This pre-computation enables efficient duplicate detection later without requiring full data reprocessing, thus maintaining data quality while preserving storage capacity.
Solution Approach 2:
The system introduces provenance information as an intermediary layer between raw data storage and data quality assessment. This intermediary contains metadata (schema features, statistical distributions, embeddings) that enables efficient comparison and detection of duplicate tables without directly analyzing the actual data content, thus maintaining both storage capacity and data quality.
2Manufacturing precision
If existing technologies scan all data to identify redundant data, then data quality is improved, but computational cost increases
Solution Approach 1:
The system segments the duplicate detection process into multiple stages: first comparing schema-level features (column names, data types), then statistical features (value distributions, null ratios), and finally using pre-computed embeddings for similarity assessment. This segmentation allows the system to quickly filter out non-duplicates at lower levels before performing more computationally expensive comparisons, thus maintaining data quality while reducing computational cost.
Solution Approach 2:
The system performs partial action by not scanning all data content to identify duplicates. Instead, it uses provenance information (pre-computed metadata about schema, statistics, and embeddings) to identify candidate duplicates, then performs selective verification only on promising candidates. This partial approach maintains data quality assessment while significantly reducing computational overhead compared to full data scanning.
3Measurement precision
If the system computes similarity scores for all table pairs, then duplicate detection accuracy is improved, but processing time increases
Solution Approach 1:
The system performs preliminary action by pre-computing embeddings and statistical features for each table and storing them in the provenance information. When duplicate detection is needed, the system directly compares these pre-computed features rather than computing them on-the-fly, thus maintaining high detection accuracy while significantly reducing processing time for similarity score computation.
Data Source
AI summary
A dataset comprising tables is received. Embeddings are generated for column titles of a table. Based on the embeddings, similar tables are clustered. The tables are organized into smaller clusters based on statistical similarities. Similarity scores are calculated for tables within the same cluster. A relatedness graph is created based on the similarity scores; similar tables are represented by nodes connected by edges. If the similarity score for a pair of tables exceeds a threshold, a table is deleted.