Semantic Clustering for Data Set Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large companies face challenges in managing tens of thousands of data sets stored in data storage systems, where similar or duplicate data sets create unnecessary overhead, confusion, and inefficiencies, leading to suboptimal data set selection due to inadequate discoverability and keyword-based search limitations.
Innovation Solution
Implementing a data set clustering system that automatically clusters data sets by similarity, ranks them based on quality, and deduplicates or merges similar sets, presenting clusters in ranked order to users during searches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If keyword-based search is used to find data sets, then search simplicity is maintained, but data set discoverability and selection accuracy deteriorate due to inability to identify similar or duplicate data sets
Solution Approach 1:
The patent introduces semantic similarity clustering as an intermediary layer between keyword search and data set selection. The system computes semantic similarity scores between search queries and data set metadata, then presents clustered results grouped by similarity. This mediator enables users to discover related data sets that keyword search would miss, while maintaining the simplicity of keyword-based input.
Solution Approach 2:
The patent replaces traditional mechanical keyword matching with semantic similarity computation based on embeddings and vector representations. Instead of exact string matching, the system uses machine learning models to understand the semantic meaning of queries and data set descriptions, enabling discovery of conceptually related data sets even with different wording.
2Quantity of substance
If all data sets are stored without deduplication, then data availability is maximized, but storage burden and system overhead increase
Solution Approach 1:
The patent implements automated deduplication by merging data sets that exceed a similarity threshold. When duplicate or near-duplicate data sets are detected through semantic similarity clustering, the system consolidates them into a single representative data set, reducing storage requirements while preserving data availability through intelligent reference management.
Solution Approach 2:
The system selectively discards redundant data sets identified through similarity analysis while recovering essential information by maintaining references to the original data sets. This allows the system to eliminate storage overhead of duplicates while preserving the ability to access the underlying data through the retained representative data set.
3Loss of information
If semantic similarity clustering is implemented, then data set discoverability and quality selection improve, but system complexity and computational overhead increase
Solution Approach 1:
The patent segments the data set management system into distinct modular components: embedding generation modules, similarity computation modules, clustering modules, and deduplication modules. Each component handles a specific aspect of semantic analysis independently, making the overall complex system more manageable, testable, and maintainable while enabling advanced discoverability features.
4Adaptability or versatility
If duplicate data sets are allowed, then data creation freedom is maintained, but confusion and inefficiency increase for users
Solution Approach 1:
The patent implements feedback mechanisms where the system continuously monitors data set similarity and automatically identifies duplicates. When potential duplicates are detected, the system provides feedback to users through notifications, suggestions for consolidation, or automatic merging actions. This feedback loop maintains user freedom to create data sets while preventing confusion through active duplicate management.
Data Source
AI summary
Embodiments of the system in include a method of clustering data sets based on the data set characteristics. A data repository stores a plurality of heterogeneous data sets. For a selected group of data sets (i) obtaining metadata; (ii) identifying upstream data sets; and (iii) extracting semantic features. The method further comprises clustering the plurality of data sets into one or more clusters based on similarity, wherein the similarity is determined based on a combination of the metadata, the upstream data sets, and the semantic features of the selected data sets. In one embodiment, the data sets are ranked within each cluster based on data set quality. The method enables the presenting of at least one cluster including one or more ranked data sets in response to a search query.