Variant Token Network Clustering for Scalable Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data clustering methods face performance and scalability issues due to the quadratic complexity of comparing records, especially when dealing with large volumes of data and numerous clusters, leading to inefficiencies in identifying cluster membership and requiring extensive computations.
Innovation Solution
A data clustering method that uses a search process to determine if a query record is close to existing clusters under an approximate distance measure, employing a narrowly targeted search with multiple queries to find candidate records and representative tokens, and then performs detailed comparisons only when necessary, allowing for incremental and batch mode operations while maintaining cluster integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional data clustering methods compare all records against each other to identify cluster membership, then clustering accuracy is maintained, but computational complexity increases quadratically with data volume
Solution Approach 1:
The patent segments the data clustering process into multiple phases: (1) extracting tokens from data records, (2) building a suffix tree structure to organize tokens, (3) performing targeted searches on the suffix tree to find candidate clusters, and (4) performing detailed comparisons only for promising candidates. This segmentation reduces the need for exhaustive pairwise comparisons while maintaining clustering accuracy.
Solution Approach 2:
The patent performs preliminary actions by pre-processing data records to extract tokens and build a suffix tree structure before actual clustering operations. This preliminary organization of data into a searchable structure enables efficient candidate identification during clustering, avoiding the need for exhaustive comparisons later.
2Measurement precision
If exhaustive comparisons are performed to ensure accurate cluster membership, then clustering precision is maintained, but processing time increases significantly
Solution Approach 1:
The patent applies partial action by performing detailed comparisons only on a subset of candidate records identified through the suffix tree search, rather than comparing against all records. The search process narrows down candidates to those most likely to match, and detailed comparisons are performed only on these limited candidates, significantly reducing processing time while maintaining accuracy.
Solution Approach 2:
The patent substitutes the mechanical exhaustive comparison system with a search-based system using suffix trees. Instead of systematically comparing every record pair, the system uses the suffix tree structure to guide searches and identify candidate clusters efficiently, replacing brute-force mechanics with intelligent search mechanisms.
3Reliability
If detailed comparisons are performed on all candidate records, then cluster integrity is ensured, but computational resources are excessively consumed
Solution Approach 1:
The patent applies local quality by performing detailed comparisons only in local regions of the search space where matches are likely to occur, as identified by the suffix tree search. Rather than uniformly processing all candidate records, the system focuses computational resources on specific local areas of the data space where cluster membership is probable, reducing overall resource consumption while maintaining integrity.
4Quantity of substance
If traditional clustering methods process large datasets, then comprehensive clustering coverage is achieved, but scalability deteriorates due to quadratic complexity
Solution Approach 1:
The patent introduces the suffix tree as an intermediary data structure between the raw data records and the clustering comparison process. This intermediary structure organizes tokens from all records in a compressed format, enabling efficient searches without requiring direct pairwise comparisons. The suffix tree mediator allows the system to scale to large datasets by reducing the computational burden from quadratic to near-linear complexity.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
Received data records, each including one or more values in one or more fields, are processed to identify one or more data clusters. The processing includes: identifying (110) tokens that each include at least one value or fragment of a value in a field or a combination of fields; generating (120) a network representing the identified tokens, with nodes of the network representing tokens and edges of the network each representing a variant relationship between tokens; and generating a graphical representation of the network with different subsets of nodes distinguished based at least in part on values associated with nodes, where a value associated with a particular node quantifies a count of a number of instances of the token represented by that particular node appearing within the received data records.