Orthogonal Transform Indexing for Incremental Data Stream Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing clustering techniques for high-dimensional streaming data are ineffective due to slow processing speeds, reliance on direct object comparisons, and inability to handle evolving data streams without prior knowledge of cluster numbers, failing to detect clusters in full dimensional spaces and assuming constant cluster numbers over time.
Innovation Solution
The implementation of an orthogonal transform-based indexing system for incremental clustering, which retrieves ranked neighbor lists and determines hub measures to identify clusters without requiring a predetermined number of clusters, using a kNN graph and Walsh-Hadamard transform for efficient nearest neighbor computation in high-dimensional spaces.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If direct comparison between pairs of data objects is used to determine clusters, then clustering accuracy is improved, but processing speed deteriorates
Solution Approach 1:
The patent segments the high-dimensional data space by projecting data objects onto multiple lower-dimensional subspaces. Each subspace contains a subset of features, allowing parallel processing of distance computations. This segmentation enables the system to maintain clustering accuracy while significantly improving processing speed by avoiding direct comparison in the full high-dimensional space.
Solution Approach 2:
The patent transforms the problem from direct high-dimensional comparison to multiple low-dimensional subspace comparisons. By changing the dimensionality approach - processing data in multiple 2D projections rather than one high-dimensional space - the system achieves both accuracy and efficiency. The dimensionality change allows parallel computation across multiple subspaces.
2Measurement precision
If iterative clustering algorithms are used, then clustering accuracy is improved, but processing speed deteriorates
Solution Approach 1:
The patent performs preliminary actions by pre-computing distance matrices in multiple 2D subspaces before the actual clustering process. These pre-computed distances are stored and reused during clustering, eliminating the need for repeated distance calculations in iterative algorithms. This preliminary computation significantly speeds up processing while maintaining accuracy.
Solution Approach 2:
The patent creates copies of the data in multiple 2D subspaces, allowing parallel distance computations to be performed on these copies rather than repeatedly accessing the original high-dimensional data. This copying approach enables efficient iterative clustering by maintaining pre-computed distance information across multiple projections.
3Productivity
If cluster numbers are predetermined, then processing speed is improved, but adaptability to evolving data streams deteriorates
Solution Approach 1:
The patent implements a dynamic clustering approach where the number of clusters is not fixed but evolves with the data stream. The system continuously monitors data characteristics and adjusts the number of clusters accordingly, allowing it to adapt to changing data patterns while maintaining efficient processing through the underlying 2D subspace framework.
Solution Approach 2:
The patent changes the parameter of cluster number from a static predetermined value to a dynamic variable that adapts to the data. By allowing the cluster count parameter to change based on data characteristics, the system achieves both speed and adaptability, resolving the contradiction between fixed parameters and evolving data requirements.
4Adaptability or versatility
If density-based projected clustering is used, then handling of different cluster shapes is improved, but processing of full high-dimensional space deteriorates
Solution Approach 1:
The patent segments the high-dimensional space into multiple 2D subspaces, each capturing different aspects of the data structure. This segmentation allows the system to handle various cluster shapes effectively in each subspace while avoiding the complexity of directly processing the full high-dimensional space. The segmented approach maintains versatility for different cluster geometries.
Solution Approach 2:
The patent addresses the limitation of projected clustering by working in multiple 2D subspaces rather than a single projected space. This dimensionality change approach allows the system to capture complex cluster shapes that may be missed in any single projection, while still avoiding the computational complexity of full high-dimensional processing.
Data Source
AI summary
Incremental clustering of a data stream via an orthogonal transform based indexing is disclosed. One example is a system including an indexing module that retrieves a ranked neighbor list for a data object in a data stream, where the ranked list is based on an orthogonal transform based indexing of an incrementally updated nearest neighbor graph. A reverse neighbor determination module identifies a reverse neighbor list for the data object, the reverse neighbor list comprising previously received data objects that include the data object in their respective ranked lists. An evaluator determines a hub measure for the data object, where the hub measure is a size of the reverse neighbor list. A hub identification module determines, based on the hub measure, if the data object is a hub, where the hub is representative of a cluster of similar data objects.