Density-Guided Hierarchical Vector Clustering to Reduce Calculation Load
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information processing systems face a significant calculation load due to repetitive calculations when grouping high-dimensional vector data using the K-nearest neighbor algorithm, leading to a combinational explosion.
Innovation Solution
Implement an information processing system that performs hierarchical clustering based on Gini coefficients to determine the density of vector data, allowing for the division or integration of clusters, thereby reducing the calculation load by focusing on dense data regions and stopping the process when the desired number of clusters is reached.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the K-nearest neighbor algorithm is used to group data by relative distances, then data can be grouped effectively, but the calculation load becomes too large due to repetitive calculations
Solution Approach 1:
The patent segments the clustering process into hierarchical levels, dividing the data space into multiple tiers of clusters. Instead of calculating all pairwise distances at once, the algorithm performs clustering at different hierarchical levels, where coarser clusters are formed first and then progressively refined. This segmentation reduces the computational complexity from O(n²) to approximately O(n log n) by avoiding redundant distance calculations across the entire dataset at each level.
2Adaptability or versatility
If hierarchical clustering is performed by specifying the number of clusters in advance, then the clustering process can be controlled, but the calculation load increases due to repetitive calculations
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing distance metrics or similarity measures during the clustering process. Before performing the full hierarchical clustering, the algorithm prepares distance matrices or approximate nearest neighbor structures that can be reused across multiple clustering iterations. This preliminary computation avoids redundant calculations when the clustering process needs to be repeated with different parameters or when refining clusters at different hierarchical levels.
3Measurement precision
If relative distances between all pieces of vector data are calculated to group similar data, then accurate grouping can be achieved, but combinational explosion occurs
Solution Approach 1:
The patent transforms the clustering problem from a direct high-dimensional space comparison into a hierarchical structure that adds a temporal/dimensional layer. Instead of computing all pairwise distances in the original high-dimensional space simultaneously, the algorithm introduces a hierarchical time dimension where clustering proceeds through multiple stages. At each stage, only relevant subsets of data are compared, effectively reducing the combinatorial complexity by distributing the computation across hierarchical levels rather than requiring all combinations at once.
Data Source
AI summary
An information processing system includes: data acquisition means for acquiring vector data of a plurality of dimensions; clustering means for grouping, based on relative distances between pieces of the vector data acquired by the data acquisition means, the pieces of the data in which the relative distances therebetween are close to each other, and performing hierarchical clustering for repeating the grouping; and density calculation means for calculating density of each of clusters of the pieces of the vector data. The clustering means divides or integrates the clusters based on the density of each of the clusters calculated by the density calculation means when the clustering means performs the hierarchical clustering.