Large-Scale Text Clustering via Semantic Vector Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current large-scale text clustering methods are time-consuming, making it inefficient to quickly cluster large amounts of text data while ensuring the protection of private data.
Innovation Solution
A method and apparatus for large-scale text clustering that uses semantic representation models to determine semantic vectors, constructs a similarity matrix, selects central texts based on maximum similarities, and performs coarse and fine clustering to efficiently group texts, reducing the need for extensive similarity comparisons.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional text clustering methods are used on large-scale text data, then clustering accuracy can be maintained, but the processing time becomes excessively long
Solution Approach 1:
The patent segments the text clustering process into multiple stages: first constructing a similarity matrix and selecting initial central texts, then iteratively finding similar texts and performing fine clustering. This segmentation allows the system to process large-scale text data efficiently by breaking down the complex clustering task into manageable steps, significantly reducing overall processing time while maintaining accuracy.
Solution Approach 2:
The patent performs preliminary actions by first constructing the similarity matrix and selecting initial central texts before proceeding with the main clustering process. This preliminary setup enables subsequent iterations to work with pre-processed data, reducing the computational burden during the actual clustering phases and improving overall efficiency.
2Measurement precision
If all texts are compared to ensure accurate clustering, then clustering precision is improved, but the computational complexity increases significantly
Solution Approach 1:
The patent applies partial action by not comparing all texts exhaustively, but rather selecting M similar texts (where M is less than the total number of texts) for each central text. This partial comparison approach maintains sufficient clustering accuracy while significantly reducing computational complexity compared to comparing all possible text pairs.
Solution Approach 2:
The patent implements local quality by performing fine clustering only on selected similar texts and their neighborhoods, rather than applying uniform processing to all texts. This localized approach concentrates computational resources on areas that most impact clustering accuracy, reducing overall complexity while maintaining precision where it matters most.
Data Source
AI summary
A method includes coarse clustering and secondary fine clustering. First, semantic vectors respectively corresponding to a plurality of texts are determined by using a semantic representation model, and a similarity matrix between the plurality of texts is determined based on the semantic vectors of the plurality of texts. Next, in a coarse clustering phase, M similar texts with maximum similarities respectively corresponding to the plurality of texts are determined from the similarity matrix, and the corresponding texts are used as selected central texts when the similarities corresponding to the M similar texts are greater than a threshold, to quickly remove a large amount of isolated noise. Then, candidate class clusters are obtained based on data corresponding to the central texts in the similarity matrix, candidate class clusters with a cross-text are combined, and then secondary fine clustering is performed on a combined class cluster.