Document Clustering Using Time-Sliced Vector Space Modeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional information retrieval systems using the vector space model struggle with clustering documents containing loosely related keywords, leading to inefficient resource usage and poor relevance, especially in databases with time-evolving content from heterogeneous sources.
Innovation Solution
The system employs vector space modeling with time slices or chunks of documents, applying singular value decomposition for dimension reduction and using principal components to identify relevant clusters, with centroid vectors computed to track focused items over time, reducing computational resources and improving search efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If typical vector space model algorithms are used for information retrieval, then information can be retrieved from databases, but huge hardware resources such as CPU time and memory resources are consumed
Solution Approach 1:
The patent segments the large database into multiple clusters based on semantic similarity. By dividing the database into smaller manageable clusters, the system can process and retrieve information from specific clusters rather than scanning the entire database, significantly reducing CPU time and memory resource consumption while maintaining effective information retrieval capability.
2Adaptability or versatility
If cluster formation based on vector space model is applied, then documents can be clustered, but the computation of dimension reduction consumes long CPU time and the clusters may not have sufficient relevancy to items that change with elapsed time
Solution Approach 1:
The patent implements dynamic cluster formation that adapts to changing items over time. The system continuously updates cluster structures based on elapsed time and evolving document contents, allowing clusters to remain relevant to changing items while using optimized algorithms that reduce dimension reduction computation time and CPU resource consumption.
3Ease of operation
If typical cluster search algorithms are used, then search can be performed on databases with rigidly determined keywords, but documents including different keywords which refer to the same contents or semantics tend to pose problems in forming clusters
Solution Approach 1:
The patent changes the fundamental parameter for clustering from rigid keyword matching to semantic similarity measurement. By using semantic analysis and similarity metrics instead of exact keyword matching, the system can effectively cluster documents that use different but semantically related keywords, greatly improving adaptability while maintaining ease of operation.
Data Source
AI summary
A system for information clustering comprising a data accumulation part for accumulating documents in a document repository, the documents having loosely related attributes, and defining a cluster between the documents being time sliced so as to define chunks of the documents; a vector space generation part for generating document-keyword vectors, the document-keyword vectors consisting of sparse numeral values depending on presence of key words; a dimension reduction part for reducing dimensions of the keywords to create a dimension reduction matrix of the document-keyword matrix; a centroid vector determination part for generating a centroid vector of the cluster, the centroid vectors being defined from keywords and weight of documents within the cluster; and an item repository for storing the centroid vectors together with the keywords and the weights of the centroid vector.