Document Clustering Using Weighted Semantic Matrices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search technologies are limited in comparing and clustering both structured and unstructured documents, particularly in the intellectual property domain, as they often rely on pre-classified scientific or patent texts, neglecting company-specific texts like marketing brochures and manuals, which are not easily integrated into searchable databases due to varying word meanings and data volume concerns.
Innovation Solution
A system that imports structured and unstructured documents into a database, generates a word/word pair index, and uses semantic analysis to build a weighted document-word matrix, allowing for clustering and comparison through techniques like Latent Semantic Indexing (LSI), with refinements based on predefined rules for stopwords, transliterations, and stemming, enabling cross-domain searches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If company-specific texts like marketing brochures and manuals are included in searchable databases, then information retrieval capability is improved, but data volume increases and storage space is consumed
Solution Approach 1:
The patent extracts only the essential semantic information from company-specific texts by generating word indexes and document-word matrices, rather than storing and processing the complete text data. This extraction approach maintains search capability while significantly reducing the effective data volume that needs to be managed.
Solution Approach 2:
The patent segments documents into individual words and creates separate word indexes and document-word matrices. This segmentation allows the system to process and search through specific terms without handling the entire document corpus, effectively managing data volume while preserving retrieval capability.
2Adaptability or versatility
If company-specific texts with varying word meanings are integrated into searchable databases, then search coverage is improved, but measurement precision deteriorates due to word ambiguity
Solution Approach 1:
The patent changes the parameter of word representation by using document-word matrix frequencies and semantic analysis to capture contextual meanings. Instead of relying on fixed word definitions, the system adjusts word importance based on their frequency and distribution across documents, thereby handling word ambiguity while maintaining search precision.
3Measurement precision
If semantic analysis techniques are applied to all document types, then clustering accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent applies semantic analysis selectively to generate word indexes and document-word matrices rather than performing exhaustive semantic processing on all documents. This partial application of semantic techniques achieves sufficient clustering accuracy without the full computational burden of complete semantic analysis across the entire corpus.
Data Source
AI summary
Certain example embodiments relate to techniques for analyzing documents. A plurality of documents/document portions are imported into a database, with at least some of the documents/document portions being structured and at least some being unstructured. The imported documents/document portions are organized into one or more collections. A selection of at least one of the one or more collections is made. An index of words and/or groups of words is built (and optionally refined in accordance with one or more predefined rules) based on each of the document or document portion in each selection. A document-word matrix is built (and optionally weighted using a semantic approach), with the matrix including a value indicative of a number of times each word and/or group of words in the index appears in each document/document portion. One or more clusters of documents are generated using the document-word matrix.