Document Similarity Index via Tag Index Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current document similarity techniques, such as Latent Semantic Indexing (LSI) and Latent Dirichlet Allocation (LDA), require significant computational resources and replicate efforts, leading to high costs and inefficiencies in characterizing and searching documents, especially when integrating disparate systems for complex document mining tasks.
Innovation Solution
The method involves computing a tag index for documents with keyword-weight pairs, identifying the most significant keywords, and calculating full similarities between documents to derive a document similarity index, with thresholds to control the number of similar documents selected for inclusion, thereby optimizing resource usage and reducing redundant operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full text search solutions and document similarity techniques (LSI, LDA) are implemented, then document characterization and similarity analysis capabilities are improved, but computational resource requirements and operational costs increase significantly
Solution Approach 1:
The patent segments the document analysis process into two distinct phases: (1) an offline phase that pre-computes and stores term frequencies and document vectors for all documents in the corpus, and (2) an online phase that rapidly calculates similarity scores using the pre-computed data. This segmentation eliminates redundant computations during similarity queries, significantly reducing operational computational resource consumption while maintaining accurate similarity analysis capabilities.
2Adaptability or versatility
If complex document mining techniques are integrated, then document processing capabilities are enhanced, but system complexity and implementation costs increase
Solution Approach 1:
The patent creates a universal document representation framework using term frequency vectors that can serve multiple document processing functions simultaneously. The same pre-computed document vectors and similarity calculation mechanism support both full-text search operations and document similarity analysis, eliminating the need for separate specialized systems and reducing overall system integration complexity.
3Reliability
If document parsing and mining operations are replicated across multiple techniques, then comprehensive document analysis is achieved, but resource efficiency decreases due to redundant operations
Solution Approach 1:
The patent performs preliminary action by pre-computing and storing term frequencies and document vectors during an offline phase before any similarity queries are executed. This preliminary preparation ensures that all necessary data for comprehensive document analysis is already available, eliminating the need to replicate parsing and mining operations during online similarity calculations, thus improving resource utilization efficiency while maintaining analysis comprehensiveness.
Data Source
AI summary
Methods, systems, and computer program products are provided for deriving and updating document similarity indices for a plurality of documents. The number of maintained similarities can be controlled to conserve CPU and storage resources.