Document Similarity Vector Conversion Using Inverted Index Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining similar document sets from a large number of documents are inefficient and lack accuracy, as they require one-by-one similarity calculations, leading to significant time costs and performance deficiencies.
Innovation Solution
A method and apparatus that convert document word sets into vectors using predefined conversion rules, allowing for the determination of similar document sets based on vector differences, with the option to use multiple conversion rules and store vectors in a multi-dimensional matrix or hash map to enhance efficiency and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If one-by-one similarity calculation is used between documents and target document, then measurement precision is maintained, but productivity deteriorates significantly due to time cost
Solution Approach 1:
The patent segments the similarity calculation process into two stages: (1) bulk conversion of all documents to vectors using inverted indexes, and (2) differential comparison of converted vectors. This segmentation allows most documents to be processed in batch rather than individually, dramatically improving productivity while maintaining accuracy through the second verification stage.
Solution Approach 2:
The patent performs preliminary conversion of document word sets to vectors before the actual similarity determination. By pre-converting all documents to vector format using inverted indexes, the system eliminates the need for repeated conversion during similarity calculations, reducing time cost while preserving measurement precision.
2Ease of operation
If traditional keyword intersection method is used, then ease of operation is maintained, but measurement precision deteriorates due to insufficient accuracy
Solution Approach 1:
The patent changes the parameter representation from discrete keyword matches to continuous vector differences. By converting documents to vectors and calculating differential values, the system achieves more precise similarity measurement while maintaining ease of operation through automated vector comparison processes.
Data Source
AI summary
Embodiments of the present disclosure provide a method and an apparatus for determining a similar document set to a target document from a plurality of documents. Each of the multiple documents and the target document may include a plurality of words, and each of words corresponds to a different integer. The method comprises: for each document among the plurality of documents and the target document, obtaining a set of integers associated with a document based on a set of words associated with the document, converting the set of integers associated with the document into a vector with a same dimension based on a predefined conversion rule; and determining the similar document set based on differences between the corresponding vectors for the multiple documents and the vector for the target document.