Text Similarity Sampling for Massive Corpus Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text similarity measurement methods struggle with high calculation costs and difficulty in processing massive texts, especially when calculating similarity between two texts independently, and require large amounts of text corpora for pre-training.
Innovation Solution
A text similarity measurement method involving constructing a joint probability distribution of two text strings, sampling to obtain a sampling string, calculating distance matrices, and determining similarity based on these matrices, reducing dimensionality and information loss while improving calculation efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional text similarity measurement methods are used to process massive texts, then measurement precision can be maintained, but calculation cost increases significantly and processing efficiency deteriorates
Solution Approach 1:
The patent segments the original text into multiple substrings and selects representative sampling strings from each segment. This segmentation approach reduces the dimensionality of the input data while preserving the essential semantic information, enabling efficient batch processing of massive texts without requiring full-text analysis
Solution Approach 2:
The patent applies partial action by processing only a sampled portion of the text rather than the complete text. By constructing probability distributions from sampling strings and using them to represent the entire text, the method achieves acceptable similarity measurement with significantly reduced computational resources
2Measurement precision
If full text processing is performed to ensure accurate similarity measurement, then measurement precision is maintained, but the complexity of the measurement process increases and independent calculation becomes difficult
Solution Approach 1:
The patent introduces probability distributions as intermediary representations between the original text and the similarity measurement process. The sampling strings serve as mediators that capture the essential characteristics of the full text, allowing similarity to be calculated from these compact representations rather than from the complete texts directly
Solution Approach 2:
The patent creates simplified copies of the original text in the form of sampling strings and their associated probability distributions. These copies retain the essential statistical properties needed for similarity measurement while being much more compact and easier to process than the original full texts
Data Source
AI summary
The present disclosure relates to a text similarity measurement method and apparatus, device, storage medium, and program product. The method includes: obtaining a first text string and a second text string; constructing a joint probability distribution of the first text string and the second text string, and sampling the joint probability distribution to obtain a sampling string; calculating a distance from the first text string to the sampling string to obtain a first distance matrix, and calculating a distance from the second text string to the sampling string to obtain a second distance matrix; and determining a similarity between the first text string and the second text string based on the first distance matrix and the second distance matrix.