Text Similarity Sampling for Lower-Cost Massive Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text similarity measurement methods struggle with high calculation costs and difficulty in processing massive texts, leading to inefficiencies and information loss, especially in big data scenarios.
Innovation Solution
A method involving constructing a joint probability distribution of two text strings, sampling to obtain a sampling string, and calculating distance matrices to determine similarity, reducing dimensionality while minimizing information loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional text similarity measurement methods are used to process massive texts, then measurement precision is maintained, but calculation cost increases significantly and processing efficiency decreases
Solution Approach 1:
The patent segments the original text into multiple substrings and selects representative sampling strings from each segment. This segmentation approach reduces the dimensionality of the input data while preserving the essential information needed for similarity measurement, thereby lowering calculation costs and improving processing efficiency for massive texts
Solution Approach 2:
Instead of processing the entire text, the patent applies partial action by selecting only certain sampling strings from the text segments. This partial processing approach maintains measurement precision while significantly reducing the computational burden associated with processing complete massive texts
2Device complexity
If dimensionality reduction is applied to reduce calculation complexity, then processing efficiency improves, but information loss increases
Solution Approach 1:
The patent performs preliminary segmentation of the text into meaningful substrings before selecting sampling strings. This preliminary action ensures that the dimensionality reduction process preserves structurally important information while reducing complexity, as the segmentation is done in advance to maintain semantic integrity
Solution Approach 2:
The patent changes the parameter of text representation from complete text to selected sampling strings. This parameter change allows for reduced calculation complexity while minimizing information loss by carefully selecting representative samples that retain the essential characteristics of the original text
3Measurement precision
If full text processing is performed to ensure accurate similarity measurement, then measurement precision is improved, but processing time increases
Solution Approach 1:
The patent extracts sampling strings from segmented text portions, taking out only the essential representative elements needed for similarity measurement. This extraction approach maintains measurement precision by preserving key information while dramatically reducing processing time compared to analyzing complete texts
Data Source
Figure 1~2
Figure 3a~3b
Figure 4a~4b
AI summary
The present disclosure relates to a text similarity measurement method and apparatus, device, storage medium, and program product. The method includes: obtaining a first text string and a second text string; constructing a joint probability distribution of the first text string and the second text string, and sampling the joint probability distribution to obtain a sampling string; calculating a distance from the first text string to the sampling string to obtain a first distance matrix, and calculating a distance from the second text string to the sampling string to obtain a second distance matrix; and determining a similarity between the first text string and the second text string based on the first distance matrix and the second distance matrix.