Text Relation Map for Accurate Semantic Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional text semantic matching techniques, such as TF-IDF, Jaccard similarity, and edit distance, have limitations in accurately measuring similarity between texts, especially when the distance between texts is greater, leading to inaccuracies in semantic matching applications.
Innovation Solution
A text processing method that calculates the distance between every two texts to construct a text relation map, where the similarity between non-associated text pairs is determined based on the path between nodes in the map, improving the accuracy of text similarity representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If conventional text semantic matching techniques (TF-IDF, Jaccard similarity, edit distance) are used, then the calculation process is simple, but the measurement precision of text similarity deteriorates when the distance between texts is greater
Solution Approach 1:
The patent introduces intermediate nodes and paths as mediators to calculate text similarity. Instead of directly comparing two texts, the method uses intermediate texts as bridges, where the similarity is computed through the product of similarities along the path. This intermediary approach enables accurate measurement of text similarity even when the texts are far apart in the semantic space, resolving the contradiction between simple calculation and precise measurement.
2Measurement precision
If direct distance calculation between every two texts is performed, then the text similarity can be obtained, but the computational complexity increases significantly for large numbers of texts
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing text similarities in a relation map during an offline phase. This pre-computation stores the results of distance calculations between texts that are within a certain threshold, so that during online querying, the system only needs to retrieve and combine pre-computed values through path multiplication, rather than performing full distance calculations for every query. This resolves the contradiction by shifting computational burden to a preliminary stage.
Solution Approach 2:
The patent segments the text similarity calculation into two distinct phases: offline pre-computation phase where the text relation map is built, and online query phase where similarities are retrieved and combined. This segmentation allows the system to handle large numbers of texts efficiently by separating the heavy computational work from the query processing, maintaining both accuracy and scalability.
3Measurement precision
If a text relation map with paths is constructed, then the text similarity representation becomes more comprehensive, but the device complexity and storage requirements increase
Solution Approach 1:
The patent applies partial action by only constructing paths and storing similarities for text pairs that meet certain criteria (e.g., within a distance threshold). Not all possible text pairs are fully processed; instead, the system focuses on building a sufficient subset of the relation map that enables accurate similarity calculation through paths. This selective approach maintains comprehensive representation while controlling the complexity and storage requirements of the data structure.
Data Source
AI summary
An embodiment of the present application provides a text processing method and apparatus, where the text processing method includes: obtaining N texts and calculating a text distance between every two texts, to obtain a text set, where the text set includes M text pairs and the text distance between the two texts of each text pair; constructing a text relation map based on the text set including the text node and the connection edge between two text nodes, where the weight of the connection edge reflects a text distance between the texts corresponding to the two text nodes; and determining a text similarity between the two texts in the non-associated text pair in the text relation map according to a path between the non-associated text pairs, where the non-associated text pair is a text pair with a text distance greater than a preset threshold value.