Text Similarity via Segment-Level Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text comparison methods struggle when comparing large articles to query texts, as relevant sections within longer articles may be overshadowed by non-relevant portions, leading to missed detections of relevant articles.
Innovation Solution
Partitioning large articles into segments based on sentences, paragraphs, or punctuation, using a segmenting algorithm trained on query texts, allows for comparison of query texts with each segment, enabling the determination of overall similarity values for the articles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large articles are compared as whole texts to query texts, then the comparison process is simple, but relevant sections may be overshadowed by non-relevant portions leading to missed detections
Solution Approach 1:
The patent divides large articles into multiple text segments (e.g., paragraphs or sections) and compares each segment individually to the query text. This segmentation allows relevant sections to be detected independently without being overshadowed by non-relevant portions, thereby improving detection accuracy while managing complexity through systematic processing of divided units.
2Measurement precision
If articles are partitioned into segments for comparison, then relevant sections can be detected more accurately, but the processing time and computational resources increase
Solution Approach 1:
By segmenting articles into smaller units, the patent enables precise similarity measurements for each segment independently. This approach improves measurement precision by focusing on relevant sections without being diluted by non-relevant content, while the modular nature of segment comparison allows for efficient processing and potential parallelization to mitigate time losses.
3Reliability
If segment-level comparison is performed, then irrelevant portions do not mask relevant sections, but the overall similarity determination becomes more complex
Solution Approach 1:
The patent segments articles to enable reliable detection of relevant sections by comparing each segment independently to the query text. This segmentation ensures that irrelevant portions do not mask relevant content, improving relevance detection reliability. The complexity of determining overall similarity from segment-level comparisons is managed through systematic aggregation of segment similarities.
Data Source
AI summary
Systems and methods are provided to compare a target sample of text to a set of textual records, each textual record including a sample of text and an indication of one or more segments of text within the sample of text. Semantic similarity values between the target sample of text and each of the textual records are determined. Determining a particular semantic similarity value between the target sample of text and a particular textual record of the corpus includes: (i) determining individual semantic similarity values between the target sample of text and each of the segments of text indicated by the particular textual record, and (ii) generating the particular semantic similarity value between the target sample of text and the particular textual record based on the individual semantic similarity values. A textual record is then selected based on the semantic similarities.