Text Vectorizer for Contextual Search Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document searching techniques fail to efficiently account for spelling variations and polysemic terms, leading to missed relevant results and inclusion of irrelevant ones, especially in large corpuses, despite advancements in AI and NLP.
Innovation Solution
A text vectorizer computes vectors for text units by generating word and phrase vectors, which are combined to produce document vectors, allowing for efficient comparison and ranking of search queries within a corpus using neural networks and unsupervised learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional keyword searching techniques are used, then literal word searching is adequate, but spelling variations and polysemic terms are not properly handled
Solution Approach 1:
The patent transforms text from discrete keyword representations into continuous vector representations, where semantic similarity is captured through vector space proximity. This parameter transformation enables the system to handle spelling variations and polysemic terms by comparing vector similarities rather than exact keyword matches, thereby improving both search accuracy and adaptability simultaneously
2Reliability
If AI and NLP techniques with auxiliary structures are applied, then contextual understanding improves, but algorithm complexity increases
Solution Approach 1:
The patent extracts the essential contextual information from complex NLP processing by representing text as vectors that capture semantic meaning. Instead of implementing full auxiliary structures like controlled vocabularies and ontologies, the invention extracts only the necessary semantic representations needed for search, thereby maintaining contextual understanding while significantly reducing algorithmic complexity
Solution Approach 2:
The patent simplifies complex NLP processing by transforming text into vector parameters that directly represent semantic meaning. This parameter transformation eliminates the need for complex segmentation, reorganization, and linking algorithms while preserving contextual understanding through vector similarity comparisons
3Productivity
If simple word-based searching algorithms are used, then processing speed is fast, but relevant results are missed in large corpuses
Solution Approach 1:
The patent maintains fast processing by transforming the search problem into vector similarity comparisons, which can be computed efficiently. By changing from discrete keyword matching to continuous vector space comparisons, the system achieves both high processing speed and improved relevance retrieval, as vector similarity captures semantic relationships that simple word-based algorithms miss
Data Source
AI summary
A text vectorizer is provided to compute vectors for respective text units. The text vectorizer computes a given vector for a given text unit by: (i) computing word vectors for respective words in the text unit; (ii) computing phrase vectors for respective phrases in the text unit; and (ii) combining the word vectors and the phrase vectors to produce the given vector for the given text unit. For a corpus of documents, the text vectorizer computes corpus vectors for the respective corpus documents. Search text is received, and, based thereon, the text vectorizer computes a search vector for the search text. Search scores are then computed for respective corpus documents by comparing the search vector with the corpus vectors of the respective corpus documents.