Semantic Search Scoring with Relevancy and Popularity for Large Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data asset search methods primarily rely on keyword searches, which are inefficient for large datasets and require users to have familiarity with specific keywords, leading to irrelevant results and resource wastage.
Innovation Solution
A system that utilizes a combination of traditional algorithms, large language models, vector construction, and vector databases to generate augmented search results by incorporating pattern matching, semantic similarity, and popularity statistics, enabling intuitive search navigation and discovery of relevant data assets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If keyword-based search algorithms are used, then users can search for data assets using specific keywords, but the search results become irrelevant for large datasets and require users to have familiarity with entity-specific keywords
Solution Approach 1:
The patent transforms the search approach by changing the parameter representation from discrete keywords to continuous vector embeddings. Data assets and search queries are both converted into vector representations, enabling semantic similarity measurement rather than exact keyword matching. This allows the system to understand the meaning and context of queries even when exact keywords are unknown to users.
Solution Approach 2:
The patent replaces the traditional mechanical keyword-matching system with a semantic vector-based system. Instead of relying on users to input precise keywords and the system performing string matching, the system uses vector databases to compute semantic similarity between query vectors and data asset vectors, automatically understanding user intent without requiring keyword familiarity.
2Productivity
If traditional search algorithms are used, then the system can handle simple searches, but it generates many irrelevant results for large datasets which wastes organizational resources and users' time
Solution Approach 1:
The patent performs preliminary action by pre-computing and storing vector embeddings for all data assets in the vector database before search time. This allows the search system to quickly compare user queries against pre-processed vector representations, significantly reducing the time needed to find relevant results in large datasets without sacrificing accuracy.
3Measurement precision
If keyword familiarity is required for effective searching, then users can perform targeted searches, but new users unfamiliar with specific keywords cannot effectively navigate or discover data assets
Solution Approach 1:
The patent creates a universal search interface that works for all users regardless of their keyword familiarity. The vector-based semantic search system handles both simple natural language queries from new users and complex targeted searches from experienced users, making the system universally accessible while maintaining search precision through semantic understanding rather than keyword matching.
Data Source
AI summary
Systems and methods for generating augmented search results are disclosed. An example method is performed by one or more processors of a search results ranking system and includes receiving a transmission over a communications network from a computing device associated with a user of the search results ranking system, the transmission including a search query, submitting, to a vector database, a token query matching a tokenized version of the search query against a plurality of data assets, submitting, to the vector database, one or more vector queries matching a vectorized version of the search query against the plurality of data assets, identifying, based on results of the token query and the one or more vector queries, contextually relevant results among the plurality of data assets, and generating augmented search results for the search query based on the contextually relevant results.