Navigable Search Graph Index for High-Throughput Dataset Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional search systems for large and complex datasets face inefficiencies in terms of latency, throughput, and accuracy, often requiring excessive computing resources and failing to achieve satisfactory recall rates, especially when indexing a large number of points on a single node.
Innovation Solution
A search graph generation system that generates a navigable search graph with vertices representing dataset objects, using a greedy search algorithm to ensure navigability and iteratively update out-neighbor data to optimize the number of hops between vertices, allowing for accurate and efficient search queries by reducing processing expenses and storage requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional search techniques are used on large datasets, then computing resources can be expended to identify accurate results, but latency increases and throughput decreases
Solution Approach 1:
The system pre-generates a search graph index structure before queries are issued. This index is built by computing vertex representations and establishing navigable paths in advance, so that when a query arrives, the system can immediately traverse the pre-computed graph rather than performing expensive computations at query time. This preliminary indexing action resolves the contradiction by shifting computational burden from query execution to index construction.
Solution Approach 2:
The patent introduces a search graph as an intermediary data structure between the raw dataset and the query processing system. This graph serves as a mediator that pre-organizes relationships among data points, enabling efficient navigation during queries. The graph index acts as the intermediary that translates complex dataset relationships into traversable paths, resolving the latency-accuracy tradeoff.
2Measurement precision
If conventional search techniques are used on large datasets, then accurate results can be identified, but computing resources are excessively consumed
Solution Approach 1:
The system segments the large dataset into a graph structure where data points are represented as vertices and relationships as edges. This segmentation transforms the monolithic search problem into a structured graph traversal problem, enabling efficient navigation through pre-computed paths. By segmenting the data into navigable graph components, the system reduces computing resource consumption during queries while maintaining accuracy.
Solution Approach 2:
The patent replaces traditional mechanical search algorithms (which systematically scan or index datasets) with a graph-based navigation approach. Instead of using conventional search mechanics that require extensive computational resources, the system uses graph traversal mechanics that leverage pre-computed navigable paths, significantly reducing computing resource consumption while preserving search accuracy.
3Speed
If the number of hops between vertices is reduced to improve search speed, then latency decreases, but the complexity of maintaining the graph structure increases
Solution Approach 1:
The system employs dynamic graph construction where the graph structure adapts to the data distribution. The graph is built dynamically by computing vertex representations and establishing navigable paths based on actual data relationships, rather than using a fixed rigid structure. This dynamic approach allows the graph to optimize hop counts while managing complexity through adaptive structure formation.
Solution Approach 2:
The patent changes key parameters during graph construction, including vertex representation dimensions, edge connection criteria, and path optimization parameters. By adjusting these parameters during the indexing phase, the system optimizes the balance between hop count (affecting search speed) and graph complexity. Parameter tuning during index construction enables the system to achieve low latency without excessive structural complexity.
Data Source
AI summary
The present disclosure relates to generating a search graph or search index to aid in receiving a search query and identifying results of a dataset based on the search query. For example, systems disclosed herein may generate a navigable search graph including vertices representative of objects or points within a dataset that enables a computing device having access to the search graph to navigate vertices of the graph along an identified path until arriving at a point within the search graph that corresponds to a value associated with the search query. Upon identifying a location within the graph corresponding to the search query, systems disclosed herein may identify a neighborhood of points (e.g., vertices) corresponding to items from the dataset and output a set of results for the search query representative of determined results for the search query.