Per-Query Shard Relevance Prediction for Product Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing shard selection algorithms in large-scale product search systems face challenges such as reliance on document-based corpora, strict latency requirements, and difficulty in quantifying infrastructure cost, especially in e-commerce contexts where product catalogs have limited text information but rich attribute data.
Innovation Solution
A deep learning-based approach is employed to predict per-query shard relevance using knowledge of product shard membership and customer feedback, reformulating the shard selection problem as a multi-label query intent classification task, utilizing light feed-forward neural networks to balance prediction quality with low latency and reduce infrastructure costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional document-based corpora are used for shard selection, then search relevance can be maintained, but infrastructure costs increase and latency increases due to processing large document corpora
Solution Approach 1:
The patent extracts and utilizes only the most critical shard-level metadata (product category, brand, price range) instead of processing entire document corpora. This selective extraction of essential features enables shard selection without the infrastructure overhead of handling complete product descriptions and reviews.
Solution Approach 2:
The system performs partial action by querying only the necessary shard metadata rather than processing all documents. This approach uses a subset of information (category, brand, price) sufficient for effective shard selection, avoiding the excessive computational cost of full document processing while maintaining search relevance.
2Measurement precision
If traditional document-based corpora are processed, then search relevance can be maintained, but query latency increases due to strict latency requirements
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing shard-level metadata (product category, brand, price range) before queries arrive. This pre-processing of shard characteristics enables immediate query response without the time-consuming task of processing complete document corpora during query execution.
Solution Approach 2:
By extracting only the essential metadata features (category, brand, price) from product catalogs and storing them at the shard level, the system eliminates the need to process lengthy product descriptions and reviews during query time, thereby reducing latency while maintaining relevance.
3Quantity of substance
If all shards are queried to ensure comprehensive search results, then search completeness improves, but infrastructure costs and compute overhead increase
Solution Approach 1:
The patent applies local quality by tailoring the search strategy to each query's specific needs. Instead of uniformly querying all shards, the system analyzes the query and selectively queries only the local shards most likely to contain relevant results based on product category, brand, and price range metadata, reducing compute overhead while maintaining completeness.
Solution Approach 2:
The system performs partial action by querying only the necessary subset of shards rather than all shards. This selective querying approach maintains search completeness for relevant products while significantly reducing the compute overhead and infrastructure costs associated with querying the entire distributed database.
4Measurement precision
If deep learning models with large parameters are used for shard selection, then prediction accuracy improves, but computational resources and training time increase
Solution Approach 1:
The patent extracts and utilizes only the most essential shard metadata features (product category, brand, price range) for training the deep learning model. This selective feature extraction reduces the model's input dimensionality and complexity while maintaining high prediction accuracy for shard relevance, avoiding the need to process entire product documents.
Data Source
AI summary
Devices and techniques are generally described for per-query prediction of shard relevance for search. In some examples, a search system may receive a first search query. A first score may be determined for a first database partition, the first score indicating a relevancy of the first search query to first data stored by the first database partition. Similarly, a second score may be determined for a second database partition, the second score indicating a relevancy of the first search query to second data stored by the second database partition. A determination may be made that the first search query is related to the first data stored by the first database partition. A determination may be made, based at least in part on the second score, that the first search query is unrelated to the second data stored by the second database partition.