Conjunctive Filtering in Embedding Retrieval via Vector Concatenation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing embedding-based information retrieval systems lack the ability to support conjunctive filtering during the candidate retrieval stage, leading to inaccurate selection of items based on user-specified filtering parameters, and existing solutions increase system complexity and cost.
Innovation Solution
Implementing a systematic approach to conjunctive filtering by encoding property values and filtering parameters as vector representations, which are then concatenated with item and query embeddings, allowing the similarity scoring model to boost scores for items satisfying filtering parameters, thereby ensuring accurate retrieval without increasing complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If embedding-based candidate retrieval is used, then retrieval speed and scalability are improved, but the ability to support conjunctive filtering is lost
Solution Approach 1:
The patent merges the embedding-based retrieval mechanism with conjunctive filtering capabilities by integrating filter parameter encoding into the embedding space. Filter parameters are encoded as vectors and combined with item embeddings through concatenation or addition, allowing the retrieval system to simultaneously perform semantic matching and hard filtering in a unified framework, thus maintaining both speed and filtering accuracy.
Solution Approach 2:
The patent introduces an intermediary encoding layer that translates filter parameters into the embedding space. This intermediary mechanism converts discrete filter conditions into continuous vector representations that can be processed by the embedding model, enabling the system to apply conjunctive filters without abandoning the embedding-based retrieval approach.
2Reliability
If traditional search indices are added to support filtering, then filtering capability is improved, but system complexity and cost increase
Solution Approach 1:
The patent combines multiple filtering capabilities into the existing embedding retrieval infrastructure. By encoding filter parameters as vectors and integrating them with the retrieval process, the system achieves conjunctive filtering without adding separate search indices or parallel filtering systems, thereby maintaining simplicity while enhancing functionality.
Solution Approach 2:
The patent makes the embedding retrieval system universal by enabling it to perform both semantic search and conjunctive filtering through a single unified mechanism. The same embedding space and retrieval algorithm handle both types of queries, eliminating the need for specialized filtering infrastructure and reducing overall system complexity.
Data Source
AI summary
Described herein is a technique to facilitate conjunctive filtering for an embedding-based information retrieval system. Each item in a content collection is initially analyzed to identify its property values for some set of properties. For each item, its property values are encoded in the form of a vector, and concatenated with an item embedding for the item to derive an enhanced item embedding. When a query is received, a query embedding is generated. Any filtering parameters received with or as part of the query are encoded in the form of a vector, and the vector is concatenated with the query embedding to derive an enhanced query embedding. The enhanced query embedding is then used in a âkâ approximate nearest neighbor search to identify items relevant to the query and having property values that satisfy the filtering parameters.