Multi-modal Search Router Balancing Compute Cost and Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Web search engines face challenges in balancing the cost of operating search equipment with the accuracy of responses, particularly between traditional search engines and chat engines powered by large language models.
Innovation Solution
A web search router is implemented to make routing decisions based on provider criteria such as compute costs and response accuracies, as well as user-specified preferences, using a criterion processor that operates a machine learning model to determine whether to route search requests to a search engine or a chat engine.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If chat engine is used to service search requests, then response accuracy is improved, but operating cost increases
Solution Approach 1:
A routing system is introduced as an intermediary between search requests and the two engine types (search engine and chat engine). The routing system evaluates each request and directs it to the most appropriate engine, ensuring that chat engine is only used when necessary for complex queries requiring content generation, while simpler queries are handled by the more cost-effective search engine.
Solution Approach 2:
The system dynamically changes the operational parameters by adjusting which engine type handles which request based on query characteristics. By analyzing request features and determining the appropriate engine type on a per-request basis, the system optimizes the balance between response accuracy and operating cost without requiring a fixed configuration.
2Loss of energy
If traditional search engine is used, then operating cost is reduced, but response accuracy for complex queries deteriorates
Solution Approach 1:
The routing system acts as an intermediary that prevents complex queries from being incorrectly handled by the traditional search engine. It evaluates query characteristics and ensures that requests requiring content generation are directed to the chat engine, thereby maintaining response accuracy while avoiding unnecessary use of expensive resources for simple queries.
Solution Approach 2:
The system segments the search request processing into two distinct pathways: one for simple queries handled by the traditional search engine and another for complex queries requiring chat engine. This segmentation allows each engine type to operate in its optimal performance zone, maintaining accuracy where needed while controlling costs overall.
3Reliability
If chat engine is used for all requests, then response accuracy is improved, but compute resource consumption increases
Solution Approach 1:
The routing system serves as an intermediary that filters and directs only those requests requiring content generation to the chat engine. By evaluating query characteristics beforehand, it prevents unnecessary chat engine usage for simple retrieval queries, thereby significantly reducing overall compute resource consumption while maintaining accuracy for queries that truly need it.
Solution Approach 2:
Instead of applying the resource-intensive chat engine to all requests (excessive action), the system applies it only partially - specifically to queries where content generation is actually needed. This partial application of the powerful chat engine achieves the necessary accuracy improvement without the prohibitive cost of universal deployment.
Data Source
AI summary
Multi-modal search systems with improved search request routing are provided. A device can include a module that identifies, based on content of a search request, provider criterion that indicates factors to be considered in making a routing decision, a criterion processor that determines, based on the provider criterion, a routing decision indicating whether to route the search request to a search engine or a chat engine based, at least in part, respective compute costs of servicing the search request using the search engine and the chat engine, respectively, and respective accuracies of responses provided responsive to the search request using the search engine and the chat engine, respectively, and an output port coupled to receive the search request and to provide the search request to the search engine or the chat engine in accord with the routing decision.