Dual-Path Data Retrieval System for Scalable Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As data volumes in databases grow, the time required to retrieve relevant data increases, hindering the usefulness of databases for online use, especially in non-relational databases that require scalability and distribution across multiple computer systems.
Innovation Solution
A system comprising client computing devices, retrieval servers, search engine servers, data access servers, and distributed data stores, where search engine software and data access software work in tandem to process queries, with search engine software optimizing instant data retrieval and data access software handling batch processing for large data quantities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in a distributed database system to achieve scalability, then the database can handle large amounts of data, but the time required to retrieve relevant data increases
Solution Approach 1:
The system segments data retrieval into two distinct pathways: a fast path for immediate results using search engine software, and a batch processing path for comprehensive results using data access software. This segmentation allows the system to handle large data volumes while providing both rapid and complete retrieval options.
Solution Approach 2:
The system performs preliminary actions by pre-processing and indexing data in the distributed database system before queries are executed. This includes organizing data in formats optimized for both fast search engine queries and comprehensive data access software queries, reducing retrieval time when queries are executed.
2Speed
If search engine software is used for fast data retrieval, then online responsiveness is improved, but the ability to process large quantities of data is limited
Solution Approach 1:
The system introduces an intermediary layer that coordinates between search engine software and data access software. This intermediary manages query routing, result aggregation, and resource allocation, allowing the system to leverage both the speed of search engine software and the capacity of data access software for processing large data quantities.
Solution Approach 2:
The system merges the capabilities of search engine software (fast retrieval) and data access software (batch processing) into a unified architecture. Queries can be routed to appropriate software based on requirements, and results from both systems can be combined to provide comprehensive answers that leverage both speed and capacity.
3Quantity of substance
If data access software handles batch processing for large data quantities, then processing capacity is improved, but responsiveness for instant retrieval is reduced
Solution Approach 1:
The system dynamically selects between batch processing and fast retrieval pathways based on query characteristics, data size, and system state. This dynamic approach allows the system to optimize for either capacity or speed depending on the specific requirements of each query, rather than being constrained to a single fixed mode of operation.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method, performed by one or more processors, including: receiving a user query; deriving, based on the user query, a search engine query for use with search engine software and a data access query for use with data access software; sending, to the search engine software, the search engine query to cause the search engine software to locate first data stored in a search engine index; sending, to the data access software, the data access query to cause the data access software to locate second data stored in a distributed data store; receiving, from the search engine software, the first data; and receiving, from the data access software, the second data.