Asynchronous Queries on Secondary Data Cores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing systems, efficiently querying secondary data cores moved to secondary storage systems is challenging due to increased time-to-first-byte (TTFB) and query cancellation timeouts, especially as the volume of log data grows.
Innovation Solution
The technique involves processing asynchronous queries that recover secondary data cores in the data plane only when sufficient storage is available, allowing for parallel query execution and dynamic management by the client.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If secondary data cores are moved to secondary storage system to make space for new data cores, then storage capacity is improved, but time-to-first-byte (TTFB) increases
Solution Approach 1:
The system performs preliminary actions by pre-fetching and caching metadata about secondary data cores before they are actually needed for querying. This includes pre-loading core identifiers, location information, and accessibility status into the data plane, so that when a query arrives, the system can quickly determine whether the required data is immediately available or needs to be retrieved from secondary storage, thereby reducing the actual TTFB.
Solution Approach 2:
The system implements dynamic data core management where data cores can transition between different states (active in data plane, inactive in secondary storage, being recovered). The system dynamically adjusts which data cores reside in the data plane versus secondary storage based on query patterns, availability requirements, and storage capacity, optimizing the balance between storage capacity and access speed.
2Measurement precision
If secondary data cores are recovered in the data plane before searching, then query accuracy is improved, but query timeout rate increases
Solution Approach 1:
The system performs preliminary recovery of secondary data cores into the data plane before query execution when storage capacity permits. This ensures that when a query is issued, the required data is already available in the fast data plane, guaranteeing query accuracy while avoiding timeouts by having the data ready in advance.
Solution Approach 2:
The system introduces an intermediary recovery mechanism that acts as a buffer between secondary storage and query processing. When a query requests data from secondary storage, the system initiates a recovery process to transfer the data core to the data plane, and the query waits for this recovery to complete. This intermediary step ensures data availability while managing the trade-off between recovery time and query timeout.
3Ease of operation
If synchronous API calls are used for querying, then query simplicity is improved, but query success rate decreases
Solution Approach 1:
The system implements a dynamic query execution model that can switch between synchronous and asynchronous modes based on data core availability. When the required data core is already in the data plane, the system uses simple synchronous API calls. When the data core needs to be recovered from secondary storage, the system transitions to asynchronous query execution, allowing the client to continue with other operations while the query processes in the background, thereby maintaining both simplicity and success rate.
4Speed
If data plane storage capacity is increased to hold more data cores, then query speed is improved, but storage efficiency decreases
Solution Approach 1:
The system segments the storage architecture into two distinct layers: the data plane for frequently accessed data cores requiring fast query performance, and secondary storage for less frequently accessed data cores. This segmentation allows the system to maintain a smaller, more efficient data plane while still providing access to a larger total dataset through the secondary storage layer, optimizing both query speed and storage efficiency.
Data Source
AI summary
The present disclosure relates generally to techniques for processing asynchronous queries across multiple data cores including secondary data cores stored in the secondary storage system in a distributed computing system. Data from secondary data cores are recovered to new data cores generated in the data plane in order to be accessible for searching. Using this technique, asynchronous queries are run in parallel allowing the client to dynamically manage the queries and receive notifications when results are available.