Split Query Timeout Control for Uneven Data Lake Response Times
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Queries executed in data lakes often fail or timeout due to varying response times across different data storage devices, leading to inefficiencies and failures.
Innovation Solution
A method for delayed split execution of queries, where subqueries are classified based on expected response times, and the timeout threshold is adjusted accordingly, allowing for staggered execution to accommodate slower devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a default timeout threshold is used for query execution, then the system maintains simple timeout control, but queries fail or timeout due to varying response times across different data storage devices
Solution Approach 1:
The patent segments the query execution process by dividing the query into subqueries corresponding to different storage devices, and applies individual timeout values to each subquery based on the specific characteristics of each device. This segmentation allows the system to handle varying response times across devices without a single rigid timeout threshold, thereby improving query execution success rate while maintaining manageable complexity through structured organization.
Solution Approach 2:
The patent implements dynamic timeout adjustment by setting different timeout values for different subqueries based on expected response times of specific storage devices. Instead of using a static default timeout threshold for all queries, the system dynamically adapts timeout parameters to match the actual performance characteristics of each storage device, resolving the contradiction between reliability and complexity.
2Reliability
If the timeout threshold is increased to accommodate slower devices, then query failures due to timeout are reduced, but the system loses responsiveness and efficiency for faster devices
Solution Approach 1:
The patent applies local quality by assigning different timeout values to different subqueries based on the specific characteristics of each storage device. Fast storage devices receive smaller timeout values to maintain responsiveness, while slower devices receive larger timeout values to prevent premature failures. This localized customization of timeout parameters resolves the contradiction between ensuring query completion and minimizing response time.
3Productivity
If queries are executed simultaneously across all storage devices, then the system achieves parallel processing efficiency, but queries fail due to mismatched response times
Solution Approach 1:
The patent performs preliminary action by evaluating the expected response time of each storage device before executing the query. Based on these pre-assessed characteristics, the system assigns appropriate timeout values to each subquery and orchestrates their execution. This preliminary preparation ensures that simultaneous parallel execution of queries across multiple storage devices can proceed without timeout failures, maintaining both high productivity and reliability.
Data Source
AI summary
Aspects include processing the query from a client device to divide the query into a plurality of subqueries, identifying a data source that corresponds to each of the plurality of subqueries, and identifying an expected response time for each of the data sources. Based on a determination that at least one of the expected response times is greater than a default timeout threshold, aspects include classifying the query as a delayed split query and updating a timeout value of the query to be a delayed timeout threshold, which is greater than the default timeout threshold. Aspects also include performing the delayed split execution of the query.