Intermediate-State Incremental SQL Execution for Repetitive Compute
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing incremental computation methods in big data systems fail to leverage the concept of state storage, leading to inefficient repetitive computation and increased resource consumption, particularly in operations involving aggregate, window, and join operators.
Innovation Solution
A general incremental computation method based on intermediate state storage, which involves selecting and persistently storing stateful operators, matching intermediate states with current execution plans, and maintaining these states through real-time monitoring and updating to avoid repetitive computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If incremental computation is performed without state storage, then the system can operate with simpler architecture, but computational efficiency deteriorates due to repetitive computation
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing intermediate results of stateful operators (aggregate, window, join) in an intermediate state storage system. These pre-computed states are saved and reused in subsequent incremental computation tasks, avoiding the need to recompute the same intermediate results and thereby significantly improving computational efficiency.
Solution Approach 2:
The patent creates and maintains copies of intermediate computation states in a dedicated storage system. These state copies are preserved across computation cycles and can be rapidly retrieved and applied to new data increments, eliminating the need to regenerate intermediate results and reducing overall computational overhead.
2Speed
If intermediate states are persisted for all operators, then computation speed improves, but storage resource consumption increases
Solution Approach 1:
The patent applies local quality by selectively persisting intermediate states only for stateful operators (aggregate, window, join) rather than all operators. This targeted approach stores intermediate results only where they provide computational benefit, optimizing the balance between execution speed improvement and storage resource consumption by avoiding unnecessary storage of states from stateless operators.
Solution Approach 2:
The patent dynamically adjusts the persistence parameter for different operators based on their statefulness characteristics. Stateful operators have their intermediate states persisted while stateless operators do not, creating a parameter-based differentiation strategy that optimizes both performance and resource usage by changing the persistence parameter according to operator type.
3Measurement precision
If state matching is performed between intermediate states and current execution plans, then computational accuracy improves, but system complexity increases
Solution Approach 1:
The patent implements feedback through a state matching mechanism that compares intermediate states from previous computations with the current execution plan. This feedback loop identifies compatible states that can be reused, ensuring computational accuracy by verifying state compatibility while managing system complexity through structured matching algorithms.
Solution Approach 2:
The patent segments the execution plan into individual operators and their corresponding intermediate states, enabling granular matching and reuse identification. By breaking down the computation into separable operator stages, the system can accurately match and reuse specific intermediate states without requiring complex full-plan comparisons, thereby improving accuracy while controlling complexity.
Data Source
AI summary
The present invention discloses a general incremental computation method based on an intermediate state, wherein the general incremental computation method comprises: rewriting a logical execution plan to obtain an incremental execution plan corresponding to the logical execution plan, which comprises the following steps: S1: acquiring and parsing a user SQL to generate a logical execution plan; S2: screening and judging operators, and selecting a persisted target operator; S3: persistently storing the persisted target operator; S4: performing state matching between the logical execution plan in the intermediate state and a current execution plan through an algorithm so that all operators find equivalent operators corresponding thereto; S5: reading a last aggregation result of the incremental execution plan and merging the last aggregation result with an incremental aggregation result computed by current incremental data to acquire a final execution plan; and S6: monitoring, updating and maintaining the final execution plan in real time.