Delta Query Processing with Reduced Calculation Plans
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analysis systems inefficiently recalculate entire data flows when only parts of a table definition or underlying data change, leading to excessive server workload and operational costs.
Innovation Solution
Implement delta processing techniques that identify and execute only the subset of operations required to produce modified data elements by reusing cached results from previous executions, generating reduced calculation plans to minimize recalculation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire data transformation pipeline is re-executed when data changes, then the result data is updated to reflect changes, but the computational time and server workload increase significantly
Solution Approach 1:
The data transformation pipeline is segmented into multiple operations, each with its own dependency graph. When data changes, only the specific segment affected by the change is identified and re-executed, rather than the entire pipeline. This is achieved by tracking dependencies between operations and data elements, allowing selective re-execution of only the necessary portion of the pipeline.
Solution Approach 2:
The patent extracts and isolates the minimal subset of operations that must be re-executed when data changes. By analyzing the dependency graph, the system identifies and extracts only the necessary operations for recalculation, separating them from the rest of the pipeline that can continue using cached results.
2Reliability
If the entire data transformation pipeline is re-executed when data changes, then the result data is updated to reflect changes, but the server workload and operational costs increase
Solution Approach 1:
The pipeline is divided into independent operable segments with tracked dependencies. When data changes, the system identifies which segments need re-execution based on dependency analysis, allowing the server to allocate resources only to necessary segments rather than executing the entire pipeline.
Solution Approach 2:
Instead of executing the complete pipeline (excessive action), the system performs only the partial action required to address the data change. This is achieved through precise dependency tracking that identifies the minimal set of operations needed for recalculation.
3Reliability
If all operations in the pipeline are re-executed, then all result data is refreshed, but computational resources are wasted on unchanged data
Solution Approach 1:
The pipeline operations are segmented and assigned to different computational nodes. Dependency information is maintained between segments, enabling the system to identify which segments require energy for re-execution and which can use cached results, optimizing energy allocation.
Solution Approach 2:
The system maintains its own dependency tracking mechanisms and cached results, allowing it to autonomously determine which operations need re-execution. This self-service capability eliminates the need for full pipeline re-execution and optimizes energy consumption by leveraging existing computational results.
Data Source
AI summary
Embodiments of the present disclosure include techniques for processing analytic data using delta processing. In one embodiment, result data is generated from source data using a data pipeline and calculation plans. The result data may be modified and the plan may be re-executed to produce the modified result data. A reduced calculation plan is generated including only a portion of the original calculation plan to reduce the computational resources needed to execute the data processing. In some embodiments, data cached by operations in the pipeline are reused so the entire pipeline does not require re-execution to produce the modified result data.