Data Processing Pipeline Failure Recovery via Compute Graph
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex data processing pipelines in electronic systems often obscure component failures, leading to inefficient re-execution of entire pipelines, wasting energy and time, and potentially delaying other tasks due to the diversity of components and computing environments.
Innovation Solution
A computer programming syntax is used to define dependencies between components, generating a compute graph that schedules components for execution efficiently, allowing for selective re-execution of failed components while preserving outputs from successfully executed upstream components and providing default outputs to downstream components.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire data processing pipeline is re-executed to recover from a component failure, then the failed component can be addressed, but significant energy and time are wasted re-executing successful components
Solution Approach 1:
The patent segments the data processing pipeline into individual components and tracks their execution status separately using a compute graph. When a failure occurs, only the failed component and its downstream dependencies are re-executed, while successful components are preserved. This segmentation allows selective recovery without re-executing the entire pipeline, directly reducing energy waste while maintaining reliability.
Solution Approach 2:
The system performs preliminary actions by executing components in a defined order through the compute graph and storing their outputs before failures occur. This allows the system to have intermediate results ready, so when a failure happens, only the necessary portion needs re-execution. The preliminary execution and output storage enable efficient recovery without redundant energy consumption.
2Reliability
If the entire data processing pipeline is re-executed to recover from a component failure, then the failed component can be addressed, but significant time is wasted re-executing successful components
Solution Approach 1:
The compute graph segments the pipeline into discrete components with tracked execution status. Upon failure, the system identifies and re-executes only the failed component and its downstream dependencies, rather than the entire pipeline. This segmentation dramatically reduces recovery time by eliminating unnecessary re-execution of successful components while ensuring reliable failure recovery.
Solution Approach 2:
The system implements feedback mechanisms by monitoring component execution status and using this information to determine recovery actions. The compute graph tracks which components have succeeded and which have failed, providing feedback that guides selective re-execution. This feedback-driven approach optimizes recovery time by avoiding re-execution of components that don't need it.
3Reliability
If the entire data processing pipeline is re-executed to recover from a component failure, then consistency can be maintained, but redundant compute cycles are generated
Solution Approach 1:
The compute graph segments the pipeline into trackable components, allowing the system to identify which segments need re-execution for consistency. Only failed components and their downstream dependencies are re-executed, maintaining data consistency without generating redundant compute cycles for successful components. This segmentation preserves productivity while ensuring reliability.
Solution Approach 2:
The system applies partial action by re-executing only the necessary portion of the pipeline (failed components and downstream dependencies) rather than the entire pipeline. This partial re-execution maintains data consistency where needed while avoiding excessive compute cycles on components that already executed successfully, thereby preserving productivity.
4Reliability
If the entire data processing pipeline is re-executed to recover from a component failure, then all components can be verified, but the complexity of tracking individual component status is increased
Solution Approach 1:
The patent merges the component dependency relationships and execution status tracking into a unified compute graph structure. This single structure serves multiple functions: tracking which components have executed, identifying dependencies, and determining recovery scope. By merging these functions into one structure, the system achieves accurate failure detection without proportionally increasing complexity.
Data Source
AI summary
Techniques are disclosed for re-executing a data processing pipeline following a failure of at least one of its components. The techniques may include a syntax for defining a compute graph associated with the data processing pipeline and receiving such a compute graph in association with a specific data processing pipeline. The technique may include executing the data processing pipeline, determining that a component of the data processing pipeline failed, and determining a portion of the data processing pipeline to execute/re-execute based at least in part on dependencies defined by the data processing pipeline in association with the failed component. Re-executing the one or more components may comprise retrieving an output saved in association with a component upon which the failed component depends.