Dataflow Life Cycles With Custom Pre And Post Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing dataflow frameworks lack the ability to incorporate customized operations and context information, making it difficult to manage and execute dataflows effectively, especially in complex business analytics scenarios.
Innovation Solution
The implementation of dataflow life cycles that associate customized codes, including pre-processing and post-processing codes, with context information, allowing for scheduled and executed operations within server computing systems, enhancing dataflow management and execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If customized operations and context information are incorporated into dataflows, then data processing capability and control are improved, but system complexity increases
Solution Approach 1:
The patent segments the dataflow execution into distinct phases: pre-processing phase (executing custom code before dataflow), main dataflow execution phase, and post-processing phase (executing custom code after dataflow). This segmentation allows customized operations to be integrated without overwhelming the system, as each phase can be managed independently with appropriate resource allocation and error handling.
Solution Approach 2:
The patent implements pre-processing custom code that executes before the main dataflow to prepare data, configure parameters, or set up required resources. This preliminary action ensures that the main dataflow receives properly prepared inputs, reducing the need for complex error handling and retry logic during execution, thereby managing system complexity while enhancing processing capability.
2Reliability
If pre-processing and post-processing codes are associated with dataflows, then execution control and information context are improved, but management complexity increases
Solution Approach 1:
The patent merges the custom code management with the existing dataflow framework by associating pre-processing and post-processing code directly with dataflow definitions. This integration allows the system to manage customized operations through the same interface and control mechanisms used for standard dataflows, reducing management complexity while maintaining execution control and reliability.
Solution Approach 2:
The patent implements feedback mechanisms where the system monitors the execution status of pre-processing and post-processing code, and uses this information to control the flow of data through the system. Execution results from custom code are fed back into the dataflow framework to determine whether to proceed, pause, or terminate subsequent processing steps, ensuring reliable execution control without requiring complex external management systems.
3Loss of information
If context information is associated with customized code, then dataflow execution informativeness is improved, but information processing overhead increases
Solution Approach 1:
The patent implements a universal context information structure that can be reused across multiple dataflows and custom code segments. By defining standardized context parameters and data structures that serve multiple purposes (tracking execution status, passing data between phases, storing configuration information), the system reduces information processing overhead while maintaining comprehensive execution informativeness across all dataflow operations.
Data Source
AI summary
System and methods for implementing dataflow life cycles are described and include forming, by a first server computing system, a dataflow life cycle by associating a dataflow with a customized code; associating, by the first server computing system, the customized code of the dataflow life cycle with context information, the customized code including one or more of pre-processing customized code and post-processing customized code; scheduling, by the first server computing system, the dataflow of the dataflow life cycle to be executed by a second server computing system when the customized code includes the pre-processing customized code and when the pre-processing customized code is successfully executed by the first server computing system; and executing, by the first server computing system, the post-processing customized code when the customized code includes the post-processing customized code and when the dataflow of the dataflow life cycle is successfully executed by the second server computing system.