Edge Execution Graphs for Zero-Downtime Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Developers face issues with application updates in edge computing environments, including application downtime, latency, scalability challenges, managing diverse data sources and destinations, and vulnerabilities from reloading user-defined dynamically linked libraries.
Innovation Solution
Implementing a programmable connector that constructs an execution graph for data processing using modules in a portable binary code format, allowing updates to target specific modules without recompiling the entire application, and using an update monitor to manage module changes without interrupting data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire application is recompiled when one module is modified, then the application can be updated, but application downtime and cold-start latency increase
Solution Approach 1:
The patent divides the application into independent, modular components that can be compiled and executed separately. Each module is packaged as an individual artifact with its own execution context, allowing selective updates without recompiling the entire application. This segmentation enables hot-swapping of individual modules while the rest of the application continues running, eliminating downtime and cold-start latency.
2Adaptability or versatility
If the underlying infrastructure is rebooted to deploy updates, then module changes can be applied, but application downtime and latency increase
Solution Approach 1:
The patent implements dynamic module loading and unloading capabilities that allow the application to adapt to module changes without static infrastructure reconfiguration. Modules can be added, removed, or updated at runtime through hot-swapping mechanisms, enabling the system to dynamically adjust its composition without rebooting the underlying infrastructure, thus maintaining continuous operation.
3Reliability
If modules are hardcoded into the application in a particular code format, then the application can execute, but the application becomes inflexible to changes
Solution Approach 1:
The patent creates a universal module execution framework that can load and execute modules in various formats without requiring them to be hardcoded in a specific code format. The system uses a standardized interface and abstraction layer that allows modules to be swapped and executed dynamically, providing both execution stability through the standardized framework and flexibility through support for multiple module formats and hot-swapping capabilities.
Data Source
AI summary
The techniques described herein improve the management and execution of data processing in an edge computing environment using a programmable connector. The programmable connector defines a plurality of data sources, one or more data destinations, a plurality of operators, and a plurality of edges that are collectively useable to construct an execution graph. The execution graph provides a structure for the processing of data on a cluster of host servers in the edge computing environment. Stated alternatively, the execution graph models the flow of data from input data sources that are connected to the edge computing environment to output data destinations that are also connected to the edge computing environment. Accordingly, a node in the execution graph represents an operator and an edge in the execution graph connects two nodes.


