Distributed Data Flow Model Resilient Edge Computing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed data flow models for edge computing suffer from high latency, increased complexity, and limited scalability due to reliance on centralized Message Queuing Telemetry Transport (MQTT) message brokers, and they fail to detect root causes of failures and recover from link failures, leading to operational losses.
Innovation Solution
A computing system that manages a distributed data flow model by establishing TCP connections and publisher-subscriber relationships between runtime nodes without a central broker, using a JSON file format for configuration, and implementing bridge wires to detect connectivity losses and determine root causes, enabling autonomous recovery and resiliency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a centralized MQTT message broker is used for data exchange between compute nodes, then message routing and delivery are simplified, but system latency increases and scalability is limited
Solution Approach 1:
The patent extracts the centralized message broker from the system architecture and replaces it with direct peer-to-peer TCP connections between runtime nodes. This eliminates the intermediate messaging layer that caused latency, allowing compute nodes to communicate directly without routing messages through a central broker. The extraction principle resolves the contradiction by removing the component that simplified routing but degraded performance.
Solution Approach 2:
Runtime nodes are equipped with embedded Pub/Sub clients that enable them to autonomously establish TCP connections and manage message routing without external broker assistance. Each node self-manages its connections to flow neighbors and handles message delivery independently, eliminating the need for centralized coordination and reducing system latency while maintaining routing functionality.
2Adaptability or versatility
If a centralized broker architecture is implemented, then message routing is simplified, but system scalability is reduced
Solution Approach 1:
The patent segments the centralized broker functionality into distributed components embedded in each runtime node. Instead of one central entity managing all routing, each node independently manages its own connections and message routing to its flow neighbors. This segmentation enables the system to scale horizontally by adding nodes without requiring proportional increases in central broker capacity or complexity.
Solution Approach 2:
The architecture transitions from a vertical hierarchy (centralized broker at top, nodes below) to a horizontal peer-to-peer network where all nodes exist at the same level. This dimensional change allows any node to connect directly with any other node through TCP connections, enabling linear scalability without the bottlenecks inherent in centralized architectures.
3Loss of time
If TCP connections are established between all flow neighbors for peer-to-peer messaging, then latency is reduced, but connection management complexity increases
Solution Approach 1:
Each runtime node autonomously manages its own TCP connections to flow neighbors using embedded Pub/Sub clients. Nodes automatically establish, maintain, and tear down connections as needed without external coordination. This self-service approach distributes connection management complexity across all nodes rather than concentrating it in a central broker, enabling direct communication that reduces latency while the complexity is handled locally by each node's client library.
4Reliability
If centralized broker is used for message exchange, then system setup is simplified, but failover and resiliency are reduced
Solution Approach 1:
The patent segments the single point of failure (centralized broker) into multiple independent connection paths between nodes. Each runtime node maintains direct TCP connections with its flow neighbors, creating redundant communication paths. If one connection or node fails, messages can be routed through alternative paths, and the Pub/Sub mechanism automatically reconnects when nodes become available again, providing failover without requiring complex centralized failure detection and recovery systems.
Data Source
AI summary
A system and method for implementing and managing a distributed data flow model is disclosed. The method includes obtaining a flow configuration file and identifying one or more socket roles and a unique identification number. The method includes establishing a TCP connection of one or more runtime nodes with the one or more flow neighbors, establishing a publisher-subscriber relationship of the one or more runtime nodes with the one or more flow neighbors, and implementing one or more bridge wires with the one or more runtime nodes and the one or more flow neighbors. Furthermore, the method includes detecting a loss of connectivity of one or more networks, determining one or more root causes of the loss of connectivity, and perform one or more operations to attain a predefined level of resiliency of the distributed data flow model.