Event Stream Processing Cluster Manager for Distributed Sensor Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deploying analytic tools and models in the Cloud that meet the requirements of large-scale, geographically distributed, and highly dynamic sensors is challenging, especially in real-time data processing from sensors on vehicles, aircraft, or watercraft, where immediate reactions are needed for events detected in real-time.
Innovation Solution
A system that manages a cluster of event stream processing engines, creating local and remote ESP models to process event block objects, with a router configuration to route these objects across computing devices, enabling seamless data processing and analysis across distributed systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a centralized cloud-based event stream processing system is used to handle large-scale sensor data, then processing capacity and analytical power are improved, but system complexity and deployment difficulty increase
Solution Approach 1:
The system divides the event stream processing functionality into distributed processing nodes that can independently handle specific data streams. Each node contains event stream processing engines that can be individually deployed and managed, reducing overall system complexity while maintaining high processing capacity through parallel operations across multiple segments.
Solution Approach 2:
The patent introduces a hierarchical dimension to the system architecture, with local edge devices performing initial processing and a cloud-based cluster manager coordinating distributed processing nodes. This multi-dimensional approach allows the system to scale processing capacity vertically (more nodes) and horizontally (deeper hierarchy) without proportionally increasing operational complexity.
2Productivity
If multiple distributed processing nodes are deployed to handle geographically distributed sensor data, then data processing efficiency is improved, but network connectivity requirements and system coordination complexity increase
Solution Approach 1:
The patent merges the functionality of multiple distributed processing nodes into a unified cluster managed by a central cluster manager. This allows individual nodes to operate independently for local data processing while the cluster manager provides centralized coordination, resource allocation, and fault tolerance, reducing the complexity of managing distributed systems without sacrificing processing efficiency.
Solution Approach 2:
The cluster manager acts as an intermediary between the cloud infrastructure and distributed processing nodes. It handles node registration, task distribution, data routing, and coordination, thereby simplifying the complexity of direct peer-to-peer node communication while maintaining high data processing efficiency through optimized task allocation.
3Loss of time
If real-time processing is implemented to enable immediate reactions to sensor events, then response time is improved, but computational resource consumption increases
Solution Approach 1:
The system performs preliminary filtering and preprocessing of sensor data at the edge devices before data leaves the local environment. Event stream processing engines pre-process incoming data streams, identifying and extracting only relevant events for further analysis. This preliminary action reduces the volume of data requiring intensive real-time processing, thereby maintaining fast response times while reducing overall computational resource consumption.
Solution Approach 2:
The patent implements selective real-time processing where only critical event streams receive full real-time processing resources, while less time-sensitive data undergoes batch processing or delayed analysis. This partial application of real-time processing to only necessary data streams maintains response times for critical operations without proportionally increasing computational resource consumption across all data streams.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A first computing device manages a cluster of event stream processing (ESP) engines (ESPEs). A local ESP model is created based on information read from a manager configuration file that includes first connection information to connect to the second computing device and second connection information to connect the third computing device. An ESPE is instantiated on the first computing device based on the created local ESP model. The event block object is received from the second computing device in a first source window of the instantiated ESPE. A remote ESP model is deployed to a remote third computing device. The manager configuration file includes an indicator of the remote ESP model. The third computing device to receive the processed event block object is selected. The processed event block object is published to a second source window defined by the remote ESP model deployed to the third computing device.