Complex Event Processing Project Movement Without Message Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing complex event processing systems face challenges in efficiently moving projects between nodes in a cluster without losing messages, as traditional methods either result in message loss or significantly degrade system performance due to the need for persistence and guarantee delivery protocols.
Innovation Solution
The system employs a cluster manager to coordinate project movement by enabling 'just in time' persistence and pausing input adapters, allowing messages to be buffered and ensuring seamless transfer between nodes without relying on message guarantee delivery protocols, thus maintaining performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If persistence is enabled and guarantee delivery protocol is used to prevent message loss during project movement, then message reliability is improved, but system performance deteriorates significantly
Solution Approach 1:
The system performs preliminary actions by pausing the input adapter before stopping the query processor, and by pre-buffering messages in the input adapter's message buffer. This ensures that no messages are lost during the project movement process, while avoiding the need for guarantee delivery protocols that would degrade performance.
Solution Approach 2:
The input adapter serves as an intermediary component that buffers messages between the message source and the query processor. During project movement, this buffer acts as a safety mechanism to hold messages that might otherwise be lost, eliminating the need for complex guarantee delivery protocols.
2Reliability
If a project is stopped in the source node before movement to maintain message integrity, then message loss is prevented, but productivity deteriorates due to service interruption
Solution Approach 1:
The system performs preliminary actions by pausing the input adapter before stopping the query processor. This sequence ensures that no new messages are accepted while the project is being moved, maintaining message integrity without requiring a complete service shutdown.
Solution Approach 2:
The system dynamically adjusts the operational state of different components during project movement. The input adapter is paused while the query processor is stopped, and the output adapter is restarted before the query processor. This dynamic state management maintains message integrity while minimizing service interruption.
3Reliability
If persistence is enabled to buffer transit stream messages and query engine status, then message loss during movement is prevented, but system complexity increases
Solution Approach 1:
The system extracts the persistence requirement from the overall system configuration and applies it only to the input adapter's message buffer during project movement. This selective approach prevents message loss without requiring full system-wide persistence, thereby reducing complexity.
Solution Approach 2:
The system applies persistence locally to the input adapter's message buffer rather than globally across the entire system. This localized persistence mechanism is sufficient to prevent message loss during project movement while avoiding the complexity of system-wide persistence configuration.
Data Source
AI summary
System, method, computer program product embodiments and combinations and sub-combinations thereof for moving a project in a complex event processing cluster are provided. In an embodiment, project movement in a complex event processing cluster initiates once project persistence capability exists. A handoff of the project from a source node to a destination node in the complex event processing cluster is coordinated by a manager without losing messages and without relying on a message guarantee delivery protocol.