Virtual Machine Daemon Service for Continuous Data Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In systems where data is streamed to organizations, the failure of virtual machines can lead to data loss and disruptions, causing delays in detecting and recovering from such failures, which can result in missed performance standards and service level agreements.
Innovation Solution
A cluster of virtual machines with daemon services that monitor dependency services and transfer ownership of data streaming jobs to a central work distribution service upon connectivity loss, allowing seamless handoff to other nodes without data duplication or overlap, ensuring continuous data streaming.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a central service monitors virtual machine failures by sending inquiries, then failure detection is possible, but detection delay occurs in large systems with numerous data streams and virtual machines
Solution Approach 1:
The system divides the monitoring function into two segments: a lightweight local agent on each virtual machine that autonomously detects its own status, and a central service that receives notifications. This segmentation eliminates the need for the central service to actively query each virtual machine, reducing detection delay while maintaining reliable failure detection across large systems.
Solution Approach 2:
Each virtual machine is pre-equipped with a monitoring agent that continuously checks the status of dependency services and prepares to notify the central service immediately upon detecting a failure. This preliminary setup ensures that failure detection occurs instantaneously without waiting for central service inquiries, resolving the contradiction between reliable detection and minimal delay.
2Reliability
If virtual machine failure is detected and another virtual machine takes over the data stream, then continuous data flow is maintained, but data loss risk exists during the transition period
Solution Approach 1:
The system pre-establishes a queue at the central service that can immediately accept and buffer data streams when a virtual machine fails. The takeover mechanism is pre-configured to redirect streams to the queue or alternative virtual machines without interruption, eliminating the transition gap that causes data loss and ensuring continuous data flow.
Solution Approach 2:
The central service maintains a buffer queue that acts as a cushion during virtual machine failures. When a failure occurs, data streams are redirected to this pre-prepared buffer, preventing data loss during the transition period while the system recovers or redistributes the stream, thus protecting against substance loss during the handover process.
3Speed
If daemon services are implemented on each node to monitor dependency services, then failure detection speed increases, but system complexity increases
Solution Approach 1:
Each virtual machine is equipped with a self-monitoring daemon service that autonomously checks the status of its own dependency services and automatically notifies the central service of failures. This self-service approach enables fast failure detection without requiring complex centralized monitoring infrastructure, as each node independently monitors itself and reports status changes.
Solution Approach 2:
The daemon service implements multiple functions within a single component: it monitors dependency service health, detects failures, communicates with the central service, and triggers local failure response actions. This multi-functionality reduces the need for separate specialized components, thereby increasing detection speed while minimizing the increase in system complexity.
Data Source
AI summary
Embodiments regard transfer of data streaming services to provide continuous data flow. An embodiment of an apparatus includes a processor to process data for streaming to one or more organizations; and a memory to store data for streaming to the one or more organizations, wherein the apparatus is to provide a centralized work distribution service to track status of each of a plurality of data streams to the one or more organizations, and a plurality of nodes, each node being a virtual machine to stream one or more data streams to the one or more organizations, each node including a first daemon service to monitor connectivity of the node to dependency services for the node and, upon detecting a loss of connection to one or more of the dependency services, the node to discontinue ownership of the one or more data streams of the node and a second daemon service to poll the centralized work distribution service for data streams that are not assigned.