Coordinating Spout for Dynamic Virtual Spout Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing event stream processing systems face inefficiencies in scaling, particularly when new tenants are added or existing data sources change, as they require redeployment of the entire topology, leading to potential service disruptions.
Innovation Solution
The implementation of a virtual spout system, where a coordinating spout instance manages dynamic creation and shutdown of virtual spout instances, allowing for on-demand scaling without disrupting services to other tenants, and supports dynamic changes in data sources by creating or removing spouts as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If separate spouts are created for each tenant at deployment time, then each spout can service its tenant's data sources, but the system requires complete redeployment when new tenants are added, causing service disruption
Solution Approach 1:
The system segments spout functionality by introducing a parent-child spout hierarchy. The parent spout is divided into multiple virtual child spouts, each serving a specific tenant. This segmentation allows individual child spouts to be dynamically created, started, stopped, or removed without affecting the parent spout or other child spouts, enabling tenant-specific scaling without complete redeployment.
Solution Approach 2:
The system transforms the static spout configuration into a dynamic structure where child spouts can be created, started, stopped, and removed at runtime based on tenant requirements. The parent spout dynamically manages the lifecycle of child spouts through configuration files and API calls, allowing the system to adapt to changing tenant needs without service disruption.
2Adaptability or versatility
If the topology is stopped and re-deployed to add new spouts, then new tenants can be serviced, but service disruption occurs for existing tenants
Solution Approach 1:
The system performs preliminary configuration setup by defining child spout configurations in a configuration file before runtime. When new tenants need to be added, the configuration file is updated and the parent spout dynamically creates and starts new child spouts without stopping the topology. This preliminary configuration approach enables seamless tenant addition while maintaining service availability for existing tenants.
3Device complexity
If a single spout is used, then the system structure is simple, but the spout can only communicate with a single data store and cannot service multiple tenants
Solution Approach 1:
The parent spout is designed as a universal component that can service multiple tenants through its child spouts. Each child spout is configured with specific parameters (tenant ID, data source credentials, etc.) to serve individual tenants. This multi-functional design allows a single parent spout instance to handle multiple data stores and tenants while maintaining a relatively simple overall architecture compared to having completely independent spout instances for each tenant.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
In one embodiment, a stream processing system ingests data from one or more data sources and converts the data into an unbounded stream for an event stream processor. The stream processing system may include a first coordinating spout instance that is deployed at a first time, the first coordinating spout instance to instantiate one or more second spout instances and deploy said one more second spout instances at a second time that is later than the first time; the one or more second spout instances to: connect to the one or more data sources, respectively, and ingest a respective portion of the data from its respective one of the one or more data sources; and output messages based on the ingested portions of the data; wherein the unbounded stream for the event stream processor is based on the messages. Other embodiments may be disclosed and/or claimed.