Event-Based Aggregation Scaling with External State Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing real-time computation solutions tightly couple state storage and stream processing, lacking flexibility and elasticity, leading to inefficiencies in handling load spikes and variations in stream processing workloads.
Innovation Solution
A cloud-managed state store, such as DynamoDB, is used to replace built-in state management, allowing for easier scaling and optimizing state size by storing user state as a single key-value record and employing optimized storage formats for event-based aggregations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If state storage and stream processing are tightly coupled within the engine, then data locality and processing efficiency are improved, but flexibility and elasticity are reduced, making it difficult to scale during load spikes
Solution Approach 1:
The system segments state storage from stream processing by introducing an external state store that is separate from the processing engine. This allows the processing engine to focus on computation while state management is handled independently, enabling flexible scaling of each component based on workload requirements.
Solution Approach 2:
The state management functionality is extracted from the processing engine and placed in an external state store. This extraction enables the system to scale state storage independently from computation resources, providing the needed elasticity to handle load spikes without redistributing state partitions.
2Productivity
If state partitions are redistributed to scale out computation during load spikes, then processing capacity is increased, but operational complexity and downtime are increased
Solution Approach 1:
By extracting state management from the processing engine into an external state store, the system eliminates the need for state redistribution when scaling. New processing nodes can access state directly from the external store without requiring complex redistribution operations, thereby reducing operational complexity and avoiding downtime.
Solution Approach 2:
The external state store acts as an intermediary between processing nodes and state data. This intermediary layer allows multiple processing nodes to access state concurrently without requiring coordination or redistribution, simplifying the scaling process and reducing operational complexity.
3Ease of operation
If built-in state management is used, then integration and ease of use are improved, but scaling flexibility and cost-effectiveness are reduced
Solution Approach 1:
State management is extracted as a separate, independently scalable service. This allows the system to maintain ease of use through integrated APIs while providing the scaling flexibility of a separate component that can be scaled independently based on workload demands.
Solution Approach 2:
The external state store serves as a universal service that can be shared across multiple processing engines and workloads. This multi-functional approach maintains ease of operation through standardized interfaces while enabling flexible scaling to serve different computational needs.
Data Source
AI summary
Methods and systems for scaling out real-time computations are disclosed. A stream of event data associated with a plurality of users is received. A set of event-based aggregations to be computed for the plurality of users is determined based on the received stream of event data. User information for the plurality of users is stored in a state store. Computation of the determined set of event-based aggregations is dynamically scaled out using the stored user information. The set of event-based aggregations is computed for the plurality of users from the received stream of event data using the dynamically scaled out computation. The computed event-based aggregations for the plurality of users are stored in the state store.