Real-Time Compute Kernel With External State Store Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing real-time computation solutions tightly couple state storage and stream processing, lacking flexibility and elasticity, leading to inefficiencies in handling load spikes and variations in stream processing workloads.
Innovation Solution
A cloud-managed state store, such as DynamoDB, is used to replace built-in state management, allowing for easier scaling and optimizing state size by storing user state as a single key-value record and employing optimized storage formats for event-based aggregations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If state storage and stream processing are tightly coupled within the engine, then data locality and processing efficiency are improved, but flexibility and elasticity are reduced, making it difficult to scale during load spikes
Solution Approach 1:
The system segments state management from stream processing by introducing an external state store that is separately managed. The stream processing engine communicates with the external state store through well-defined APIs, allowing independent scaling and management of state and compute resources. This segmentation resolves the contradiction by enabling both efficient processing (through optimized state access patterns) and flexibility (through independent state store scaling).
Solution Approach 2:
The state management functionality is extracted from the stream processing engine and placed in an external state store. This extraction allows the state store to be independently scaled, managed, and optimized without being constrained by the processing engine's architecture. The engine retains only the necessary coordination logic, achieving both processing efficiency and system flexibility.
2Ease of operation
If built-in state management is used, then operational simplicity is maintained, but scaling capability and cost-effectiveness are reduced
Solution Approach 1:
A managed state store service acts as an intermediary between the stream processing engine and the underlying storage infrastructure. This intermediary abstracts away the complexity of state management operations while enabling scalable, cost-effective storage solutions. The engine interacts with the state store through simple, well-defined interfaces, maintaining operational simplicity while gaining scaling capabilities.
3Productivity
If state is distributed across processing nodes, then processing parallelism is improved, but state redistribution overhead increases during scaling operations
Solution Approach 1:
By extracting state management from the processing nodes and placing it in an external state store, the system eliminates the need for state redistribution during scaling operations. The external state store maintains centralized state that can be accessed by any processing node, removing the time-consuming state shuffling that occurs in distributed state models during scale-up or scale-down events.
4Adaptability or versatility
If cloud-managed state store is used, then scaling elasticity and cost-effectiveness are improved, but system complexity increases
Solution Approach 1:
The managed state store serves as an intermediary that handles the complexity of state management operations, including scaling, persistence, and recovery. This allows the stream processing engine to maintain simple, clean interfaces while leveraging the sophisticated capabilities of the external state store. The complexity is encapsulated in the state store service rather than the processing engine.
Data Source
AI summary
Methods and systems for using a cloud-managed state store are disclosed. A stream of data is received via a network. State information for a real-time computation workload is stored in a cloud-managed state store. The real-time computation workload is scaled out by utilizing the cloud-managed state store to retrieve state information. The stream of data is processed using a processing engine and utilizing the retrieved state information from the cloud-managed state store. Results of processing the stream of data are stored in the cloud-managed state store.