Keyed Data Stream Processing with In-Memory State Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for processing data streams lack efficiency in handling and modifying applications, requiring multiple rounds of development and testing, limiting user involvement and increasing the time to implement changes requested by end-users.
Innovation Solution
A computer-implemented method and system that processes keyed data items in a stream by maintaining state for each unique value of a key, allowing independent processing of data items based on their key values, using an in-memory data grid to manage state and execute executable logic specified in a dataflow graph, enabling real-time processing and modification of data records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional rule-based processing is used where each record is processed independently, then the system architecture is simple, but the time to implement changes and user involvement is severely limited
Solution Approach 1:
The patent segments the data stream processing by introducing key-based grouping, where records with the same key value are processed together as a group rather than independently. This creates logical partitions in the data flow that enable stateful processing while maintaining manageable system complexity through structured organization.
Solution Approach 2:
The patent implements dynamic processing by maintaining state information across multiple records with the same key, allowing the system to adapt its processing behavior based on accumulated state. This enables flexible modification of processing logic without requiring complete system redesign, as state can be dynamically updated based on incoming records.
2Manufacturing precision
If multiple rounds of development and testing are performed, then processing accuracy is improved, but the time to implement changes increases significantly
Solution Approach 1:
The patent performs preliminary state accumulation during the data processing phase itself, rather than requiring separate testing phases. By maintaining and updating state information as records are processed, the system validates processing logic in real-time, reducing the need for multiple iterative development and testing cycles.
Solution Approach 2:
The patent enables continuous processing and validation by maintaining state across record boundaries. This allows processing accuracy to be maintained through continuous state-based validation rather than discrete testing rounds, eliminating idle time between development and testing iterations.
3Speed
If state is maintained for each unique key value, then real-time processing capability is improved, but memory requirements and system complexity increase
Solution Approach 1:
The patent applies local quality by maintaining state information specifically for each unique key value rather than globally for all records. This localized state management allows real-time processing for grouped records while optimizing memory usage by only storing necessary state information for active key groups, rather than maintaining universal state.
4Productivity
If independent record processing is used, then system simplicity is maintained, but productivity and efficiency of data stream processing decrease
Solution Approach 1:
The patent segments the data stream into key-based groups, allowing parallel processing of different key groups while maintaining the simplicity of independent processing within each group. This segmentation enables improved overall productivity through better resource utilization without significantly increasing individual processing unit complexity.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
A method implemented by a data processing system for processing data items of a stream of data items, including: accessing a specification that represents the executable logic, wherein a state of the specification for a particular value of the key specifies one or more portions of the executable logic that are executable in that state; receiving, over an input device or port, data items of a stream of data; for a first one of the data items of the stream, identifying a first state of the specification for a value of the key associated with that first one of the data items; processing, by the data processing system, the first one of the data items according to one or more portions of executable logic that are represented in the specification as being associated with the first state.