Stream Processing Context Data Affinity Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Stream processing systems face high latency and data locking issues due to the need for external database access for context data, which hinders real-time processing and throughput in handling unbounded sequences of data.
Innovation Solution
Context data is partitioned across multiple computing nodes with fast operational memory, allowing each node to access and modify context data locally, reducing the need for external database calls and minimizing latency through data affinity routing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If context data is accessed from external databases, then data consistency and reliability are improved, but processing latency increases and throughput decreases
Solution Approach 1:
The patent segments context data into two categories: frequently accessed context data stored in operational memory on each computing node, and less frequently accessed context data stored in external databases. This segmentation allows the system to access hot data quickly while maintaining cold data in centralized storage, resolving the contradiction between fast access and data consistency.
Solution Approach 2:
The patent implements preliminary action by pre-loading context data into operational memory before it is needed for processing. Computing nodes maintain local copies of context data in fast memory, so when events require context information, the data is already available locally without requiring external database queries, thus eliminating latency.
2Quantity of substance
If context data is stored in external databases, then data storage capacity is improved, but access speed and processing throughput deteriorate
Solution Approach 1:
The patent applies local quality by giving each computing node local copies of context data in its operational memory, tailored to its specific processing needs. Each node maintains only the context data relevant to its assigned events and functional modules, optimizing local access speed while the external database maintains the complete data set for nodes that need it.
Solution Approach 2:
The patent introduces operational memory as an intermediary layer between external databases and event processing functional modules. This intermediary cache layer holds frequently accessed context data, mediating between the slow external storage and fast processing requirements, thus improving throughput without sacrificing storage capacity.
3Measurement precision
If context data is accessed externally, then data accuracy is improved, but data locking issues arise and real-time processing is hindered
Solution Approach 1:
The patent merges the context data storage and processing functions by co-locating context data and event processing functional modules within the same computing nodes. This merging eliminates the need for external database access for frequently used context data, allowing real-time processing while maintaining data accuracy through consistent local state management.
Solution Approach 2:
The patent implements self-service by enabling computing nodes to serve their own context data requirements from local operational memory without requiring external database queries. Each node independently manages its local context data, updating and maintaining it as needed, which eliminates data locking issues and enables true real-time processing.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for stream processing with contextual data affinity. One of the methods includes receiving an event at a computing node of a stream processing system that includes one or more computing nodes and data representing a topology of functional modules. At a particular functional module of the topology, an operation that requests access to context data related to the event is determined based on a data attribute of the event. The context data is partitioned among the one or more computing nodes. Context data related to the event is obtained from the computing node. The operation of the particular functional module is processed using the context data, and a modified event is generated. The modified event is provided to a subsequent functional module according to the topology of functional modules.