Stream Processing Context Data Affinity Partitioning

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Stream processing systems face high latency and inefficiencies in accessing context data due to reliance on external databases and inter-process communications, which hinder real-time processing and throughput.

Innovation Solution

The system partitions context data across multiple computing nodes with fast operational memory, allowing each node to store and quickly access relevant context data, reducing the need for external database calls and minimizing data locking issues by executing functional modules within the same operating system process.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If context data is stored in external databases, then data storage capacity is improved, but access latency increases

Engineering Contradiction:

Improvedata storage capacityVSAvoidaccess latency

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent segments context data into two categories: frequently accessed context data stored in operational memory of computing nodes, and less frequently accessed context data stored in external databases. This segmentation allows the system to optimize for both storage capacity and access latency by placing data in the most appropriate storage location based on access patterns.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary action by pre-loading frequently accessed context data into operational memory of computing nodes before it is needed for event processing. This advance preparation eliminates the need for real-time database access during event processing, thereby reducing access latency.

Inventive Principle:

Principle #10Preliminary action

2Adaptability or versatility

If context data is accessed via inter-process communication, then data sharing flexibility is improved, but processing speed decreases

Engineering Contradiction:

Improvedata sharing flexibilityVSAvoidprocessing speed

Core Design Contradiction:

Adaptability or versatilityVSSpeed

Solution Approach 1:

The patent merges the context data storage function with the event processing function by colocating frequently accessed context data in the operational memory of the same computing nodes that process events. This merging eliminates the need for inter-process communication when accessing context data, thereby improving processing speed while maintaining data sharing flexibility through the distributed architecture.

Inventive Principle:

Principle #5Merging (Combining)

3Speed

If context data is stored in operational memory of computing nodes, then access speed is improved, but system complexity increases

Engineering Contradiction:

Improveaccess speedVSAvoidsystem complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The system applies local quality by giving each computing node the specific quality of having frequently accessed context data locally stored in its operational memory. This local optimization improves access speed for each node without requiring all nodes to have all data, thereby managing system complexity through selective local storage rather than universal duplication.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system implements dynamics by allowing the distribution of context data across computing nodes to be dynamically adjusted based on access patterns and system conditions. The architecture can adapt to changing requirements by redistributing context data, balancing the trade-off between access speed and system complexity as needs evolve.

Inventive Principle:

Principle #15Dynamics

4Reliability

If external database calls are made for context data, then data consistency is improved, but throughput decreases

Engineering Contradiction:

Improvedata consistencyVSAvoidthroughput

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system applies partial action by making external database calls only for context data that is not cached in operational memory or when data consistency updates are required. For the majority of frequently accessed context data, the system uses the cached copies in operational memory, thereby maintaining high throughput while periodically ensuring data consistency through selective database access.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS9075670B1Stream processing with context data affinity

Publication Date: 2015.07.07 VMWARE INC

AI summary

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for stream processing with contextual data affinity. One of the methods includes receiving an event at a computing node of a stream processing system that includes one or more computing nodes and data representing a topology of functional modules. At a particular functional module of the topology, an operation that requests access to context data related to the event is determined based on a data attribute of the event. The context data is partitioned among the one or more computing nodes. Context data related to the event is obtained from the computing node. The operation of the particular functional module is processed using the context data, and a modified event is generated. The modified event is provided to a subsequent functional module according to the topology of functional modules.