Parallel Event Processing in Database Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face inefficiencies in processing and notifying events, particularly due to the differing processing times of in-memory and persistent events, which can lead to delays and resource mismanagement.
Innovation Solution
Implementing a parallel event processing architecture with multiple channels and slave processes, where in-memory events are processed separately from persistent events, and publishers select channels based on load, class, and activity to ensure efficient ordering and resource optimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If in-memory events and persistent events are processed in the same queue, then queue management is simplified, but processing speed and resource efficiency deteriorate due to the differing processing times of the two event types
Solution Approach 1:
The patent divides the single event queue into separate queues for in-memory events and persistent events. This segmentation allows each queue type to be processed by specialized slave processes optimized for their specific event type, resolving the contradiction by sacrificing some queue management complexity to gain significant processing speed improvements through parallel specialized processing.
Solution Approach 2:
The patent introduces a new dimension of event processing by creating parallel processing channels with different levels of persistence. Instead of a single-dimensional queue, it establishes multiple dimensions (in-memory vs. persistent) that can be processed simultaneously, allowing fast in-memory events to be handled quickly without being blocked by slower persistent events requiring disk I/O.
2Adaptability or versatility
If persistent events are processed alongside in-memory events, then all events are handled uniformly, but resource consumption increases due to the slower processing speed of persistent events
Solution Approach 1:
The patent segments event processing into distinct pathways: one for in-memory events and another for persistent events. This allows the system to maintain adaptability by handling all event types while optimizing resource usage by applying different processing strategies to different event segments, preventing high-resource persistent event processing from impacting overall system efficiency.
Solution Approach 2:
The patent applies local quality by assigning different processing characteristics to different event types. In-memory events receive fast, low-resource processing appropriate for their transient nature, while persistent events receive more resource-intensive processing suitable for their durability requirements. This localized optimization resolves the contradiction between uniform handling and resource efficiency.
3Device complexity
If a single process handles all event notifications, then process management is simpler, but notification delays occur due to the varying processing times of different event classes
Solution Approach 1:
The patent segments event notification handling into multiple specialized slave processes, each dedicated to specific event types or queue channels. This segmentation eliminates notification delays by allowing parallel processing of different event classes simultaneously, while the coordinator process maintains overall management simplicity by assigning and monitoring these specialized processes.
Solution Approach 2:
The patent transitions from a single-dimensional process model to a multi-dimensional architecture with coordinator and slave processes operating in parallel. This dimensional expansion allows the system to maintain simple coordinator-level management while achieving fast notifications through parallel specialized slave processes that can handle different event types simultaneously without mutual interference.
Data Source
AI summary
Techniques for processing events in parallel are provided. Multiple publishers publish events in parallel to a plurality of channels. Each channel is a queue that maintains a list of events that are awaiting to be processed by one or more slave processes. In one approach, all events that need to be persistent are published to a persistent channel, whereas all events that do not need to be persistent are published to one or more in-memory channels. In another approach, for each event, a publisher determines which channel of the plurality of channel to publish the event. The determination is based, at least in part, on the number of events in each channel and, optionally, the type of each event in each channel. In either approach, each event from a single publisher is published to the same channel.