Workflow Event Processing With Watermark-Based Batch Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems struggle to efficiently process continuous data streams due to a lack of processing flexibility, especially in handling real-time event processing needs for applications like sensor data and network monitoring, where traditional SQL queries are inadequate for unbounded data sets.
Innovation Solution
An event processing system that updates a database in real-time by receiving events from a continuous stream, combining events with a shared key attribute, and generating a single message to update the database, using techniques like User Defined Aggregation Functions and in-memory stitching to handle thousands of events per second.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional SQL queries are used to process continuous event streams, then data can be queried from stored databases, but the system cannot efficiently handle real-time processing of unbounded data streams
Solution Approach 1:
The system segments the continuous event stream into manageable batches using watermarks that define time intervals. Events are processed in discrete batches rather than continuously, allowing the system to handle unbounded streams while maintaining real-time processing capability. Each batch is processed independently through the staging area, enabling parallel processing and improving throughput.
Solution Approach 2:
The system performs preliminary actions by pre-processing events in the staging area before they reach the database. Events are buffered, sorted, and prepared in advance using watermarks that define processing boundaries. This preliminary organization of data allows for efficient batch processing and reduces the overhead of continuous database operations.
2Reliability
If events are processed individually and sent to the database immediately, then real-time updates can be achieved, but optimistic locking faults occur and system performance degrades
Solution Approach 1:
The system merges multiple individual event processing operations into a single batch update operation. Events that arrive during the same watermark interval are combined and processed together as a group, reducing the number of separate database transactions. This batching approach eliminates optimistic locking faults while maintaining real-time update reliability and improving overall processing throughput.
Solution Approach 2:
The system maintains continuous processing by keeping events in the staging area buffered between watermark intervals. Rather than stopping to process each event individually, the system continuously accumulates events and processes them in continuous batches, ensuring uninterrupted flow and eliminating the stop-start nature of individual transactions that causes locking issues.
3Productivity
If a single database update message is sent for thousands of events, then database performance improves, but the complexity of processing and combining events increases
Solution Approach 1:
The system introduces a staging area as an intermediary between event sources and the database. This intermediate buffer receives, buffers, and organizes events before they are sent to the database. The staging area handles the complexity of event combination and preparation, while the database receives simplified batch update messages. This intermediary layer isolates the complexity from the database operations.
Solution Approach 2:
The system uses periodic watermark intervals to trigger batch processing. Rather than processing events continuously or individually, the system processes events in periodic batches defined by watermark time intervals. This periodic action simplifies the processing logic by creating clear boundaries between batches, making it easier to manage and scale the event processing system.
Data Source
AI summary
Event processing techniques for updating a database in real time based on events in a continuous event stream are disclosed. The techniques can update the database to incorporate information from thousands of received events per second. The events can include metrics measuring milestones for an organizational process defined by a user. Moreover, multiple streams can include metrics from many tenants concurrently. The techniques include receiving, from a first user device, information identifying a group identifier for a first action object. The techniques then include assigning the identifier to the first action object and to at least one other action object. The techniques then include transmitting, to a service provider, data identifying the assignment of the group identifier, receiving second information identifying events processed by the service provider, identifying events corresponding to the identifier, and generating a user interface configured to present elements corresponding to the identified events.