In-Memory Event Stitching for High-Throughput Database Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems struggle to efficiently process and manage continuous data streams from applications like sensor data and financial tickers, lacking the processing flexibility required for real-time event processing needs.
Innovation Solution
An event processing system that updates a database in real-time by receiving events from a continuous stream, combining records with shared key attributes, and generating a single message to update the database, using techniques like in-memory stitching and User Defined Aggregation Functions to handle large volumes of events efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional database systems process continuous data streams using standard SQL queries on finite data sets, then query execution is simple and reliable, but the system cannot handle real-time event processing requirements and generates significant latency
Solution Approach 1:
The system performs preliminary actions by buffering incoming events in memory before they are processed and written to the database. Events are accumulated in a staging area and processed in batches, allowing the database to be updated periodically rather than continuously, which reduces processing latency while maintaining throughput.
Solution Approach 2:
The patent merges multiple event processing operations into a single database update operation. By combining multiple individual insert/update operations into one batch operation, the system reduces the overhead of repeated database transactions and achieves higher throughput with lower latency.
2Reliability
If the system updates the database continuously for each incoming event, then data freshness is maintained, but optimistic locking faults occur and system reliability decreases
Solution Approach 1:
The system performs preliminary buffering of events in memory before database updates are executed. By preparing multiple events in advance and then executing them as a single atomic operation, the system avoids repeated optimistic locking attempts while maintaining data freshness and improving reliability.
Solution Approach 2:
Multiple database update operations are merged into a single atomic transaction. This combining approach ensures that either all events are successfully processed together or the entire batch is rolled back, eliminating optimistic locking faults while maintaining high throughput.
3Productivity
If the system processes events individually one by one, then processing simplicity is maintained, but the ability to handle large volumes of events per second is limited
Solution Approach 1:
The event processing system is segmented into distinct phases: event buffering phase, processing phase, and database update phase. Events are first collected in memory during the buffering phase, then processed together during the processing phase, and finally written to the database as a single atomic operation. This segmentation enables high throughput while keeping each individual phase simple.
Solution Approach 2:
The system performs preliminary buffering of events in memory before executing database operations. This preliminary action separates the high-speed event reception from the more careful database writing process, allowing the system to handle large volumes of events per second without increasing the complexity of individual processing steps.
4Adaptability or versatility
If traditional databases store and process data in tables with SQL queries, then data storage is efficient, but the system lacks temporal focus and cannot process continuous unbounded data streams
Solution Approach 1:
The system introduces dynamic buffering in memory that adapts to incoming event streams. Rather than static table storage, the system dynamically collects events in a staging area and processes them in batches, enabling it to handle continuous unbounded data streams while maintaining real-time processing speed through periodic batch operations.
Data Source
AI summary
Event processing techniques for updating a database in real time based on events in a continuous event stream are disclosed. The techniques can update the database to incorporate information from thousands of received events per second. The events can include metrics measuring milestones for an organizational process defined by a user. Moreover, multiple streams can include metrics from many tenants concurrently. The techniques include receiving a plurality of events corresponding to a time interval and identifying that two events share a value of a key attribute. The techniques then include causing to be generated an in-memory combined database record comprising a combination of first and second attribute information from the two events and the value of the key attribute. In response to a trigger signal, the techniques include transmitting a single message, based at least in part on the combined database record, to update the database.