Parallel Change Event Processing in Multi-Tenant Database Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database management systems face performance issues when tracking and disseminating database changes, as they often require serial processing of large volumes of change events, which can delay analysis and impact system performance.
Innovation Solution
The implementation of parallel processing technology allows a server to access and store change events from multiple logs in a persistent data store separate from the database, processing them out of order to create an ordered set of events, thereby reducing the load on the database management system and enabling efficient change tracking and dissemination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If serial processing of change events is used, then system simplicity is maintained, but processing speed and productivity deteriorate
Solution Approach 1:
The patent divides the single change event processing stream into multiple parallel streams, each handling a portion of the change events. Multiple processing threads or processes simultaneously handle different segments of the log data, transforming a sequential operation into parallel operations that increase throughput without proportionally increasing system complexity.
Solution Approach 2:
The patent introduces parallelism as a new dimension to the processing architecture by utilizing multiple CPU cores or processing threads simultaneously. This dimensional expansion from single-threaded to multi-threaded processing enables the system to handle larger volumes of change events without linearly increasing processing time.
2Productivity
If change tracking is performed within the database management system, then integration is simple, but system performance and scalability worsen
Solution Approach 1:
The patent extracts the change event processing functionality from the core database management system and places it in a separate, dedicated processing component. This externalization allows the database to focus on its primary functions while the extracted change tracking module handles event processing independently, preventing performance degradation of the core database operations.
Solution Approach 2:
The patent introduces an intermediary component that sits between the database and the change tracking processing. This intermediary captures change events from the database and forwards them to parallel processing threads, acting as a buffer that decouples the database operations from the intensive processing requirements of change event analysis.
3Reliability
If all change events are processed in order, then data consistency is maintained, but processing time and delay increase
Solution Approach 1:
The patent performs preliminary actions by pre-processing and buffering change events before final ordering and delivery. Change events are initially captured and stored in a buffer structure that allows for parallel processing while maintaining the ability to restore chronological order if needed, enabling time-sensitive operations to proceed without waiting for complete sequential processing.
Solution Approach 2:
The patent implements a dynamic processing approach where the strict ordering requirement is applied selectively based on the specific needs of different operations. For operations that require consistency, events are ordered; for operations that can tolerate out-of-order processing, events are handled immediately, allowing the system to adapt its processing strictness to the operational context.
Data Source
AI summary
Technology for tracking changes to a database and providing the changes to a subscribing computing device. In an illustrative implementation, a method may involve subscribing a computing device to changes of a portion of a multi-tenant database and accessing a plurality of change events stored in a log of the multi-tenant database in parallel. The method may further involve storing the plurality of change events in a persistent data store separate from the multi-tenant database and processing the plurality of change events to create an ordered set of change events. The ordered set of change events may then be provided to the computing device that subscribed to the changes.