Parallel Database Log Replay via Object Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database recovery methods using single-threaded execution can lead to significant delays, especially in OLTP systems with numerous smaller transactions, as they replay logs synchronously, resulting in inefficient data recovery processes.
Innovation Solution
Implementing parallel worker queues to execute log entries, where entries associated with the same object are processed serially by a single queue and those with different object IDs are executed in parallel across multiple queues, utilizing synthetic log records for synchronization and asynchronous processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If single-threaded execution is used to replay log entries, then data consistency is maintained, but recovery time increases significantly
Solution Approach 1:
The patent segments the log replay process into multiple parallel worker queues, each handling a subset of log entries. Log entries are divided and distributed across multiple workers based on object ID hashing, allowing concurrent processing while maintaining consistency within each segment. This segmentation enables parallel execution without compromising data integrity.
Solution Approach 2:
The patent introduces intermediary synchronization mechanisms including synthetic log records, commit markers, and coordination structures that mediate between parallel workers. These intermediaries ensure that parallel operations maintain proper ordering and consistency, acting as coordinators that enable parallelism while preserving reliability constraints.
2Loss of time
If parallel worker queues are used to replay log entries, then recovery time is reduced, but system complexity increases
Solution Approach 1:
The system is segmented into independent worker queues that can operate autonomously, reducing the complexity of coordination. Each worker manages its own subset of log entries, and the segmentation allows for modular design where complexity is distributed rather than centralized, making the overall system more manageable.
Solution Approach 2:
The patent uses synthetic log records that are copies or representations of actual log entries, allowing parallel workers to process these copies without directly manipulating the original log data. This copying mechanism simplifies the complexity by creating isolated work units that can be processed independently.
3Productivity
If log entries are processed in parallel across multiple workers, then throughput increases, but ensuring serial execution for same object becomes more difficult
Solution Approach 1:
The patent applies local quality by ensuring that each worker queue maintains serial execution semantics for its assigned subset of log entries (local scope), while the overall system achieves parallel throughput (global scope). This local seriality within distributed parallel workers resolves the contradiction by making consistency enforcement local rather than global.
Solution Approach 2:
The system changes the parameter of worker assignment by using hash-based routing of log entries to workers. This parameter change (from sequential processing to hash-distributed parallel processing) enables throughput improvement while the hashing function provides a simple, deterministic mechanism to ensure that log entries for the same object are consistently routed to the same worker, maintaining serial execution without complex coordination.
Data Source
AI summary
A plurality of transactions are executed in a data storage application. As a result, log entries according to the plurality of transactions are stored in at least one log. During a data recovery operation, log entries from the log are dispatched into a plurality of worker queues. Each worker queue corresponds to a different one of a plurality of logical central processing units. The log entries are replayed using the plurality of worker queues such that log entries associated with a same object are executed serially by a single worker queue and at least a portion of the log entries associated with different objects are executed in parallel by different worker queues. Related apparatus, systems, techniques and articles are also described.