Distributed Database Log Archiving via Parallel Hash Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of creating backups of log files in a distributed database system is time-consuming due to the large number and size of log files, which limits the efficiency of archiving and restoration processes.
Innovation Solution
The archiving process is parallelized by distributing log records across multiple computing nodes, allowing for a higher degree of parallelization than the original distributed database, and using a time-invariant and idempotent hash function to map records to appropriate nodes for archiving and restoration, preserving the shard hierarchy and ensuring data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If log files are archived sequentially using traditional backup methods, then data integrity is maintained, but the archiving process takes a considerable length of time
Solution Approach 1:
The patent divides the archiving process into multiple parallel operations by creating multiple archive streams from different log files simultaneously. Each stream processes a portion of the log data independently, allowing concurrent writing to the archive storage. This segmentation enables the system to maintain data integrity through structured parallel processing while significantly reducing total archiving time compared to sequential methods.
Solution Approach 2:
The patent performs preliminary actions by first identifying and organizing log files into groups that can be processed in parallel, and by pre-establishing the archive stream structures before actual data transfer begins. This preparation phase enables subsequent parallel processing to proceed efficiently without coordination overhead during the critical data transfer phase, thus reducing overall archiving time while maintaining integrity.
2Productivity
If the number of computing nodes is increased to improve archiving speed, then productivity increases, but device complexity increases
Solution Approach 1:
The patent designs the archive stream computing nodes to perform multiple functions: they can simultaneously process multiple log file streams, handle data transformation, manage error correction, and support both archiving and restoration operations. This multi-functionality allows the system to achieve high archiving speeds with a moderate number of nodes rather than requiring a large specialized infrastructure, thereby increasing productivity without proportionally increasing device complexity.
3Loss of time
If parallelization is applied to the archiving process, then archiving time is reduced, but ensuring data integrity becomes more difficult
Solution Approach 1:
The patent implements feedback mechanisms where each archive stream continuously reports its processing status, data integrity checks, and error conditions to a central coordination system. This feedback enables real-time monitoring and validation of parallel processing operations, allowing the system to detect and correct issues while maintaining high-speed parallel archiving and restoration without compromising data integrity.
Solution Approach 2:
The patent incorporates error detection and correction capabilities, along with checksum validation, into the parallel archive stream processing before data is fully written or restored. By preparing these protective measures in advance within the parallel processing framework, the system cushions against potential data integrity issues that could arise from parallel operations, ensuring reliability while maintaining speed.
Data Source
AI summary
Transactional log data for a distributed database may be archived by a number of computing nodes scalable to a specified concurrency parameter. A log record may be mapped to a computing node for performance of an archiving operation. The log record may be stored with information indicative of a position, in a hierarchy of shards, of the shard from which the log record originated. The log record may be replayed in an order, relative to other records, that is based on the shard's position in the shard hierarchy. The log record may be replayed on a shard identified based on a time-invariant and idempotent hash function.