Multipart Transaction Log Upload to Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In object-based storage systems, multipart uploading of transaction logs from multiple nodes for disaster recovery is inefficient due to duplicate entries and mixed log entries from different tenants and data sources, making de-duplication and efficient updating challenging.
Innovation Solution
A method that splits transaction logs into log entries, groups them by data source, writes them to separate blocks, identifies sequences, and uploads only unique blocks to object-based storage, enabling de-duplication and efficient updating by processing and uploading transaction logs from multiple nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all transaction logs from multiple nodes are uploaded to object-based storage, then disaster recovery completeness is improved, but storage redundancy increases due to duplicate entries
Solution Approach 1:
The transaction logs are segmented into distinct blocks, with each block containing log entries from a specific time range or data source. This segmentation enables identification and elimination of duplicate blocks across multiple nodes, reducing storage redundancy while preserving all unique transaction data for complete disaster recovery
Solution Approach 2:
Before uploading transaction logs to object-based storage, the system performs preliminary processing to split logs into blocks, identify duplicates, and filter out redundant entries. This preliminary de-duplication action prevents unnecessary data from being uploaded, reducing storage redundancy while ensuring all unique transaction logs are preserved for disaster recovery
2Reliability
If transaction logs are split into multiple parts for upload, then upload reliability is improved, but processing complexity increases
Solution Approach 1:
Transaction logs are divided into smaller blocks that can be uploaded independently to object-based storage. This segmentation improves upload reliability by allowing partial uploads and retries of failed blocks without requiring re-upload of entire logs. The systematic blocking approach manages complexity through structured organization rather than ad-hoc processing
Solution Approach 2:
The system implements feedback mechanisms to track which blocks have been successfully uploaded and which require retry. This feedback loop manages the complexity of multipart uploads by providing clear state information, enabling automated retry logic and progress tracking without requiring complex manual intervention
3Quantity of substance
If log entries from different data sources are grouped together, then storage efficiency is improved, but de-duplication difficulty increases
Solution Approach 1:
Log entries are organized into blocks that group entries from different data sources together in a structured format. Each block contains metadata identifying its source and content, enabling efficient storage while maintaining the ability to identify and detect duplicates through systematic comparison of block contents and metadata
Solution Approach 2:
The system uses metadata tagging and identification markers (analogous to color coding) to distinguish log entries from different data sources within grouped blocks. These identifiers enable efficient de-duplication by providing quick comparison points, allowing the system to detect duplicates even when entries from different sources are grouped together, thus maintaining both storage efficiency and de-duplication capability
Data Source
AI summary
A system may include a memory and a processor in communication with the memory configured to perform operations. The may operations include obtaining transaction logs in blocks from nodes of a data storage system. The operations may include, for each transaction log, splitting the transaction log into log entries, grouping log entries into groups associated with a same data source, and writing the log entries of the groups to empty blocks such that log entries from different groups do not share a same block. The operations may include identifying a same sequence of log entries from the written transaction logs and uploading first blocks of a first transaction log, including the same sequence of log entries, to an object-based storage without uploading second blocks of a second transaction log including the same sequence of log entries to the object-based storage.