Deduplicated Log Flushing With Unified Metadata Binding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication methods in data storage systems incur high latency and inefficiency due to separate transactions for committing duplicate and unique data, leading to redundant locking and contention for shared metadata pages, which can result in stale entries and delayed log flushing.
Innovation Solution
The solution involves partitioning the deduplication process into two transactions: a first transaction for updating reference counts of VLB entries and a second unified transaction for binding both unique and duplicate data pages to logical addresses, eliminating the separate duplicate data commit transaction and integrating its processing efficiently into other transactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If separate transactions are used for committing duplicate and unique data, then data consistency can be maintained, but latency increases and productivity decreases due to redundant locking and contention
Solution Approach 1:
The patent merges the duplicate data commit transaction and unique data commit transaction into a single unified transaction. This consolidation eliminates redundant locking operations and metadata page contention, allowing both duplicate and unique data to be committed simultaneously in one transactional unit, thereby reducing latency and improving log flushing efficiency while maintaining data consistency through the unified transaction's atomicity
2Reliability
If separate transactions are used for duplicate and unique data, then transactional integrity is preserved, but time consumption increases due to sequential processing
Solution Approach 1:
The patent combines separate duplicate data commit and unique data commit transactions into a single unified transaction that processes both types of data simultaneously. This merging eliminates sequential processing steps and redundant locking operations, reducing the total time required for deduplication while preserving transactional integrity through the unified transaction's atomic commitment mechanism
3Adaptability or versatility
If multiple transactions access shared metadata pages, then data updates can be performed, but locking contention increases and stale entries occur
Solution Approach 1:
The patent merges multiple transactions that access shared metadata pages into a single unified transaction. This consolidation reduces locking contention by eliminating multiple lock acquisition and release cycles, and prevents stale entries by ensuring all metadata updates are committed atomically together, thereby simplifying the locking mechanism while maintaining metadata update capability
Data Source
AI summary
Techniques can include: recording, in a log, log entries for write operations writing data pages to logical addresses; and flushing the log entries from the log, wherein flushing includes performing deduplication of the data pages. Flushing can include: partitioning the data pages into i) a first portion of the data pages that are determined to be duplicates of existing data pages, and ii) a second portion of the data pages determined to be unique data pages; atomically incrementing reference counts each corresponding to one of the existing data pages determined as identical to a data page of the first portion; and atomically binding the logical addresses to the data pages, including: binding the first portion of data pages to a corresponding third portion of the logical addresses; and binding the second portion of data pages to a corresponding fourth portion of the logical addresses.