Journaling Reliability via Solid-State De-duplication Hash Keys
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
File systems face reliability issues due to failures like power outages, leading to inconsistent states and data loss, and existing journaling mechanisms trade off between performance and reliability, while also being disruptive during consistency checks.
Innovation Solution
The method employs reliable asynchronous solid-state device-based de-duplication by computing a hash key for write data, looking it up in a de-duplication table, and writing a journal transaction with the hash key, which reduces the need for storing full data blocks and improves recovery from metadata and data corruption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If every transaction is forced to commit to disk before returning to the initiator, then reliability is improved, but disk I/O increases and performance decreases
Solution Approach 1:
The patent pre-computes and stores hash keys of data blocks in the de-duplication table before journaling is needed. When a write operation occurs, the hash key lookup determines if data already exists, allowing the system to journal only metadata references rather than full data blocks. This preliminary organization enables faster commit operations while maintaining reliability.
Solution Approach 2:
Instead of copying full data blocks to the journal, the patent stores hash keys that reference existing data blocks. The journal contains metadata and hash keys rather than redundant data copies, significantly reducing the amount of data written to disk while maintaining the ability to recover from failures.
2Productivity
If data is written to the journal at regular intervals, then performance is improved, but reliability decreases because some transactions may not be on disk when a fault occurs
Solution Approach 1:
The patent introduces a de-duplication table as an intermediary data structure that maps hash keys to physical data block locations. This table acts as a reference index that allows the journal to store compact metadata references instead of full data blocks, reducing journal write size and improving performance while maintaining transaction integrity through hash key verification.
3Reliability
If the file system performs consistency checks, then reliability is improved, but the file system becomes unavailable during the check
Solution Approach 1:
The patent maintains a de-duplication table that continuously tracks data block locations and hash keys during normal operations. This preliminary maintenance of accurate metadata structures eliminates the need for extensive consistency checks, as the journal and file system state are kept synchronized through ongoing hash key tracking rather than periodic disruptive checks.
4Reliability
If de-duplication is performed synchronously, then reliability is improved, but performance overhead increases
Solution Approach 1:
The patent pre-computes hash keys for incoming data blocks before the write operation completes and stores them in the de-duplication table. This preliminary hashing allows the journal to reference existing blocks by hash key rather than copying data, reducing I/O overhead while maintaining data integrity through hash verification.
Solution Approach 2:
The de-duplication table is updated continuously during normal write operations rather than being checked synchronously during each journal commit. This continuous maintenance of the de-duplication structure allows hash key lookups to be performed efficiently without blocking write operations, maintaining both performance and reliability.
Data Source
AI summary
A mechanism is provided in a data processing system for reliable asynchronous solid-state device based de-duplication. Responsive to receiving a write request to write data to the file system, the mechanism sends the write request to the file system, and in parallel, computes a hash key for the write data. The mechanism looks up the hash key in a de-duplication table. The de-duplication table is stored in a memory or a solid-state storage device. Responsive to the hash key not existing in the de-duplication table, the mechanism writes the write data to a storage device, writes a journal transaction comprising the hash key, and updates the de-duplication table to reference the write data in the storage device.