Log-Structured Merge Tree Data Deduplication and Archive Truncation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data backup systems face inefficiencies due to fragmented files resulting from frequent backups, leading to resource-intensive processes and slow access times, as well as limitations in sparse file operations across different file systems.
Innovation Solution
The implementation of a system using log-structured merge (LSM) trees to manage data blocks by storing hash values, segment identifiers, and physical addresses, allowing for efficient deduplication, block referencing, and truncation of unused blocks to maintain a compact data archive.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If sparsing operations are used to remove unused sectors from fragmented files, then storage space is reduced, but the file system must support sparsing operations and transfer operations fill unused regions with zeroes
Solution Approach 1:
The patent extracts the index metadata (hash values, segment identifiers, physical addresses) from the fragmented file structure and stores it in a separate LSM tree data structure. This separation allows the actual data blocks to be stored contiguously in the data archive while the index handles the logical addressing, eliminating the need for sparsing operations and their associated limitations.
Solution Approach 2:
The LSM tree acts as an intermediary layer between the logical file structure and the physical data storage. It maintains mappings from hash values to physical addresses, enabling efficient access to data blocks without requiring the file system to support sparsing operations. This intermediary structure resolves the conflict between storage efficiency and file system compatibility.
2Reliability
If frequent backups are performed to prevent data loss, then data protection is improved, but the backup process becomes resource intensive and time consuming
Solution Approach 1:
The patent segments the backup data into fixed-size data blocks and organizes them in a data archive. Each block is independently addressed through the LSM tree index, allowing incremental backups to only process and store changed blocks rather than performing full backups. This segmentation enables frequent backups with reduced resource consumption.
Solution Approach 2:
The system performs preliminary hashing of data blocks to create unique identifiers before storage. The LSM tree pre-organizes these hash values with their corresponding physical addresses, enabling rapid identification and retrieval of data blocks during backup and recovery operations. This preliminary organization significantly speeds up frequent backup processes.
3Productivity
If data blocks are appended to the end of the data archive during incremental backups, then new data is added efficiently, but unused sectors increase and access speed decreases
Solution Approach 1:
The patent replaces the traditional file system mechanical structure (with contiguous file requirements and sparsing operations) with a log-structured merge tree data structure. The LSM tree provides logical addressing through hash values and segment identifiers, while physical data blocks are stored sequentially in the archive. This substitution eliminates the trade-off between append efficiency and access speed.
Solution Approach 2:
The patent introduces a new dimension of organization by separating the logical indexing structure (LSM tree in memory) from the physical data storage (data archive on disk). This dimensional separation allows efficient sequential appending to the archive while maintaining fast access through the in-memory index, resolving the contradiction between backup efficiency and access speed.
Data Source
AI summary
A system and method is provided for data storage using log-structured merge (LSM) trees. An example method includes storing data blocks for a number of files backup data files in a data archive, storing data values and corresponding segment identifiers for each data block in a first LSM tree, and storing the segment identifiers and associated physical addresses for each data block in a second LSM tree. The method further includes determining that one or more data blocks is no longer referenced by the backup data file and storing, in a third LSM tree, the physical address of the unused space in the data archive. Finally, the method includes copying data blocks from the end of the data archive to unused memory space at the physical address in the data archive and then deleting these data block at the end position of the archive to truncate the archive.