LSM Metadata Indices for Space-Efficient Change Journaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems require additional storage space and resources to maintain change journal entries, leading to duplicate I/O operations and inefficiencies in managing metadata updates.
Innovation Solution
Implementing a log structure merge (LSM) tree-based metadata index with multiple indexing layers, where lower level indices are merged into higher levels as updates occur, allowing for a space-efficient change journal that tracks metadata changes chronologically.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If change journal entries are stored separately from the metadata index, then chronological tracking of changes is achieved, but storage space is duplicated and I/O operations are increased
Solution Approach 1:
The patent merges the change journal functionality with the LSM tree metadata index by identifying specific indices within the LSM tree as change journal entries. Instead of maintaining a separate change journal, the system utilizes the inherent structure of the LSM tree indices to track changes chronologically, thereby eliminating duplicate storage while preserving change tracking capability
Solution Approach 2:
The LSM tree indices serve dual purposes: they function as both the metadata index for efficient data retrieval and as the change journal for tracking modifications chronologically. This multi-functionality eliminates the need for separate change journal storage while maintaining both indexing and change tracking capabilities
2Reliability
If change journal entries are stored separately from the metadata index, then chronological tracking of changes is achieved, but I/O operations are duplicated
Solution Approach 1:
The patent combines change journal operations with metadata index operations by identifying LSM tree indices as change journal entries. This merging eliminates duplicate I/O operations since the same data structure serves both indexing and change logging purposes, improving overall system productivity
Solution Approach 2:
The LSM tree indices perform multiple functions simultaneously: they enable efficient metadata retrieval and chronologically track changes. This multi-functionality reduces the total number of I/O operations required, as a single index structure fulfills both roles without requiring separate read/write operations for change journaling
3Ease of operation
If traditional file systems use hard disk drives to store information and create index, then data can be indexed and searched, but storage space is substantially expensive
Solution Approach 1:
The patent segments the metadata index into multiple levels within the LSM tree structure, organizing indices hierarchically. This segmentation allows efficient searching and indexing operations while optimizing storage utilization by avoiding redundant data storage across separate structures
Solution Approach 2:
The patent merges the change journal functionality with the LSM tree metadata index, eliminating the need for separate change journal storage. This consolidation reduces total storage space requirements while maintaining both indexing and chronological change tracking capabilities
Data Source
AI summary
A space-efficient change journal for a storage system is provided. The storage system includes a memory having computer readable instructions stored therein. The system further includes a processor configured to access a log structure merge (LSM) tree-based metadata index having metadata for the storage system. The LSM tree-based metadata index includes indices placed in a plurality of indexing layers and one or more indices are merged within the indexing layers in response to updates to metadata, or as a background task. The processor is configured to identify one or more indices of the LSM tree-based metadata index as entries of a change journal of the storage system. The processor is further configured to maintain the change journal of the storage system based upon the identified entries.