Distributed Metadata Journaling for Scalable Flash Storage Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face challenges in efficiently managing metadata in distributed storage environments, particularly in handling load imbalances and failures, which affect performance and scalability.
Innovation Solution
A distributed storage system with a metadata journaling mechanism that utilizes a virtual file system (VFS) architecture, where metadata is spread across multiple servers, employing a bucket-based organization with distributed failure-resilient address spaces (DFRAS) and journal linked lists to manage metadata operations, allowing for load distribution and failure recovery without coordination among nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If metadata is centralized in conventional storage systems, then coordination and management are simplified, but performance and scalability deteriorate due to load imbalances and single points of failure
Solution Approach 1:
The patent segments metadata management by organizing it into distributed metadata journals across multiple nodes, with each node maintaining independent journals for specific data ranges. This segmentation eliminates single points of failure and improves scalability while maintaining manageable complexity through modular organization.
Solution Approach 2:
The patent introduces a new dimensional organization for metadata using distributed journals with sequence numbers and range-based allocation. Instead of centralized hierarchical management, metadata is distributed across multiple dimensions (nodes, journals, sequences, ranges), enabling parallel access and improved performance without proportionally increasing management complexity.
2Reliability
If metadata is distributed across multiple nodes, then scalability and fault tolerance improve, but coordination overhead and system complexity increase
Solution Approach 1:
The patent applies preliminary action by pre-allocating metadata journal ranges to specific nodes and pre-establishing journal structures with sequence numbers. This preparation eliminates the need for complex runtime coordination when failures occur, as the system can independently recover using pre-configured journal information without requiring complex inter-node communication.
Solution Approach 2:
Each node in the distributed system performs self-service for metadata management by maintaining its own journals independently and recovering from failures using its local journal information. This self-service approach eliminates the need for complex centralized coordination while maintaining high reliability through independent failure recovery capabilities.
3Reliability
If traditional journaling is used in distributed systems, then data integrity is maintained, but write amplification increases due to redundant metadata writes
Solution Approach 1:
The patent extracts the journaling function from traditional centralized file system metadata structures and implements it as independent distributed metadata journals. Each journal maintains only the specific metadata entries it is responsible for, eliminating redundant writes that occur in traditional systems where all metadata changes must be logged across the entire metadata structure.
Solution Approach 2:
The patent applies local quality by making each metadata journal specialized for specific data ranges and operations. Each journal is optimized for its local responsibility area, maintaining only the metadata entries relevant to its assigned range. This specialization reduces write amplification by eliminating the need to update unrelated metadata entries during write operations.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operably coupled to one or more of a plurality of storage devices. Each computing device is operable to compress one or more blocks of data and append a journal in front of the data. The journal and the data are written concurrently to flash memory. Each computing device is also operable to maintain a metadata registry that records changes in the flash memory. In the event of a power failure, the journal and previous journals may be used to verify the state of the metadata registry.