Page-Based Metadata System for Distributed Filesystems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed filesystems face challenges in maintaining globally accessible metadata efficiently, as existing solutions either incur high costs during updates or scanning operations, and rely on centralized resources, leading to latency issues.
Innovation Solution
A page-based metadata system where metadata inodes are grouped into pages, mapped, and stored as a manifest page, allowing efficient updates and scanning by using inode and page identifiers, reducing reliance on centralized resources and minimizing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If metadata is stored in a centralized manner, then metadata management is simplified, but access latency increases and single points of failure are created
Solution Approach 1:
The patent segments metadata into discrete inodes that can be independently stored, updated, and accessed across distributed storage devices. Each inode represents a separate metadata unit that can be managed autonomously, eliminating the need for centralized metadata storage while reducing access latency through distributed access patterns.
2Reliability
If metadata is updated frequently in distributed systems, then data consistency is maintained, but update costs and network overhead increase
Solution Approach 1:
The patent extracts metadata from data blocks and stores it separately as independent inodes. This separation allows metadata to be updated independently without requiring data block modifications, reducing update costs and network overhead while maintaining data consistency through targeted metadata updates rather than full data synchronization.
Solution Approach 2:
The system performs preliminary actions by pre-allocating and pre-storing metadata inodes in the distributed filesystem before data operations occur. This allows metadata to be ready for immediate access and update during data operations, reducing the overhead and cost of metadata updates during active data processing.
3Loss of information
If all metadata is scanned for operations, then complete data access is ensured, but scanning time and computational cost increase
Solution Approach 1:
By segmenting metadata into discrete inodes, the system can scan only the specific inode relevant to the data operation rather than scanning all metadata. This targeted approach ensures complete data access through precise inode lookup while dramatically reducing scanning time and computational cost compared to full metadata scans.
4Quantity of substance
If distributed storage devices are added to the network, then storage capacity and redundancy increase, but metadata management complexity increases
Solution Approach 1:
The patent creates a universal inode structure that functions consistently across all distributed storage devices regardless of their individual characteristics. This standardized inode format allows metadata to be managed uniformly across the distributed network, enabling storage capacity and redundancy to increase through device addition without proportionally increasing metadata management complexity.
Data Source
AI summary
Methods and systems are described for storing metadata in a network of distributed remote storage devices. An example computer implemented method includes identifying a plurality of metadata inodes, grouping the plurality of metadata inodes into one or more metadata pages, mapping the plurality of metadata inodes and the plurality of metadata pages, and storing the mapping as a manifest page, and storing data to and retrieving data from the distributed remote storage devices using the plurality of metadata inodes.