Distributed Inode Pools for Metadata Bottleneck Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Most file systems lack features for supporting mixed types of storage and face metadata bottlenecks that limit performance scaling in multi-core and distributed systems, leading to inefficiencies in storage and recovery from system failures.
Innovation Solution
A file system is divided into pools of inodes (vdisks) with each inode pool serving a separate processor, allowing for independent operation and reduced inter-processor communication, and data is stored in multiple segments with different classes of service, enabling automated policies to choose the appropriate segment for files or file portions dynamically.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a file system is implemented as a single centralized structure, then it is easier to manage and implement, but it creates metadata bottlenecks that limit performance scaling in multi-core and distributed systems
Solution Approach 1:
The file system is divided into multiple independent inode pools (vdisks), each served by a separate processor. This segmentation eliminates the metadata bottleneck by distributing file system operations across multiple processors, allowing each processor to independently manage its own inode pool without contention for shared metadata structures.
Solution Approach 2:
The patent introduces a new dimensional organization by creating a multi-dimensional file system structure where data can be accessed through multiple independent pathways. Each inode pool operates in its own address space and can be managed independently, adding a dimension of parallelism that enables scalability across multiple cores and distributed systems.
2Productivity
If all data is stored in a single storage pool, then storage management is simpler, but it reduces storage efficiency and ability to support mixed types of storage with different service requirements
Solution Approach 1:
Different segments within the file system are assigned different classes of service based on their specific requirements. Hot data can be stored in segments with faster access characteristics while cold data uses slower, more capacity-oriented storage. Each segment can be independently optimized for its specific workload characteristics.
Solution Approach 2:
The file system structure provides universal support for multiple storage types and service classes within a single unified interface. The same file system can simultaneously support high-performance SSD storage for frequently accessed files and high-capacity HDD storage for archival data, with automated policies determining optimal placement.
3Reliability
If the entire file system is checked for consistency after a failure, then consistency is ensured, but it requires extensive computation and IO operations even when only small portions are affected
Solution Approach 1:
The file system is divided into independent inode pools that can be checked and repaired independently. When an inconsistency is detected, the system can isolate and repair only the affected inode pool rather than checking the entire file system, dramatically reducing recovery time and computational overhead.
Solution Approach 2:
The patent introduces a persistent operations table that acts as an intermediary to track and manage consistency across inode pools. This table records operations and their states, allowing the system to quickly identify which specific pools require consistency checking rather than performing a blanket check on all pools.
4Productivity
If multiple processors share a common file system metadata structure, then implementation is simpler, but inter-processor communication overhead increases and limits scaling
Solution Approach 1:
The file system metadata is segmented into separate inode pools, each owned and managed by a specific processor. This eliminates the need for complex inter-processor coordination and locking mechanisms, as each processor operates independently on its own metadata structures. Inter-processor communication is minimized to only those operations that explicitly require cross-pool coordination.
Data Source
AI summary
A storage system includes a plurality of vdisks, with each vdisk containing a plurality of storage segments, and each segment providing a specific class of service (CoS) for storage. Each vdisk stores files with data and meta data distributed among its storage segments. A storage system includes a memory having multiple classes of service. The system includes an interface for storing a file as blocks of data associated with a class of service in the memory. The interface chooses the class of service for a block on a block by block basis for storage. A file system for storing a file. A file system for storing includes a plurality of vdisks. A method for storing a file.