File System Partitioning for Memory Footprint Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Scaling file system capacity is challenging, particularly for de-duplicated systems, due to high memory requirements, which are expensive and energy-consuming, necessitating a reduction in memory footprint.
Innovation Solution
The file system is partitioned into multiple partitions that share system memory, with metadata allowing access to associated data, and a scheduling algorithm that swaps partitions in and out of memory based on IO requests to optimize memory usage and reduce swapping costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If de-duplicated file system maintains metadata in memory for every block of physical data, then data access efficiency is improved, but memory footprint increases significantly
Solution Approach 1:
The patent divides the file system into multiple partitions, each with its own metadata structures in memory. This segmentation allows the system to load only the metadata for active partitions into memory, rather than maintaining metadata for the entire file system, thereby reducing memory footprint while preserving access efficiency for active data.
Solution Approach 2:
The patent implements dynamic partition swapping between memory and disk. Partitions are swapped in and out of memory based on access patterns and system needs, allowing the memory footprint to adapt dynamically rather than maintaining static metadata for all partitions simultaneously. This dynamic approach reduces average memory usage while maintaining efficient access to active partitions.
2Volume of stationary object
If file system capacity is scaled up, then storage capability is improved, but memory requirements increase proportionally
Solution Approach 1:
By segmenting the file system into multiple partitions that can be independently managed and swapped, the system can scale storage capacity by adding more partitions to disk without proportionally increasing memory requirements. Only the metadata for currently active partitions needs to reside in memory.
Solution Approach 2:
The patent introduces a partition swapping mechanism that acts as an intermediary between disk storage and memory. This intermediary allows large-scale data to be stored on disk while maintaining efficient memory usage by selectively loading only necessary partition metadata into memory when needed.
3Quantity of substance
If partitions are swapped in and out of memory frequently, then memory utilization is optimized, but swapping costs increase
Solution Approach 1:
The patent implements periodic swapping schedules and caching strategies where frequently accessed partitions are kept in memory longer, and swapping operations are performed in batches during low-activity periods. This periodic approach amortizes swapping costs over time rather than incurring frequent individual swap overheads.
Solution Approach 2:
The system performs preliminary actions by pre-loading partition metadata into memory before it is needed and using caching mechanisms to retain recently accessed partitions. This preliminary action reduces the frequency of actual swapping operations, thereby amortizing swapping costs across multiple IO operations while maintaining optimized memory utilization.
Data Source
AI summary
According to one embodiment, a file system (FS) of a storage system is partitioned into a plurality of FS partitions, where each FS partition stores segments of data files. In response to a request for writing a file to the storage system, the file is stored in a first of the FS partitions that is selected based on a time attribute of the file, such that files having similar time attributes are stored in an identical FS partition.