Stream Multiplexing for Deduplicating Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplicating storage systems are limited in the number of streams they can support due to allocated resources, which restricts their ability to handle a large number of file systems and namespaces efficiently.
Innovation Solution
The implementation of stream multiplexing, where a first number of streams are created for file system managers to access data blocks, with stream identifiers mapped to each block, allowing access through a smaller number of dedicated streams, thereby increasing stream capability without changing the number of streams.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a limited number of dedicated streams are allocated to access disks, then resource allocation per stream is ensured, but the number of supported file systems and namespaces is restricted
Solution Approach 1:
Multiple file system streams are merged and multiplexed onto a smaller number of dedicated storage streams. The stream multiplexer combines multiple input streams from different file systems and namespaces, mapping them to a limited number of dedicated storage streams, thereby allowing more file systems to be supported than the number of dedicated streams.
Solution Approach 2:
Dedicated storage streams are made universal by allowing them to serve multiple file systems and namespaces through the stream multiplexer. Instead of each stream being dedicated to a single file system, the same stream can handle multiple file systems by receiving multiplexed input from the stream multiplexer, increasing system versatility.
2Adaptability or versatility
If more streams are created to handle larger number of file systems, then stream capability increases, but resource allocation per stream decreases
Solution Approach 1:
The stream multiplexer merges multiple file system streams into a smaller number of dedicated storage streams, allowing the system to support more file systems without increasing the total number of dedicated streams. This consolidation maintains resource utilization efficiency while expanding stream capability.
Solution Approach 2:
The patent introduces a new dimension of stream management by creating virtual streams at the file system level while mapping them to physical dedicated streams at the storage level. This dimensional separation allows the system to present a large number of streams to file systems while actually using fewer physical streams for storage access.
3Speed
If dedicated streams are allocated for each file system, then access efficiency is improved, but the number of supported streams is limited
Solution Approach 1:
Multiple file system streams are merged and multiplexed onto dedicated storage streams, allowing the system to maintain dedicated stream access efficiency for multiple file systems simultaneously. The multiplexer manages the merging process to preserve access performance while supporting more file systems than the number of dedicated streams.
Data Source
AI summary
In one embodiment, a method includes creating a first number of streams for a file system manager of a deduplicating storage system to access concurrently a type of data blocks, where each stream is for one file system and is identified by a stream identifier. The method further includes mapping stream identifiers to each of the type of data blocks passing through the first number of streams. The method further includes accessing the type of data blocks in storage units of the deduplicating storage system through a second number of streams, where the second number of streams are dedicated to the type of data blocks in the deduplicating storage system, where the second number is smaller than the first number, where the data blocks are tracked according to the mapped stream identifiers, and where the data blocks are stored in the storage units after a deduplication process to remove duplication.