Tiered File Storage System for Distributed Data Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed file systems face challenges in managing large numbers of small files efficiently, leading to high costs and inefficiencies in cloud storage, especially when dealing with cold data that is rarely accessed, and require solutions that balance cost, storage capacity, and data access speed.
Innovation Solution
A tiered file storage system that uses a file server in communication with remote third-party servers for policy-based automated tiering, allowing for the efficient offloading of cold data to a cloud-based object storage tier, maintaining hot and cold data separation while enabling cost-effective storage and rapid data access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If cold data is stored in third-party cloud object stores, then storage cost is reduced, but data access speed deteriorates
Solution Approach 1:
The patent segments data into hot data and cold data categories, storing hot data locally in the distributed file system and cold data remotely in third-party cloud object stores. This segmentation allows the system to optimize for both fast access (local hot data) and low cost (remote cold data), resolving the contradiction between storage cost and data access speed.
2Quantity of substance
If small files are stored in cloud object stores, then storage capacity is increased, but storage efficiency deteriorates due to transaction costs
Solution Approach 1:
The patent merges multiple small files into a single archive file before storing in the cloud object store. By combining many small files into one larger archive, the system reduces the number of expensive cloud transactions (uploads, downloads, deletions) while maintaining the ability to access individual files through the archive, thus improving storage efficiency while preserving storage capacity.
3Volume of stationary object
If cold data is offloaded to cloud storage, then on-premise storage resources are freed, but system complexity increases due to data management overhead
Solution Approach 1:
The patent implements automated tiering that allows the system to self-manage data placement between hot and cold storage tiers based on access patterns. The system automatically identifies cold data, archives it to cloud storage, and retrieves it when needed, without requiring manual intervention. This automation reduces the perceived complexity for users while still providing sophisticated data management capabilities.
4Loss of energy
If message streams are partially offloaded to cold tier, then storage cost is reduced, but access complexity increases due to distributed data locations
Solution Approach 1:
The patent introduces an intermediary archive file that acts as a container for cold data portions of message streams. The archive file provides a unified interface for accessing distributed cold data segments, shielding users from the complexity of data being split across multiple locations. The system automatically manages the segmentation and reassembly of message stream data, reducing access complexity while enabling cost-effective partial offloading.
Data Source
AI summary
A file server receives a request for data from a user device. The data is represented at the file server by a virtual cluster descriptor. The file server queries an identifier map using an identifier of the virtual cluster descriptor. Responsive to the identifier map indicating that the requested data is stored at a location remote from the file server, the file server accesses a cold tier translation table that stores a mapping between an identifier of each of a plurality of virtual cluster descriptors and a storage location of data associated with the respective virtual cluster descriptor. The cold tier translation table is queried using the identifier of the virtual cluster descriptor to identify a storage location of the requested data, and the data is loaded to the file server from the identified storage location.