Multi-tier Storage File Relocation via Cloud Stubs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional storage tiering arrangements in information processing systems are inflexible and inefficient, particularly when transferring large files between storage tiers, leading to wasteful use of resources.
Innovation Solution
Implementing a multi-tier storage system that controllably relocates compressed and deduplicated files from a hidden data store of a file system to a cloud-based object store, using a cloud tiering appliance to manage data movement and generate stubs for user access, thereby optimizing data transfer between storage tiers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional storage tiering arrangements are used to transfer large files between storage tiers, then data can be moved between tiers, but the transfer process is inefficient and wasteful of storage system resources
Solution Approach 1:
The patent segments the file transfer process by separating the actual data content from its metadata. Only metadata (file descriptors, access patterns, permissions) is transferred between storage tiers, while the actual file data remains in place or is accessed via distributed file system protocols. This segmentation eliminates the need to physically move large amounts of data, dramatically improving transfer efficiency and reducing resource waste.
Solution Approach 2:
The patent introduces a distributed file system layer as an intermediary between the storage tiers and clients. This intermediary manages file access across tier boundaries, allowing files to be logically moved or accessed in cloud storage without physical data migration. The intermediary handles metadata management and coordinates access, enabling efficient tiering operations without the inefficiencies of conventional direct data transfer methods.
2Adaptability or versatility
If files are moved between storage tiers in conventional arrangements, then storage capacity can be optimized, but the process is unduly inflexible for parallel file systems
Solution Approach 1:
The patent makes the storage tiering system universal by designing it to work with any distributed file system protocol and any cloud storage interface. The tiering layer presents a unified interface to clients regardless of underlying storage location, and can manage files across multiple cloud providers or storage types. This multi-functionality provides flexibility without proportionally increasing complexity, as the same core mechanisms handle diverse storage scenarios.
Solution Approach 2:
The patent uses copying of metadata rather than physical data movement to achieve flexibility. File descriptors, access patterns, and metadata are copied and managed in the distributed file system layer, allowing virtual file movement between tiers without complex physical migration operations. This copying approach enables flexible tiering arrangements while keeping the actual data storage simple and manageable.
3Reliability
If large files are transferred between storage tiers using conventional methods, then data relocation can occur, but storage system resources are wasted during the transfer process
Solution Approach 1:
The patent extracts the essential file identification and access information (metadata) from the full file data, separating these components for the tiering operation. By taking out only the necessary metadata for managing file location and access across tiers, the system achieves reliable data relocation without consuming resources proportional to the full file size. The actual file data remains intact in its original location or is accessed remotely without physical transfer.
Data Source
AI summary
An apparatus in one embodiment comprises a storage system having at least first and second storage tiers each comprising a plurality of storage devices. The first storage tier implements a file system having a user visible name space and a corresponding hidden data store. The second storage tier implements a cloud-based object store. The storage system is configured to controllably relocate files from the hidden data store of the file system of the first storage tier to the cloud-based object store of the second storage tier. In conjunction with the relocation of a given one of the files, the given file is replaced in the hidden data store with a corresponding stub providing user access via the user visible name space to an object comprising the given file in the cloud-based object store of the second storage tier. The controllable relocation is illustratively performed in accordance with one or more configured policies.