Pass-through Write Policies for Distributed Storage Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The 'write recall full' policy in data networks is inefficient, as it can result in long processing times for small writes to large files, fails if primary storage is full, assumes increased file value for primary storage, and leads to version inconsistencies between primary and secondary storage.
Innovation Solution
Implementing 'pass-through write' policies that allow primary storage to perform read-write access to offline files in secondary storage without recalling them, including 'pass-through overwrite' and 'pass-through multi-version' policies, and a directory write pass-through policy for direct writing to secondary storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If write recall full policy is used to bring offline files online for client access, then read-write access performance is improved, but processing time increases and system resources are consumed
Solution Approach 1:
The patent segments the write operation into two independent paths: one for creating new files and another for modifying existing files. This segmentation allows the system to handle different write scenarios differently - creating new files directly in secondary storage while modifying existing files through the recall mechanism, thereby optimizing processing time and resource consumption.
Solution Approach 2:
The patent applies different quality characteristics to different parts of the storage system. Primary storage maintains high-speed access for frequently accessed files, while secondary storage provides cost-effective storage for less frequently accessed files. The pass-through mechanism allows write operations to bypass the slow secondary storage path when primary storage is available, optimizing the local quality of access for each storage tier.
2Ease of operation
If write recall full policy is used, then file access is enabled, but storage failures occur when primary storage is full
Solution Approach 1:
The patent extracts the file creation operation from the primary storage system and places it in the secondary storage system. When a client creates a new file, the system directly writes to secondary storage rather than requiring primary storage space, thereby eliminating the storage fullness constraint and improving reliability.
Solution Approach 2:
The patent introduces a pass-through mechanism as an intermediary between client write operations and storage systems. This intermediary allows write operations to be directed to secondary storage when primary storage is full, mediating the conflict between file access capability and storage capacity constraints.
3Productivity
If write recall full policy is used, then files are brought online, but version inconsistencies are created between primary and secondary storage
Solution Approach 1:
The patent inverts the traditional recall mechanism by allowing write operations to pass through directly to secondary storage without first recalling files to primary storage. This inversion eliminates the create-new-file scenario that causes version inconsistencies, as files are created directly in their final destination rather than being created in primary storage and then migrated.
Solution Approach 2:
The patent enables the storage system to self-manage version consistency by automatically directing write operations to the appropriate storage location. The pass-through mechanism ensures that files are created directly in secondary storage where they belong, eliminating the need for manual intervention to resolve version conflicts.
4Ease of operation
If write recall full policy is used, then client access is provided, but storage costs increase due to primary storage consumption
Solution Approach 1:
The patent segments write operations into two distinct paths: new file creation goes to secondary storage, while existing file modifications go through the recall mechanism to primary storage. This segmentation significantly reduces the quantity of data stored in primary storage, as only modified portions of existing files need to be recalled rather than creating new files in primary storage.
Data Source
AI summary
A hierarchical storage system includes file servers and a policy engine server. Offline attributes are added to file system inodes in a primary file server, file system parameters are added in the primary server, offline read and write access method fields are added to a connection database, and the primary file server uses these attributes and parameters for selecting a particular read method or write method for access to an offline file or section of an offline file. The write methods follow a “write recall full” policy, a “pass-through write” policy, a “pass-through multi-version” policy, or a “directory write pass-through” policy. The pass-through multi-version policy results in a new offline version of a file each time that a client opens and writes to a multi-version file. The directory write pass-through policy results in a new offline file when a file is created within a write pass-through directory.