Application-Specified Data Placement in Parallel Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage tiering techniques in parallel computing environments often result in poor performance due to inappropriate data placement policies, leading to inefficiencies in data storage and management, particularly in high-performance computing applications.
Innovation Solution
A method is introduced where a user or application can specify how and where files are stored in a multi-tier storage system, using storage daemons to execute commands such as plfs_protect, plfs_trim, plfs_query, and plfs_info to control data placement, replication, compression, and deletion based on file characteristics and access patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If automated tiering techniques are used to control data placement, then storage management is simplified, but data placement accuracy deteriorates leading to poor performance
Solution Approach 1:
The patent introduces a specification mechanism as an intermediary between the application and the storage system. The application provides placement specifications (e.g., preferred storage nodes, replication factors, access patterns) to a storage manager, which then uses these specifications to make informed data placement decisions. This intermediary layer combines automated storage management with application-specific knowledge, resolving the contradiction between ease of operation and placement accuracy.
2Extent of automation
If heuristic methods are used for automated tiering, then storage management becomes automatic, but performance deteriorates due to inappropriate data placement
Solution Approach 1:
The patent applies preliminary action by having the application specify data placement requirements in advance, before actual data storage occurs. The specification includes information about access patterns, preferred storage locations, and replication needs. The storage system then uses these pre-provided specifications to make accurate placement decisions, avoiding the need for heuristic guessing and thereby maintaining both automation and high performance.
3Quantity of substance
If distributed storage is used to increase capacity, then storage capacity is improved, but data management complexity increases
Solution Approach 1:
The patent creates a universal specification interface that works across distributed storage nodes. The same specification mechanism handles data placement, replication, and access pattern management across the entire distributed system. This multi-functional specification approach simplifies data management in distributed storage by providing a unified control mechanism that works regardless of the number or configuration of storage nodes.
Data Source
AI summary
Techniques are provided for storing files in a parallel computing system based on a user-specification. A plurality of files generated by a distributed application in a parallel computing system are stored by obtaining a specification from the distributed application indicating how the plurality of files should be stored; and storing one or more of the plurality of files in one or more storage nodes of a multi-tier storage system based on the specification. The plurality of files comprise a plurality of complete files and/or a plurality of sub-files. The specification can optionally be processed by a daemon executing on one or more nodes in a multi-tier storage system. The specification indicates how the plurality of files should be stored, for example, identifying one or more storage nodes where the plurality of files should be stored.