Unified File Namespace for HPC Data Access Arbitration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High performance computing systems face inefficiencies in data storage, transfer, and processing due to the large amounts of data involved, leading to performance lag and inefficient use of network resources, particularly in managing sensitive and private information across disparate file systems.
Innovation Solution
A data processing controller that arbitrates between local and remote computing clusters, storing data in a standardized format and managing access through a centralized server, allowing access from various file systems and ensuring secure processing of sensitive data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored and managed across multiple disparate file systems in high performance computing systems, then data accessibility and processing capability are improved, but network resource efficiency deteriorates and performance lag increases
Solution Approach 1:
The patent merges multiple disparate file systems into a unified virtual file system through a virtualization layer. This virtualization layer presents a consistent interface to clients while abstracting the underlying physical storage infrastructure, thereby improving network resource efficiency by eliminating redundant data transfers and optimizations across different file systems.
Solution Approach 2:
The patent introduces a virtualization layer as an intermediary between clients and physical storage systems. This intermediary manages data caching, replication, and access routing to optimize network resource utilization while maintaining seamless access to data across distributed storage locations.
2Productivity
If centralized data storage is implemented to improve data processing efficiency, then network resource utilization is optimized, but data access complexity increases
Solution Approach 1:
The virtualization layer serves as an intermediary that simplifies data access by providing a unified interface. Clients interact with data through standardized protocols without needing to understand the complex underlying storage architecture, including data replication and caching mechanisms managed automatically by the virtualization layer.
Solution Approach 2:
The virtual file system provides universal access to data regardless of its physical location or the specific storage system it resides in. This multi-functional approach allows the same access interface to work across different storage infrastructures, reducing the complexity burden on clients.
3Reliability
If manual arbitration of data processing operations is performed, then data security and privacy control are improved, but processing time and operational complexity increase
Solution Approach 1:
The virtualization layer implements self-service data processing arbitration by automatically determining optimal data handling based on predefined policies. The system autonomously manages data replication, caching decisions, and access routing without requiring manual intervention, thereby reducing processing time while maintaining security through policy-based control.
Solution Approach 2:
The system performs preliminary actions by pre-establishing data handling policies and caching strategies. These pre-configured rules enable the virtualization layer to make rapid, automated decisions about data processing without manual arbitration, reducing both processing time and operational complexity while maintaining security standards.
Data Source
AI summary
A data processing architecture controls data processing arbitration in a high performance computing system that includes one or more premises. Individual premises can include one or more server computers executing an instance of a local file system and including one or more temporary data storage devices. Individual instances of the local file system can access files stored in objects of a primary data store. Individual objects of the primary data store can be accessed using a common identifier indicating a storage location of the individual objects in the primary data store.