Hot Folder File Triggers in Converged Compute NAS
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional NAS file systems are passive and do not drive decisions at the application level, leading to inefficient resource utilization and performance issues when handling data-intensive scenarios, as they lack the ability to run high-performance computing tasks and are unaware of business rules.
Innovation Solution
The implementation of a converged compute and file system that utilizes 'hot folders' within a scale-out NAS file system, where configuration data specifies which folders are marked as 'hot' for processing by high-performance computing nodes, allowing for intelligent file processing and transformation based on defined rules, thereby enabling efficient data processing and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If client applications poll folders at regular time intervals for new files or changes, then files can be detected and processed, but network resources are unnecessarily consumed and system performance deteriorates as the number of clients increases
Solution Approach 1:
The file system itself performs the polling function through a file trigger mechanism. When a file is created or modified in a hot folder, the file system automatically generates a trigger event and notifies waiting applications. This eliminates the need for client applications to continuously poll, as the system serves itself by detecting file changes and initiating the appropriate notifications, thereby resolving the contradiction between reliable file detection and network resource consumption
Solution Approach 2:
Applications register their interest in specific folders and file patterns in advance. The file system maintains a mapping between triggers and applications, so when a file change occurs, the pre-registered applications are immediately notified. This preliminary registration approach allows the system to prepare notification routes beforehand, eliminating the need for continuous polling while ensuring reliable file detection
2Quantity of substance
If conventional NAS file systems are used for data storage, then storage functionality is provided, but the systems are passive and cannot run high-performance computing tasks or understand business rules
Solution Approach 1:
The patent creates a multi-functional system where the file system not only performs traditional storage functions but also executes compute services. When a trigger event occurs, the system can dynamically launch compute containers that perform data processing, transformation, or analysis directly on the stored data. This universal approach allows the same infrastructure to handle both storage and computing tasks, resolving the contradiction between storage capacity and computational versatility
Solution Approach 2:
The file trigger acts as an intermediary between the passive file system and active computing resources. When a file is created or modified, the trigger mechanism mediates by notifying registered applications and potentially launching compute services. This intermediary layer enables the file system to become active and intelligent without losing its core storage functionality, allowing it to both store data and initiate computational processes based on business rules
3Productivity
If applications run in a busy-wait state until files arrive or are modified, then computations can be performed upon file arrival, but resources are wasted while applications are doing nothing
Solution Approach 1:
Instead of continuous busy-waiting, the system uses event-driven periodic action. Applications register their interest in specific file patterns and folders, then enter a waiting state. When a file is created or modified matching the registered criteria, a trigger event is generated and the application is notified. This transforms continuous resource consumption into event-based periodic action, maintaining computation execution capability while eliminating wasteful idle resource consumption
Solution Approach 2:
The file trigger mechanism establishes a feedback loop between the file system and applications. When files are created or modified, the system automatically generates feedback in the form of trigger events that notify waiting applications. This feedback mechanism eliminates the need for applications to continuously check for file changes, allowing them to remain in a low-resource state until actual file events occur, thus resolving the contradiction between computation execution and idle resource consumption
Data Source
AI summary
A hot folder mechanism is employed to provide a truly integrated architecture for easy-to-use, easy-to-deploy scale-out computation and scale-out storage. A folder of an Ingest, Transform, Store (ITS)-Network attached storage (NAS) system can be configured as “hot.” The configured hot folder can then detect changes on its content, analyze such content, perform transform services on the content, and output the computation results as files on other specified output folders. In one aspect, file system nodes of the ITS-NAS can present the content to high performance computing (HPC) compute nodes of the ITS-NAS as well as to dispatch the execution of transform services to the HPC compute nodes.