Dynamic File Caching for Workflow Node Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computational workflows face performance bottlenecks due to inefficient communication between workflow nodes, which are often implemented by different developers and scientists, leading to suboptimal interaction with file systems.
Innovation Solution
The method involves dynamic file caching, where each workflow node is associated with a data cache on a local storage device, and workflows are partitioned into sub-graphs based on expected file input/output, allowing nodes to access a distributed filesystem and utilize high-bandwidth storage devices for optimized performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If workflow nodes communicate over file systems to handle large volumes of data, then data transfer capability is improved, but communication performance between workflow nodes deteriorates due to suboptimal interaction with file systems
Solution Approach 1:
The patent introduces a file system cache as an intermediary layer between workflow nodes and the underlying file system. This cache buffer resolves the contradiction by providing high-speed data access for large volume transfers while maintaining efficient communication performance through optimized read/write operations.
Solution Approach 2:
The system performs preliminary actions by pre-loading data into the file system cache before workflow nodes need it. This advance preparation ensures that when large volumes of data need to be transferred, the data is already available in high-speed cache memory, thus maintaining communication performance.
2Productivity
If dynamic file caching is implemented with local storage devices, then workflow execution performance is improved, but device complexity increases due to cache management overhead
Solution Approach 1:
The file system cache is designed to manage itself through automatic eviction policies and transparent operations. The cache autonomously decides what data to retain or evict based on usage patterns, eliminating the need for complex external management mechanisms while maintaining high workflow execution performance.
Solution Approach 2:
The system dynamically adjusts cache parameters such as cache size, eviction policies, and data retention strategies based on workflow characteristics and system load. These parameter changes optimize performance for different scenarios without requiring complex manual configuration or management.
3Speed
If workflow nodes are scheduled on compute nodes hosting their associated data cache, then data access speed is improved, but scheduling complexity increases
Solution Approach 1:
The scheduling system is enhanced with multi-functionality to simultaneously consider both workflow node computation requirements and data cache location. The unified scheduler performs both traditional resource allocation and cache-aware task placement, achieving fast data access without requiring separate complex scheduling mechanisms.
Data Source
AI summary
A computer implemented method for optimizing performance of workflow includes associating each of a plurality of workflow nodes in a workflow with a data cache and managing the data cache on a local storage device on one of one or more compute nodes. A scheduler can request execution of the tasks of a given one of the plurality of workflow nodes on one of the one of more compute nodes that hosts the data cache associated with the given one of the plurality of workflow nodes. Each of the plurality of workflow nodes is permitted to access a distributed filesystem that is visible to each of the plurality of compute nodes. The data cache stores data produced by the tasks of the given one of the plurality of workflow nodes.