Configurable Automated File Transfer System for Large-Scale Data Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file transfer systems are inadequate for large-scale, automated file transfers and lack criteria for determining which files to transfer, leading to inefficiencies and increased storage needs.
Innovation Solution
The Automatic File Transfer (AFT) system reads configuration files to retrieve and push files based on specified criteria, using rsync for efficient data transfer and maintaining a hierarchical configuration structure to manage file transfers across multiple hosts, with features for filtering, archiving, and notification mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional file transfer tools are used for large-scale file transfers, then individual file transfer capability is provided, but automation and efficiency are insufficient
Solution Approach 1:
The system enables self-service automation through configuration files that define transfer criteria, source/destination hosts, and file patterns. Once configured, the system automatically performs large-scale file transfers without requiring manual intervention for each file, resolving the contradiction between automation extent and transfer efficiency
Solution Approach 2:
The system changes parameters by introducing configurable criteria (file patterns, size ranges, date modifiers) that transform the file transfer process from manual individual operations to automated batch processing, significantly improving productivity while maintaining full automation capability
2Reliability
If all files are transferred without criteria, then complete file transfer is achieved, but storage space requirements increase and transfer time increases
Solution Approach 1:
The system extracts and applies filtering criteria from configuration files to selectively transfer only relevant files based on patterns, size ranges, and date modifiers. This extraction of unnecessary files reduces storage space requirements and transfer time while maintaining reliability by ensuring all required files are transferred according to the defined criteria
Solution Approach 2:
The system performs partial action by transferring only the subset of files that match the configured criteria rather than all files. This selective approach reduces the quantity of data transferred and stored while maintaining transfer completeness for the relevant file set
3Ease of manufacture
If manual file transfer processes are used, then simple transfer operations are performed, but scalability and management capability deteriorate
Solution Approach 1:
The system segments the file transfer functionality into modular configuration files that can independently define transfer criteria, hosts, and patterns. This segmentation allows the system to maintain ease of setup through simple configuration files while achieving large-scale transfer capability by combining multiple configuration sets, resolving the contradiction between simplicity and scalability
4Device complexity
If no notification mechanism is implemented, then transfer process is simpler, but monitoring and error detection capability is reduced
Solution Approach 1:
The system implements feedback through notification mechanisms that alert users when configured file transfers are not retrieved or pushed successfully. This feedback loop provides essential transfer status information and error detection capability while maintaining relatively simple system structure by using straightforward notification protocols
Data Source
AI summary
Method and apparatus for transferring files between hosts including reading a first configuration file to obtain file retrieval criteria, logging into a retrieval host to retrieve a file based on the file retrieval criteria, saving retrieved file to a local host, reading a second configuration file to obtain file push criteria, including instructions for where to push the retrieved file, pushing the retrieved file to a push host based on the file push criteria, pushing an enabled hosts file from the local host to the push host, the enabled host file containing configuration criteria and sending a notification to the retrieval host if a file meeting the configuration criteria has not been retrieved from the retrieval host to the local host, or sending a notification from the local host to the push host if a file meeting the configuration criteria has not been pushed to the push host.