Managed File Transfer Dynamic Scaling and Segregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional managed file transfer systems face challenges such as complex architecture, potential downtime during maintenance, insufficient storage, high-latency networks, and poorly integrated processes, which disrupt production and incur costly service interruptions.
Innovation Solution
A scalable and load-balanced managed file transfer system with a network interface, engine, database, and shared file storage, featuring dynamic scaling, service demarcation, endpoint scanning, auto recovery, and scheduling modules to optimize file transfer processes, ensure reliability, and minimize downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional managed file transfer systems are used with complex architecture, then file transfer functionality is provided, but system complexity increases and maintenance causes downtime
Solution Approach 1:
The system is divided into multiple independent server components including application servers, file transfer servers, database servers, and gateway servers. Each server performs a specific function and can be maintained independently without affecting other parts of the system, thus reducing overall system complexity while maintaining reliability.
Solution Approach 2:
A message queue system acts as an intermediary between different server components, enabling asynchronous communication and decoupling the system architecture. This mediator layer allows servers to be maintained or scaled independently while maintaining system reliability.
2Ease of manufacture
If servers are stopped and reloaded for maintenance, then system updates can be applied, but production processes are disrupted and downtime occurs
Solution Approach 1:
The system implements dynamic server replacement where new server instances can be added and brought online without stopping the existing system. The load balancer dynamically distributes traffic between old and new servers, allowing maintenance to be performed without downtime.
Solution Approach 2:
The message queue system ensures continuous processing of file transfer requests by maintaining a persistent queue of pending operations. When servers are updated, the queue preserves the workflow state, allowing seamless resumption of operations without interruption to production processes.
3Adaptability or versatility
If multiple applications are hosted across multiple server components, then file transfer functionality is provided, but coordination during maintenance becomes complex
Solution Approach 1:
The gateway server provides universal access functionality that works across all file transfer applications and protocols. This single component handles authentication, protocol translation, and routing for multiple applications, reducing the need for separate coordination mechanisms for each application.
Solution Approach 2:
The system implements health check and status monitoring mechanisms that provide feedback about server availability and operational state. This automated feedback system reduces manual coordination complexity by automatically detecting and managing server status during maintenance operations.
4Productivity
If storage space is insufficient on target server, then file transfer capability is limited, but transfer operations fail
Solution Approach 1:
The system implements a distributed storage architecture where file storage is separated from compute resources and distributed across multiple storage servers rather than relying on single-server local storage. This dimensional change from localized to distributed storage dramatically increases available storage capacity while maintaining transfer throughput.
5Speed
If network latency is high, then file transfer speed is reduced, but transfer operations become unreliable
Solution Approach 1:
The system performs preliminary actions including forward error correction encoding and data chunking before transmission. These preparatory measures enable reliable reconstruction of transferred files even when network latency or packet loss occurs, maintaining both speed and reliability.
Data Source
AI summary
An apparatus, methods, and computer program products are described which transfer files/data from one system to other. Source and target system can be external trading partners or internal enterprise applications. File transfers can be scaled horizontally or vertically, depending on the requirement without any server or application downtime. File transfers can also be managed and executed in different server/infrastructure zones as per the different regulatory requirements (e.g., PCI, CPNI, etc.) or security sensitivity using service demarcation. File transfer platforms can be patched or upgraded by isolating one or multiple components at a time without disrupting running production processes. Auto-recovery component can handle file transfer failures automatically without any manual intervention making the system self-healing. Periodic endpoint scanning is added to the platform to proactively scan and detect connectivity issues with trading partners or internal systems involved in file transfers.