Distributed Data Transfer With Autonomous File Reassembly Checks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing big data analysis systems face challenges in managing large, complex datasets due to issues in data capture, storage, analysis, sharing, transfer, visualization, querying, updating, and information privacy, particularly in distributed networks.
Innovation Solution
A distributed analytics development platform (DADP) with a user interface layer, application programming interface layer, and orchestration layer, utilizing autonomous processing nodes to manage and transfer large data files efficiently across a network, including a distributed packaging and reassembly system (DPRS) for handling excess size data files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If multiple autonomous processing nodes discover and reassemble data blocks in parallel, then data transfer speed is improved, but computational resources are wasted due to redundant processing
Solution Approach 1:
The system implements a feedback mechanism where autonomous processing nodes check a distributed file system to determine if a data block has already been reassembled before initiating reassembly operations. This feedback loop prevents redundant processing by allowing nodes to query the system state and adjust their behavior accordingly, thereby maintaining high data transfer speed while eliminating wasted computational resources on duplicate reassembly operations.
2Device complexity
If autonomous processing nodes operate without active management, then system complexity is reduced, but coordination overhead increases
Solution Approach 1:
Autonomous processing nodes operate independently without requiring active management or centralized coordination. Each node autonomously discovers data chunks, determines whether reassembly is needed by checking the distributed file system, and executes reassembly operations independently. This self-service approach eliminates the need for complex inter-node coordination mechanisms, reducing system management complexity while minimizing coordination overhead through autonomous decision-making.
3Adaptability or versatility
If data blocks are reassembled from distributed chunks, then data file transfer capability is improved, but risk of redundant reassembly operations increases
Solution Approach 1:
Before autonomous processing nodes initiate data block reassembly operations, they perform preliminary checks by querying the distributed file system to determine if the data block has already been reassembled. This preliminary action prevents redundant reassembly operations while maintaining the system's ability to transfer data files by reassembling distributed chunks, thereby improving reliability by eliminating duplicate processing while preserving data transfer capability.
Data Source
AI summary
Apparatus and associated methods relate to a distributed analytics development platform (DADP) capable of automatically maintaining a multiple user development environment in real-time. In an illustrative example, a DADP includes a user interface (UI) layer, an application programming interface (API) layer, and an orchestration layer. The orchestration layer, for example, includes tool instances deployed for each of the multiple users. The orchestration layer may further include a multi-instance common orchestration service (COS) having an orchestration service instance (OSI) deployed in each of the tool instances. The COS, for example, may access a current state of each tool instance associated with the user in the orchestration layer and update a dynamic system state profile based on a current state of each of the tool instances. Various embodiments may advantageously provide an autonomously updated analytic development environment for deployment and maintenance of the tool instances in real-time.


