Distributed Data Transfer With Autonomous File Reassembly Checks

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Existing big data analysis systems face challenges in managing large, complex datasets due to issues in data capture, storage, analysis, sharing, transfer, visualization, querying, updating, and information privacy, particularly in distributed networks.

Innovation Solution

A distributed analytics development platform (DADP) with a user interface layer, application programming interface layer, and orchestration layer, utilizing autonomous processing nodes to manage and transfer large data files efficiently across a network, including a distributed packaging and reassembly system (DPRS) for handling excess size data files.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If multiple autonomous processing nodes discover and reassemble data blocks in parallel, then data transfer speed is improved, but computational resources are wasted due to redundant processing

Engineering Contradiction:
Improvedata transfer speedVSAvoidcomputational resource usage
Core Design Contradiction:
SpeedVSLoss of energy

Solution Approach 1:

The system implements a feedback mechanism where autonomous processing nodes check a distributed file system to determine if a data block has already been reassembled before initiating reassembly operations. This feedback loop prevents redundant processing by allowing nodes to query the system state and adjust their behavior accordingly, thereby maintaining high data transfer speed while eliminating wasted computational resources on duplicate reassembly operations.

Inventive Principle:
Principle #23Feedback

2Device complexity

If autonomous processing nodes operate without active management, then system complexity is reduced, but coordination overhead increases

Engineering Contradiction:
Improvesystem management complexityVSAvoidcoordination overhead
Core Design Contradiction:
Device complexityVSLoss of time

Solution Approach 1:

Autonomous processing nodes operate independently without requiring active management or centralized coordination. Each node autonomously discovers data chunks, determines whether reassembly is needed by checking the distributed file system, and executes reassembly operations independently. This self-service approach eliminates the need for complex inter-node coordination mechanisms, reducing system management complexity while minimizing coordination overhead through autonomous decision-making.

Inventive Principle:
Principle #25Self-service

3Adaptability or versatility

If data blocks are reassembled from distributed chunks, then data file transfer capability is improved, but risk of redundant reassembly operations increases

Engineering Contradiction:
Improvedata file transfer capabilityVSAvoidredundant processing prevention
Core Design Contradiction:
Adaptability or versatilityVSReliability

Solution Approach 1:

Before autonomous processing nodes initiate data block reassembly operations, they perform preliminary checks by querying the distributed file system to determine if the data block has already been reassembled. This preliminary action prevents redundant reassembly operations while maintaining the system's ability to transfer data files by reassembling distributed chunks, thereby improving reliability by eliminating duplicate processing while preserving data transfer capability.

Inventive Principle:
Principle #10Preliminary action

Data Source

PatentUS20260079729A1Asynchronous distributed data transfer system
Publication Date: 2026.03.19 ANALYTICS HQ LLC
  • US20260079729A1 patent drawing
  • US20260079729A1 patent drawing
  • US20260079729A1 patent drawing

AI summary

Apparatus and associated methods relate to a distributed analytics development platform (DADP) capable of automatically maintaining a multiple user development environment in real-time. In an illustrative example, a DADP includes a user interface (UI) layer, an application programming interface (API) layer, and an orchestration layer. The orchestration layer, for example, includes tool instances deployed for each of the multiple users. The orchestration layer may further include a multi-instance common orchestration service (COS) having an orchestration service instance (OSI) deployed in each of the tool instances. The COS, for example, may access a current state of each tool instance associated with the user in the orchestration layer and update a dynamic system state profile based on a current state of each of the tool instances. Various embodiments may advantageously provide an autonomously updated analytic development environment for deployment and maintenance of the tool instances in real-time.