Data Transfer Manager for Big Data Replication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for transferring data between computation frameworks in big data environments are resource-intensive due to the need to invoke computations, which consumes significant computing resources and is unsustainable for large data sets.

Innovation Solution

A system with a data transfer manager that copies data between computation frameworks without invoking computations, by obtaining a schema, determining storage locations, adapting schemas, and storing data directly in the receiving framework's storage, thereby reducing the reliance on computation frameworks during the transfer process.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is transferred between computation frameworks by invoking computations, then data transfer can be accomplished, but computing resource costs increase significantly

Engineering Contradiction:

Improvedata transfer efficiencyVSAvoidcomputing resource cost

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent extracts the computation invocation step from the data transfer process. Instead of invoking computations to transfer data between frameworks, the system directly copies data at the storage level, removing the unnecessary computation layer that consumes resources.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent introduces a data transfer mechanism that acts as an intermediary between computation frameworks. This intermediary operates at the storage level, enabling direct data copying without requiring the computation frameworks themselves to be invoked, thus reducing their resource consumption.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If data is copied between computing devices, then each device gains access to the same information, but storage resources are consumed

Engineering Contradiction:

Improvedata accessibilityVSAvoidstorage resource usage

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The patent segments the data transfer process into schema adaptation and data copying phases. By first adapting schemas to ensure compatibility and then performing efficient bulk copying, the system achieves broad data accessibility while optimizing storage resource utilization through structured approaches.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11086901B2Method and system for efficient data replication in big data environment

Publication Date: 2021.08.10 EMC IP HLDG CO LLC

AI summary

A system includes a persistent storage and a data transfer manager. The persistent storage stores sending entity storage resources and receiving entity storage resources. The data transfer manager obtains a data transfer request for data in the sending entity storage resources. In response to obtaining the data transfer request, the data transfer manager obtains a sending entity schema associated with the data; determines a current storage location of the data using the obtained sending entity schema; determines a future storage location for a copy of the data in the receiving entity storage resources; stores a copy of the data at the determined future storage location; adapts the sending entity schema based on the determined future storage location; and modifies a receiving entity schema based on the adapted sending entity schema.