Distributed Storage Data Transfer via RDMA Segmentation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed storage systems experience processing delays and reduced performance due to overheads in data access and synchronization across multiple storage apparatuses, particularly during write operations, as existing techniques like NVMe-oF and RDMA increase processing loads and latency.

Innovation Solution

A distributed storage system architecture where a data processing apparatus stores write data in memory and sends transfer requests to storage apparatuses, allowing for RDMA-based data transfers between nodes, reducing the need for metadata management and processing overheads, and optimizing data replication and access through parallel data transfers.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data is synchronized among multiple storage apparatuses in a distributed storage system, then data redundancy and reliability are improved, but processing delay increases due to synchronization overhead

Engineering Contradiction:

Improvedata redundancyVSAvoidprocessing delay

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent segments the data transfer process into multiple independent paths: direct transfer from source storage apparatus to destination storage apparatus, and alternative transfers through intermediate storage apparatuses. This segmentation allows parallel execution of transfer operations, reducing synchronization overhead while maintaining data redundancy across multiple nodes.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary actions by pre-establishing transfer paths and pre-positioning data in intermediate storage apparatuses before actual synchronization is needed. This allows the system to prepare data in advance at multiple locations, so that when synchronization is required, data can be retrieved quickly from pre-positioned locations rather than transferring from the original source.

Inventive Principle:

Principle #10Preliminary action

2Speed

If NVMe over Fabrics (NVMe-oF) is utilized for remote storage access, then storage performance is improved, but processing overhead increases

Engineering Contradiction:

Improvestorage access speedVSAvoidprocessing overhead

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The patent implements self-service mechanisms where storage apparatuses autonomously manage data transfer operations without requiring centralized coordination. Each storage apparatus independently handles transfer requests, manages its own data synchronization, and communicates directly with other storage apparatuses, eliminating the need for complex centralized control logic and reducing processing overhead.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent introduces intermediate storage apparatuses that act as mediators in data transfer operations. These intermediaries buffer and forward data between source and destination storage apparatuses, reducing the processing burden on any single node and simplifying the overall transfer protocol by breaking down complex multi-node transfers into simpler sequential or parallel segments.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Quantity of substance

If multiple storage apparatuses are used in a distributed system, then storage capacity and reliability are improved, but access overhead increases

Engineering Contradiction:

Improvestorage capacityVSAvoidaccess overhead

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent segments the distributed storage system into autonomous storage nodes that each manage their own data locally. This segmentation allows the system to scale storage capacity by simply adding more independent nodes without increasing access overhead, as each node operates autonomously and can be accessed independently rather than requiring coordinated access across the entire distributed system.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11782615B2Information processing system, non-transitory computer-readable recording medium having stored therein storage controlling program, and storage controller

Publication Date: 2023.10.10 1FINITY INC

AI summary

An information processing system includes a data processing apparatus and multiple storage apparatuses. The data processing apparatus stores, into a memory, write data to be written to a first storage apparatus, and sends, to the first storage apparatus, a first transfer request containing information of a storing location of the write data in the memory. The first storage apparatus executes a transfer process to read the write data from the storing location, and writes the write data to a first storage area in the first storage apparatus; sends, to a second storage apparatus, a second transfer request instructing a transfer of the write data from the memory to the second storage apparatus; and sends, to a third storage apparatus, a third transfer request instructing a transfer of the write data from the first storage area to the third storage apparatus.