Multi-Source Data Collection With Dynamic RDMA Path Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data collection methods require a large amount of network resources when collecting data from a large number of sources, especially with the use of remote direct memory access (RDMA), as they necessitate pre-setting communication paths for each flow.
Innovation Solution
A system and method where a controller determines transmission policies for data sources, which autonomously request and manage communication paths based on data transfer requirements, allowing efficient allocation and release of paths as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If communication paths are pre-set for each flow using RDMA, then data transfer speed is improved, but network resource consumption increases
Solution Approach 1:
Multiple data flows from different data sources are merged into a single shared communication path to the server. Instead of establishing separate dedicated paths for each flow, the system combines multiple flows and allocates the shared path dynamically, reducing the total number of communication paths required while maintaining high transfer speeds through RDMA.
Solution Approach 2:
A single communication path is designed to serve multiple functions by handling data flows from multiple different data sources simultaneously. The shared path can be dynamically allocated to different flows based on demand, making the network infrastructure more versatile and efficient while reducing overall resource consumption.
2Reliability
If dedicated communication paths are established for each data source, then data transfer reliability is improved, but device complexity increases
Solution Approach 1:
A path management controller is introduced as an intermediary between data sources and the server. This mediator handles the complex tasks of establishing, managing, and tearing down communication paths dynamically. The controller receives path establishment requests, allocates shared paths appropriately, and ensures reliable data transfer without requiring individual data sources to manage complex path configurations themselves.
Solution Approach 2:
Data sources are empowered to autonomously request communication path establishment based on their own data transfer requirements and policies. Each data source can independently initiate path setup requests to the path management controller, reducing the need for centralized control and simplifying the overall system architecture while maintaining reliability through automated path management.
Data Source
AI summary
An object of the present disclosure is to provide a system that does not require a large amount of network resources even in a case where data is collected from a large number of data sources. The present disclosure provides a system that collects data from a plurality of data sources to a server, wherein each of the data sources sends data transfer requirements to a controller, the controller determines a transmission policy of each of the data sources on the basis of the data transfer requirements, each of the data sources autonomously sends a request for generating a communication path on the basis of the transmission policy, the controller generates a communication path on the basis of the request for generating a communication path from each of the data sources, and each of the data sources transmits data by use of the generated communication path.