Parallel Data Streaming Between Cloud and Massively Parallel Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems face challenges in securely sharing data between cloud-based applications and massively parallel systems, particularly 'share nothing' systems, due to security risks and inefficiencies in data retrieval, where entire datasets are loaded offline instead of retrieving only required data, which is costly for large datasets.
Innovation Solution
A method for parallel data streaming between computer systems using authenticated connections, socket-based communication, and multi-threading to securely exchange authorized data, employing TCP/IP sockets or RDMA, allowing direct and secure data exchange with strict access control through credential information and entitlement rules.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is stored in a publicly accessible location for sharing, then data accessibility between disparate systems is improved, but security risks increase
Solution Approach 1:
The patent introduces an intermediary authentication mechanism that mediates between data storage and access. Instead of direct public access or complete isolation, the system uses authentication servers and credential verification as intermediaries to control data flow, allowing secure access without public exposure
Solution Approach 2:
The system dynamically changes access parameters (authentication credentials, entitlement rules) based on the requesting system's identity and permissions. Access control is not static but adapts to different systems and operations, enabling selective data sharing with appropriate security measures
2Ease of operation
If entire datasets are loaded offline for data sharing, then data availability is improved, but cost and inefficiency increase
Solution Approach 1:
The patent extracts and transmits only the specific data subsets required by the querying system rather than loading entire datasets. The system identifies and retrieves only the necessary data portions, eliminating the waste of transferring and storing unnecessary data
Solution Approach 2:
Instead of performing the excessive action of loading complete datasets, the system performs partial action by retrieving only the specific data needed for the query. This selective approach reduces resource consumption while maintaining data availability
3Productivity
If parallel data streaming is implemented between systems, then data transfer efficiency is improved, but system complexity increases
Solution Approach 1:
The patent segments data transfer into multiple parallel streams handled by separate tasks. Each task processes a portion of the data independently, enabling concurrent transmission and improving overall transfer efficiency while maintaining manageable complexity through modular task design
Data Source
AI summary
Embodiments relate to parallel data streaming between a first computer system and a second computer system. Aspects include transmitting a request to establish an authenticated connection between a processing job on the first computer system and a process on the second computer system and transmitting a query to the process on the second computer system over the authenticated connection. Aspects further include creating one or more tasks on the first computer system configured to receive data from the second computer system in parallel and reading data received by the one or more tasks by the processing job on the first computer system.