Hadoop Data Replication Mechanism for Secure Cloud Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large organizations face challenges in securely and scalably providing access to confidential data stored in private networks to clients, as traditional methods like FTP and direct downloads are limited in scalability and real-time access.
Innovation Solution
A computer-implemented method that receives a data request from a client, verifies authorization, establishes a secure connection with a public cloud infrastructure, provisions a client-specific repository, and executes redundancy operations to create a persistent state of the data, enabling secure and real-time access to the data in a Hadoop master data repository.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional mechanisms like FTP and direct download are used for data access, then clients can access data files, but scalability and real-time access capability are limited
Solution Approach 1:
The patent introduces a data repository service as an intermediary component that sits between the Hadoop distributed file system and clients. This service receives client requests, retrieves data from HDFS, and delivers it to clients, thereby enabling scalable real-time access without requiring clients to directly connect to the Hadoop cluster. The intermediary architecture allows the system to handle multiple client requests concurrently while maintaining control over data access patterns.
Solution Approach 2:
The patent segments the data access function into separate components: the Hadoop distributed file system for data storage and the data repository service for data retrieval. This segmentation allows the storage layer and access layer to operate independently, enabling the system to scale the access layer to serve multiple clients simultaneously while the storage layer maintains data integrity. Each client interaction is handled as a separate unit through the service layer.
2Reliability
If confidential data is stored in a private network for security, then data security is maintained, but access for external clients becomes difficult
Solution Approach 1:
The data repository service acts as a secure intermediary that external clients can access without needing direct connectivity to the private Hadoop network. The service receives authenticated client requests, retrieves data from the secure HDFS storage, and delivers it through controlled channels. This intermediary layer maintains security boundaries while enabling external access, as the service can implement authentication, authorization, and data formatting controls.
Solution Approach 2:
The patent creates a virtual copy of the data access interface through the data repository service. Instead of requiring clients to access the actual Hadoop cluster, the service provides a replicated access point that mimics the data repository functionality. This virtual copy allows clients to interact with data as if it were locally available, while all actual data retrieval operations occur through the secure private network interface.
3Ease of operation
If data is replicated to public cloud infrastructure, then client access is improved, but network security may be compromised
Solution Approach 1:
The data repository service serves as a security intermediary positioned between the private Hadoop network and public cloud infrastructure. It receives client requests from public networks, retrieves data from the private HDFS through controlled interfaces, and delivers data to clients in the public cloud. This intermediary maintains security by ensuring that no direct connections are established between external systems and the Hadoop cluster, while still enabling public access through the service layer.
Data Source
AI summary
Systems and methods for populating data in a client specific repository in a public cloud infrastructure is described. The system includes a private master data repository (MDR) implemented in a private network and having data stored therein. A client submits a data request to access the MDR and to perform an operation on data within the MDR. The data request may include information for identifying the client and information for determining whether the client is authorized to perform the requested operation. The system further includes a persistence mechanism configured to interface with the MDR in response to the data request. Upon verifying the client is authorized to access the MDR, the persistence mechanism may be configured to provision a directory in the public cloud infrastructure and to create and populate data in the client specific repository as a persistent state of the data in the MDR.