Secure Distributed File System for Cloud Big Data Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions for secure Big Data analytics in the cloud either require trusting public cloud service providers with sensitive data or face administrative and compliance issues due to inflexible encryption methods, lacking fine-grained control and non-compliance with regulatory requirements.
Innovation Solution
Implementing a Secure Distributed File System (SDFS) that encrypts and decrypts data files using sharing policy identifiers and security keys, allowing for fine-grained control over data security and integration with existing Hadoop platforms without modifying the underlying operating system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is encrypted before being uploaded to the public cloud, then data security is improved, but analytics processing capability deteriorates because analytics algorithms cannot process encrypted data
Solution Approach 1:
The system segments data into two distinct forms: encrypted data files stored in the cloud for security, and unencrypted metadata files containing analytical information for processing. This segmentation allows analytics algorithms to work with the unencrypted metadata while the encrypted data files remain secure in cloud storage, resolving the contradiction between data security and analytics processing capability
Solution Approach 2:
The system introduces an intermediary layer (the metadata file system) that sits between the encrypted data files and analytics processing algorithms. The metadata files act as intermediaries that contain unencrypted information about the data, enabling analytics processing without requiring direct access to or decryption of the encrypted data files
2Reliability
If the underlying file system is fully encrypted, then data security is improved, but system flexibility and fine-grained control deteriorate
Solution Approach 1:
The system applies encryption selectively at the file level rather than uniformly across the entire file system. Each data file can be independently encrypted with its own security parameters, allowing different levels of security for different files while maintaining unencrypted metadata files for analytics processing. This local quality approach enables fine-grained control over which files are secured and with what level of protection
3Reliability
If the underlying operating system is modified for encryption, then data security is improved, but administrative complexity and management costs increase
Solution Approach 1:
The system extracts the encryption functionality from the underlying operating system and implements it as a separate file system layer. This extraction allows the operating system to remain unchanged and unmodified, while encryption capabilities are provided by the custom file system implementation. Consequently, administrative complexity is reduced because no OS modifications are needed, and standard OS tools can be used for management
Data Source
AI summary
Methods are provided for enabling secure big data analytics in the cloud. A method includes reading a secure file, by a Secure Distributed File System in a public cloud service provider. The reading step includes computing a hash of a name of the secure file to obtain a hashed file name, finding metadata for the secure file using the hashed file name, retrieving a sharing policy identifier from the metadata, and obtaining authorization from an external entity to decrypt the secure file. The reading step further includes extracting a security key and encrypted data file names from the metadata using the sharing policy identifier, requesting one or more encrypted data files that form the secure file from a node of the public cloud service provider, and at least one of decrypting and reconstructing plaintext data for the secure file from the one or more encrypted data files.