Cloud-Native File Exporter for Data Accessibility
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprise data stored in cloud-native global file systems is not directly accessible by cloud-native tools, limiting the ability to use business intelligence and analytics tools due to data being stored in a format that is not native to these services.
Innovation Solution
A file exporter is introduced to extract and export data from the global file system in a native object format to an unencrypted target, such as an S3 bucket or Azure storage account, creating a short-lived copy that can be leveraged by cloud-native tools, allowing them to operate on the data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in a cloud-native global file system with encryption and obfuscation, then security is improved, but accessibility by cloud-native tools deteriorates
Solution Approach 1:
A data exporter component is introduced as an intermediary between the encrypted cloud-native global file system and cloud-native tools. The exporter extracts data from the encrypted file system, applies decryption and de-obfuscation, and delivers the data in native formats to cloud-native tools, enabling them to access and process the data without directly interacting with the encrypted storage layer
Solution Approach 2:
The system architecture is segmented into distinct functional layers: the encrypted cloud-native global file system layer for secure storage, the data exporter layer for data extraction and format conversion, and the cloud-native tools layer for data processing. This segmentation allows each layer to operate independently with optimized characteristics - security in storage, accessibility in processing
2Reliability
If data is stored in a non-native object format in the cloud, then security and centralized management are improved, but direct operation by cloud-native tools deteriorates
Solution Approach 1:
The data exporter serves as a mediator that translates between the centralized cloud-stored format and the native formats required by various cloud-native tools. It receives requests from tools, fetches data from centralized cloud storage, converts the format, and returns the data in the appropriate native format, enabling seamless operation without compromising centralized management
Solution Approach 2:
The data exporter creates temporary copies of data from the centralized cloud storage in native formats suitable for cloud-native tools. These copies are short-lived and purpose-specific, allowing tools to operate on familiar data formats while the master copy remains securely stored in the centralized cloud-native global file system
Data Source
AI summary
A cloud-native global file system is augmented to include a file exporter (or, more generally, a file export tool) that facilitates an enterprise customer's use of a cloud-native tool that would otherwise be unable to operate against the global file system's underlying file system representation. In a typical use case, the file exporter is configured to extract in a native object format and to an unencrypted target all or a portion of a volume's data from the underlying file system representation. In this manner, the exporter creates a copy of the data set that the enterprise user can then leverage against the desired cloud-native tool or other cloud services that are not under the management or control of the global file system service provider.