Credential-less External Stage Objects for Secure Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional network-based database systems face security vulnerabilities and inefficiencies when accessing external cloud storage due to the need for sharing secret security credentials, which can lead to data exfiltration risks and performance issues with external tables.
Innovation Solution
The implementation of credential-less external stage objects that allow data loading and unloading without exchanging security credentials, using cloud identity objects and proxy identities to manage access permissions, enabling secure and efficient data operations across different storage formats and locations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If secret security credentials are shared to access external cloud storage, then data access is enabled, but security vulnerabilities and data exfiltration risks increase
Solution Approach 1:
The patent introduces cloud identity objects as intermediaries between the network-based database system and external cloud storage. These identity objects act as mediators that enable data access without requiring direct sharing of secret security credentials. The intermediary mechanism allows the database system to access external storage through authenticated identity objects while maintaining security boundaries, thus resolving the contradiction between enabling data access and preventing data exfiltration.
2Ease of operation
If credentials are exchanged for accessing external storage, then data loading/unloading is enabled, but system complexity and security management overhead increase
Solution Approach 1:
The patent implements self-service mechanisms where cloud identity objects automatically manage their own authentication and authorization. The identity objects contain embedded credentials and access permissions that are automatically used during data loading and unloading operations, eliminating the need for manual credential exchange and management. This self-service approach simplifies the operation of data access while reducing the complexity of credential management.
3Adaptability or versatility
If external tables are used for different storage formats, then adaptability to various file formats is improved, but performance issues occur
Solution Approach 1:
The patent creates a universal interface through cloud identity objects that can access multiple external storage formats (such as Iceberg tables and other file formats) through a standardized authentication mechanism. The identity objects serve as multi-functional connectors that work across different storage systems, enabling the database to access various file formats without compromising performance through format-specific optimization layers.
Data Source
AI summary
The subject technology receives a command to commit a table in a different table format on an external volume. The subject technology generates a first snapshot of the table on internal storage. The subject technology generates a first list of metadata files on the internal storage. The subject technology generates, based on the first list of metadata files, a first set of metadata files on the internal storage. The subject technology generates a second snapshot of the table on the external volume. The subject technology generates a second list of metadata files on the external volume. The subject technology generates, based on the second list of metadata files, a second set of metadata files on the external volume. The subject technology generates a first set of data files in a different file format on the external volume.