Cloud Token Data Sharing for Scalable Governed Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data sharing protocols, such as pre-signed URLs, do not scale well for large data tables and fail to leverage internal optimizations provided by data processing services, limiting the ability to provide a holistic view of data assets.
Innovation Solution
A data sharing protocol that aligns with the data governance of the data processing service, using cloud tokens to grant temporary access to shared data assets, allowing recipients to access and utilize internal optimizations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If pre-signed URLs are used for data sharing, then security credentials can grant time-limited permission to download data assets, but the sharing protocol does not scale well to large tables and cannot take advantage of internal optimizations
Solution Approach 1:
The patent introduces a sharing server as an intermediary component that mediates between data providers and recipients. This server receives sharing requests, generates optimized access protocols, and coordinates with the data processing service. The intermediary enables scalable data sharing by handling the complexity of permission management and optimization strategies centrally, allowing recipients to access large tables efficiently while maintaining security through controlled credential distribution.
Solution Approach 2:
The patent changes the parameters of the sharing protocol by transitioning from standard pre-signed URLs to optimized protocols that leverage internal data processing service capabilities. This includes adjusting access patterns, utilizing internal optimization mechanisms, and modifying credential structures to enable better performance and scalability for large data tables while maintaining the time-limited security constraints.
2Reliability
If a separate sharing server and protocol are implemented, then security concerns can be addressed, but the protocol cannot take advantage of internal optimizations to the data table
Solution Approach 1:
The patent creates a sharing protocol that serves multiple functions simultaneously: it maintains security through controlled credential distribution while also leveraging internal data processing optimizations. The protocol is designed to be universal, working with various data table structures and sizes, and adapting to different security requirements. This multi-functionality allows the same protocol framework to provide both security and performance optimization without requiring separate mechanisms.
3Ease of operation
If pre-signed URLs with cloud user credentials are used, then access to shared data assets can be granted, but the sharing protocol does not scale well and limits the recipient's holistic view of the data asset
Solution Approach 1:
The patent segments the data access protocol into optimized components that can handle large tables efficiently. Instead of treating each data asset access as a separate pre-signed URL operation, the system divides access into optimized segments that leverage internal data processing capabilities. This segmentation enables recipients to access comprehensive views of large data assets while maintaining scalable performance through distributed, optimized access patterns.
Data Source
AI summary
A data processing service receives indication that a recipient will request access to data assets of a provider and provides a request for credentials from a recipient governance module. The recipient governance module stores a recipient metastore including an object for a provider metastore. In response to determining that the assets are associated with the provider metastore, the service provides a request for credentials to a provider governance module. The provider governance module stores the provider metastore describing data assets of the provider and permissions for accessing data assets. The provider metastore includes a recipient object attached to the data assets with an identifier for the recipient metastore. In response to verifying that the recipient was provided access to the data assets, the service provides a token to the recipient governance module. The service then provides the token to a computing resource to provide access to the data assets.