UDF Stream Data Access With Distributed File Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud-based data systems face challenges in scaling user defined functions (UDFs) to access large files or a large number of files efficiently, limiting practical application.
Innovation Solution
A function file access system that stores UDF signatures and allows sharing and execution across a distributed database, with execution nodes caching file contents locally and managing access through credentials, enabling just-in-time retrieval and caching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If user defined functions are implemented on databases to process data, then data processing capability is improved, but scalability to access large files or large numbers of files deteriorates
Solution Approach 1:
The system segments file access operations by introducing execution nodes that handle individual file retrieval and caching independently. Each execution node can cache files locally, allowing parallel access to multiple files without blocking other operations. This segmentation enables the system to scale to handle large numbers of files by distributing access operations across multiple nodes.
Solution Approach 2:
The system performs preliminary actions by pre-caching files that are likely to be accessed by UDFs. The execution nodes retrieve and cache file contents in local memory before they are actually needed by the database queries. This preliminary caching action reduces retrieval latency when UDFs need to access these files, improving overall system performance.
2Reliability
If files are retrieved and processed in real-time without caching, then data freshness is improved, but retrieval latency increases
Solution Approach 1:
The system retrieves and caches files in advance before they are actually needed by UDFs. Execution nodes proactively fetch file contents from cloud storage and store them in local memory. This preliminary action ensures that when UDFs need to access these files, the data is already available locally, eliminating retrieval latency while maintaining data freshness through controlled caching strategies.
Solution Approach 2:
The execution nodes act as intermediaries between the database system and cloud storage. They buffer file contents in local memory, mediating between the need for fast data access and the reality of cloud storage retrieval times. This intermediary caching layer reduces the time penalty of cloud access while maintaining reliable data delivery to UDFs.
3Quantity of substance
If cloud storage is accessed directly for every file operation, then storage capacity is improved, but access speed deteriorates
Solution Approach 1:
The system segments file access by introducing local cache memory at execution nodes. Frequently accessed files are cached locally rather than being retrieved from cloud storage for every operation. This segmentation of access patterns - with hot data in local memory and cold data in cloud storage - dramatically improves access speed for active files while maintaining access to unlimited cloud storage capacity.
Solution Approach 2:
The system applies local quality by providing fast local cache memory at execution nodes for frequently accessed files, while relying on cloud storage for less frequently accessed data. This creates different access characteristics for different data - hot data gets fast local access while cold data uses cloud storage, optimizing overall system performance.
Data Source
AI summary
Techniques for accessing data objects using UDF streams include decoding by at least one hardware processor of a first computing node, a first request for a UDF. The first request is generated at a second computing node by one of a plurality of input streams of the UDF. A second request for one or more data objects associated with the UDF is generated at the first computing node using an execution thread assigned to the first request. An authorization of the execution thread is performed using data access rights assigned to the first request. The one or more data objects are retrieved based on the authorization in response to the second request. The one or more data objects are stored in a storage location accessible by the execution thread.