Serverless Code Execution on Object Storage Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in enforcing data manipulation functions directly within the storage system and require complex external services to manage data manipulations, which can be insecure and inefficient.
Innovation Solution
The integration of on-demand code execution systems with object storage services allows for the insertion of data processing pipelines into the I/O path, enabling serverless task executions and local stream handling to simplify data manipulations without requiring network communication, thereby enhancing security and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data manipulation functions are enforced within the storage system using external services, then data processing capability is improved, but system complexity and security risks increase
Solution Approach 1:
The storage system executes data manipulation functions directly within its own architecture using integrated code execution environments, eliminating the need for external services. The system serves its own data processing needs through built-in capabilities that allow user-defined code to run locally on storage nodes, thereby reducing system complexity while maintaining enhanced data processing capability.
Solution Approach 2:
The patent combines data storage and data manipulation capabilities into a single integrated system. By merging the code execution environment directly within the storage system architecture, it eliminates the separation between storage infrastructure and processing services, reducing overall system complexity while preserving versatile data processing functionality.
2Adaptability or versatility
If external services are used for data manipulation, then processing flexibility is improved, but security risks increase
Solution Approach 1:
The patent extracts the code execution capability from external services and embeds it directly within the storage system. This extraction removes the security vulnerability of external service dependencies while preserving processing flexibility, as the execution environment is now controlled and isolated within the storage system's own security boundaries.
Solution Approach 2:
The system implements local code execution on individual storage nodes rather than relying on centralized external services. This local quality approach allows each node to independently execute data manipulation code in an isolated environment, maintaining processing flexibility while enhancing security through distributed, localized execution that reduces attack surface and eliminates network-based security risks.
3Adaptability or versatility
If complex external services manage data manipulations, then functionality is improved, but efficiency decreases
Solution Approach 1:
The patent segments data manipulation functionality into distributed code execution units that run locally on storage nodes. Instead of using a monolithic external service, the system divides processing tasks across multiple independent execution environments within the storage system, enabling parallel processing and eliminating network communication overhead, thereby improving efficiency while maintaining versatile functionality.
Solution Approach 2:
The system performs data manipulation operations directly at the storage layer before data needs to be transferred or processed elsewhere. By executing code locally on storage nodes, the system eliminates the need for data to be moved to external services for processing, reducing network I/O and improving overall processing efficiency while preserving full manipulation functionality.
Data Source
AI summary
Systems and methods are described for providing input data to and reading output data from one or more owner-specified functions. A function can implement a data manipulation, such as filtering out sensitive data before reading or writing the data. Input data can be provided to the function as an entire object, a stream of objects, or on-demand based on data or portions thereof that the function attempts to access. The function can provide output data as a single instance of output data or a stream of output data regardless of the manner in which input data is provided to the function. The input data can be stored in a staging area that is accessible to the function via a local file system. Similarly, the function can store the output data in a staging area that is accessible to the function via a local file system.