On-Demand Indexing in Object Storage I/O Path
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in enforcing data manipulations within object storage services without modifying the service interface, leading to complexity and security concerns, especially when handling personally identifying information and requiring users to manage network communications for data manipulations.
Innovation Solution
The integration of on-demand code execution systems with object storage services allows for the configuration of input and output streams, enabling serverless task executions to perform data manipulations within the I/O path without requiring network communication, thus simplifying code creation and enhancing security by using restrictive network access models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data manipulations are enforced within object storage services by modifying the service interface, then data processing capabilities are improved, but system complexity and security risks increase
Solution Approach 1:
The patent introduces an on-demand code execution system as an intermediary between the object storage service and data manipulation requirements. This mediator executes user-submitted code in isolated environments without requiring modifications to the core storage service interface, thereby enabling flexible data processing while maintaining service simplicity and security boundaries
2Adaptability or versatility
If users manage network communications for data manipulations, then data processing flexibility is improved, but security concerns increase
Solution Approach 1:
The patent creates an inert execution environment by implementing restrictive network access models and isolated code execution contexts. User-submitted code runs in sandboxed environments with controlled network permissions, allowing flexible data manipulation while preventing unauthorized external communications and security threats
3Productivity
If code execution systems are integrated with object storage services, then data processing efficiency is improved, but system complexity increases
Solution Approach 1:
The patent segments the system into distinct functional components: the object storage service handles data operations, while the on-demand code execution system handles processing logic. This segmentation allows each component to operate independently with well-defined interfaces, improving data processing efficiency while managing complexity through clear separation of concerns
Data Source
AI summary
Systems and methods are described for modifying input and output (I/O) to an object storage service by implementing one or more owner-specified functions to I/O requests. A function can implement a data manipulation, such as filtering out sensitive data before reading or writing the data. The functions can be applied prior to implementing a request method (e.g., GET or PUT) specified within the I/O request, such that the data to which the method is applied may not match the object specified within the request. For example, a user may request to obtain (e.g., GET) a data set. The data set may be passed to a function that filters sensitive data to the data set, and the GET request method may then be applied to the output of the function. In this manner, owners of objects on an object storage service are provided with greater control of objects stored or retrieved from the service.