Object Storage Service On-Demand Data Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in enforcing data manipulations at storage systems and require complex external services to manage data manipulations, which can be insecure and inefficient.
Innovation Solution
The integration of serverless task executions with object storage services allows for direct insertion of data manipulations into the I/O path using input and output stream handles, simplifying code execution and enhancing security by restricting network communications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If external services are used to manage data manipulations, then data processing functionality is provided, but security is compromised and system complexity increases
Solution Approach 1:
The patent merges the code execution environment directly into the object storage service by integrating a function execution engine within the storage system architecture. This allows data manipulations to be executed as part of the storage I/O operations without requiring separate external services, thereby reducing system complexity while maintaining security through controlled execution environments.
Solution Approach 2:
The patent introduces an intermediary execution layer that sits between the object storage service and external systems. This intermediary executes user-submitted code in a controlled environment, allowing data manipulations to be performed securely without direct external service involvement, thus improving security while managing complexity through abstraction.
2Productivity
If data manipulations are enforced at storage systems, then data processing efficiency is improved, but interface complexity increases
Solution Approach 1:
The patent enables the object storage service to automatically execute data manipulations as part of its native I/O operations. The storage system performs data processing self-service by integrating code execution directly into the storage pipeline, eliminating the need for complex external data processing interfaces while improving processing efficiency.
Solution Approach 2:
The patent makes the object storage service multi-functional by enabling it to perform both data storage and data manipulation operations through a unified interface. The storage service can execute various types of code (filtering, transformation, enrichment) directly on stored objects, reducing the need for separate data processing systems and simplifying the overall interface architecture.
3Adaptability or versatility
If code execution is integrated into object storage service, then data manipulation capability is enhanced, but security risks increase
Solution Approach 1:
The patent applies local quality by creating isolated execution environments for user-submitted code within the object storage service. Each code execution occurs in a controlled, sandboxed context with limited access to system resources, allowing diverse data manipulation capabilities while mitigating security risks through localized execution boundaries.
Solution Approach 2:
The patent creates an inert execution environment where user-submitted code runs in a controlled sandbox with restricted network access and system resource permissions. This inert atmosphere allows flexible data manipulation capabilities while preventing malicious code from affecting the broader system, thus enhancing adaptability while controlling security risks.
Data Source
AI summary
Systems and methods are described for modifying input and output (I/O) to an object storage service by implementing one or more owner-specified functions to I/O requests. A function can implement a data manipulation, such as filtering out sensitive data before reading or writing the data. The functions can be applied prior to implementing a request method (e.g., GET or PUT) specified within the I/O request, such that the data to which the method is applied my not match the object specified within the request. For example, a user may request to obtain (e.g., GET) a data set. The data set may be passed to a function that filters sensitive data to the data set, and the GET request method may then be applied to the output of the function. In this manner, owners of objects on an object storage service are provided with greater control of objects stored or retrieved from the service.