Unstructured Storage Service Data Modification Without Export
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional computing systems face inefficiencies in managing and processing data storage and modification tasks, particularly when dealing with computationally intensive operations, as they often require clients to export and re-import data, consuming network bandwidth and processing resources.
Innovation Solution
An unstructured storage service model that utilizes both internal and external compute logic to perform data-modifying commands without exporting source data, allowing clients to offload tasks based on computational intensity and providing features like eventual consistency, authentication, and metering for cost management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If clients perform data-modifying commands locally by exporting data to their own systems, then data processing flexibility is improved, but network bandwidth consumption increases and processing efficiency decreases
Solution Approach 1:
The patent introduces an intermediary processing system between the storage service and clients. This intermediary receives data-modifying commands from clients, performs the actual data processing operations on the stored data, and returns results to clients. This mediator approach allows clients to maintain processing flexibility through command specification while the intermediary handles the computationally intensive operations close to the data storage, minimizing network bandwidth consumption by avoiding full data export/import cycles.
2Ease of operation
If clients export data to perform computations locally, then data processing control is improved, but processing time increases
Solution Approach 1:
The system performs preliminary actions by maintaining data in a processed or pre-computed state within the storage service infrastructure. When clients issue data-modifying commands, the system leverages pre-established processing pipelines and pre-positioned computation resources to execute operations rapidly without requiring full data export. This preliminary preparation of processing environments and data states significantly reduces processing time while preserving client control through command specification.
3Loss of energy
If storage service performs data-modifying commands internally, then network bandwidth usage decreases, but service complexity increases
Solution Approach 1:
The storage service is designed with multi-functionality, integrating both data storage and data processing capabilities within a unified system architecture. The storage service nodes perform multiple functions: storing data, receiving data-modifying commands, executing computation operations, and managing data lifecycle. This universal approach consolidates what would otherwise be separate storage and computing infrastructures, reducing network bandwidth usage by keeping operations internal while managing complexity through integrated service design rather than separate systems requiring extensive intercommunication.
4Device complexity
If computationally intensive tasks are performed externally, then storage service simplicity is improved, but processing efficiency decreases
Solution Approach 1:
The system segments computationally intensive tasks into discrete, manageable data-modifying commands that can be independently processed. Rather than handling monolithic external processing jobs, the storage service breaks down complex operations into atomic command units (e.g., filter, transform, aggregate operations) that can be executed efficiently on stored data. This segmentation allows the service to maintain relative simplicity by processing only the necessary data portions for each command while achieving high processing efficiency through optimized, targeted computations on data already residing in the storage infrastructure.
Data Source
AI summary
Embodiments of system and method for performing object-modifying commands in an unstructured storage service are described. Embodiments may include utilizing one or more nodes to implement an unstructured storage service including a service interface configured to service data storage requests and data retrieval requests from remote clients. The unstructured storage service is configured to isolate access to each of multiple storage domains of the service to different clients. Embodiments may also include receiving from a given remote client, a request to store a data object to a given storage domain of the service. The request further specifying a command to modify source data for the object prior to storing the object. Embodiments may also include using the unstructured storage service to perform the command without exporting the source data from the unstructured storage service to the given remote client prior to completion of the command.