Executable Objects in Distributed Storage for In-Place Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage systems face high costs and resource inefficiencies due to the need to transfer data from storage nodes to compute nodes for processing, utilizing extensive network and compute resources, and often result in idle compute and memory resources on object storage nodes being underutilized.
Innovation Solution
Implementing executable objects within object storage systems that can run on storage nodes, allowing data processing to occur in-place, reducing the need for data transfer by utilizing idle resources and optimizing network usage through in-place data-centric pipelining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transferred from storage nodes to compute nodes for processing, then data processing capability is improved, but network resources and transfer time are consumed
Solution Approach 1:
The patent merges storage and compute functions by running executable objects directly on storage nodes. This eliminates the need to separate data transfer and processing operations, allowing processing to occur in-place where data resides, thereby eliminating transfer time while maintaining processing capability.
Solution Approach 2:
Instead of transferring data to compute nodes for processing (traditional approach), the patent inverts the approach by transferring executable objects to storage nodes and running them there. This reverses the conventional data flow pattern and eliminates the need for large data transfers.
2Productivity
If data is transferred from storage nodes to compute nodes for processing, then data processing capability is improved, but compute and memory resources are consumed
Solution Approach 1:
Storage nodes perform processing functions using their own idle compute and memory resources. Rather than requiring dedicated compute nodes, the storage nodes serve themselves by executing processing tasks locally, thereby utilizing previously wasted resources without consuming additional external compute resources.
Solution Approach 2:
Storage nodes are designed to perform multiple functions: data storage and data processing. By making storage nodes universal and capable of executing executable objects, the system eliminates the need for separate dedicated compute nodes, reducing overall resource consumption while maintaining processing capability.
3Productivity
If executable objects are stored and run on storage nodes, then resource utilization is improved, but device complexity increases
Solution Approach 1:
Executable objects are stored on storage nodes in advance, ready to be executed when needed. This preliminary preparation allows the storage nodes to quickly launch processing tasks without complex runtime configuration, simplifying the overall system operation while enabling rich functionality.
Solution Approach 2:
The system dynamically manages executable objects on storage nodes, allowing flexible allocation and execution based on demand. This dynamic approach enables resource-intensive processing capabilities while maintaining operational simplicity through automated resource management and scheduling mechanisms.
Data Source
AI summary
Systems, methods, and machine-readable media are disclosed for running an executable object on an object storage system. An executable object including executable code is stored on a first storage node of an object storage system. The first storage node receives a request to run the executable object. The first storage node identifies the physical location of one or more data objects of interest to be processed by the executable object. The first storage node runs the executable object to process the identified one or more data objects. The first storage node receives a request from a second storage node of the object storage system for the executable object in response to one or more data objects of interest being located at the second storage node. The first storage node sends a copy of the executable object to the second storage node.