Storage Node Serial Processing for Low-Transfer Data Functions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale distributed storage systems face inefficiencies in data processing due to the need for transferring large data sets for operations like map-reduce, which can be resource-intensive and inefficient, especially when the output is small compared to the input.
Innovation Solution
Implementing a system where storage nodes decode and process erasure-encoded symbols locally, generating intermediate contexts using serial-functions, and sending these contexts between nodes to reduce data transfer and enhance efficiency and reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is fetched from storage system and reconstructed for client-side processing, then data processing operations can be performed, but data transfer efficiency deteriorates due to large data sets being moved across the network
Solution Approach 1:
The patent segments the data processing workflow into distributed map operations executed locally at storage nodes and a centralized reduce operation. Each storage node processes its local erasure-coded symbols independently through the map-function, generating intermediate contexts that are then aggregated. This segmentation eliminates the need to transfer the entire data set to the client, moving only small intermediate context results across the network.
Solution Approach 2:
The patent introduces a new dimension of processing by enabling map-reduce operations to execute distributed across the storage network infrastructure itself rather than requiring centralized client-side processing. Storage nodes become active processing participants, transforming the system from a simple storage-fetch model to a distributed computing architecture where processing occurs in parallel across multiple nodes.
2Productivity
If all data is reconstructed and sent to client system for map-reduce operations, then complete data processing can be performed, but processing time increases due to data transfer bottlenecks
Solution Approach 1:
The patent applies preliminary action by having storage nodes decode erasure-coded symbols and execute map-functions locally before any data transfer occurs. This pre-processing generates intermediate contexts that contain only the essential results needed for the reduce operation, eliminating the time-consuming step of transferring the complete original data set across the network.
Solution Approach 2:
The patent enables continuous useful action by allowing map-operations to execute in parallel across multiple storage nodes simultaneously rather than sequentially after data transfer. Each node processes its local symbols concurrently, and intermediate contexts are continuously generated and prepared for aggregation, maximizing utilization of available processing resources.
3Loss of energy
If storage nodes process erasure-coded symbols locally, then data transfer is reduced, but system complexity increases due to distributed processing coordination
Solution Approach 1:
The patent applies universality by enabling storage nodes to perform multiple functions: data storage, erasure-code decoding, and map-function execution. This multi-functionality reduces the need for separate dedicated processing infrastructure, as the existing storage nodes are enhanced to handle the full workflow locally, simplifying the overall system architecture despite the added processing capabilities.
Solution Approach 2:
The patent implements self-service by enabling storage nodes to autonomously decode their local erasure-coded symbols and execute map-functions without requiring external intervention or data retrieval. Each node independently processes its stored symbols, generating intermediate contexts that are then aggregated by the reduce-function, allowing the system to serve itself without client-side processing overhead.
Data Source
AI summary
Example storage systems, storage nodes, and methods provide storage node processing of data functions, such as serial functions. Storage nodes are configured to partition decode erasure encoded symbols, identify subunits of a data unit from the decoded symbols, process the subunits using a serial function to generate intermediate contexts, and send the intermediate context to a next storage node for continued processing using the serial function.