Storage Node Map-Reduce Processing for Distributed Data Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems face inefficiencies in data processing due to the need to transfer large data sets for operations like map-reduce, which can be resource-intensive and inefficient, especially when the output is small compared to the input.
Innovation Solution
A system where storage nodes process subunits of data locally using predefined functions, generating intermediate contexts that are aggregated to produce a final result, reducing the need for extensive data transfer and improving efficiency by executing functions closer to the stored data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is fetched from storage system to client for processing, then data operations can be executed, but data transfer bandwidth is inefficiently utilized and processing time increases
Solution Approach 1:
Instead of fetching data from storage to client for processing, the invention inverts the approach by bringing the processing functions to the storage nodes. Map-reduce operations are executed directly on the distributed symbols within the storage system, eliminating the need to transfer large datasets across the network while maintaining processing capability.
Solution Approach 2:
The invention introduces an intermediary processing layer within the storage system that acts as a mediator between data storage and client access. Storage nodes execute predefined functions on stored symbols and return only the processed results to clients, serving as an intermediate computation layer that avoids full data retrieval.
2Reliability
If all data is reconstructed and sent to client system, then complete data processing can be performed, but network bandwidth is underutilized when output is small relative to input
Solution Approach 1:
The invention extracts only the necessary computed results from the storage system and returns them to clients, rather than transferring all input data. By executing map-reduce operations within the storage nodes, the system extracts only the final reduced output, minimizing network transfer while ensuring processing completeness.
Solution Approach 2:
The invention segments the data processing function into distributed operations across multiple storage nodes. Each node independently processes its local symbols through map and reduce functions, allowing parallel execution and eliminating the need to reconstruct and transfer entire datasets, thus reducing network bandwidth consumption while maintaining processing integrity.
3Adaptability or versatility
If map-reduce operations are executed on fetched data, then data analysis can be performed, but resource efficiency decreases due to redundant data transfer
Solution Approach 1:
The invention makes storage nodes multi-functional by enabling them to execute not only storage operations but also data processing operations. Storage nodes are equipped to perform map-reduce functions locally, combining storage and computation capabilities in a single component, thereby reducing the need for separate data transfer and processing infrastructure.
Data Source
AI summary
Example storage systems, storage nodes, and methods provide storage node processing of predefined data functions, such as map-reduce functions. Storage nodes storing a plurality of symbols for a data unit are configured to select a predefined function using a data type of the data unit. Each storage node identifies subunits of the data unit from the symbols and processes the subunits using the predefined function to generate function results. A final result is returned based on the function results from each storage node.