Distributed Data Management Unit Parallel Read Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data management systems storing data in distributed storages face inefficiencies in data storage and read operations, particularly in improving network speed and data management performance, especially with increased data traffic through networks like mobile and IoT systems.
Innovation Solution
A data management system that reads parts of requested data from multiple distributed storages, combines them to form a complete version, and adaptively adjusts the amounts of data read based on the I/O performance of each storage node, optimizing bandwidth allocation to enhance read processing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If duplicates of data are stored in a plurality of distributed storages, then data reliability is improved, but data read speed deteriorates
Solution Approach 1:
The patent divides the data reading process into multiple parallel operations by segmenting the requested data into different parts and reading them simultaneously from multiple distributed storages. The distributed data management unit splits the read request into multiple sub-requests, each targeting a specific storage node, thereby converting a single sequential read operation into multiple parallel operations that improve overall read speed while maintaining data reliability through duplicate storage.
2Speed
If data is read from multiple distributed storages, then data read speed is improved, but system complexity increases
Solution Approach 1:
The patent introduces a distributed data management unit as an intermediary between the client and multiple distributed storages. This intermediary component coordinates the complex operations of splitting read requests, managing communication with multiple storage nodes, handling data assembly, and error recovery. By centralizing these coordination functions in a dedicated management unit, the system achieves parallel read operations without proportionally increasing overall system complexity.
3Stability of the object's composition
If fixed bandwidth is allocated to each storage node, then system stability is improved, but data read efficiency deteriorates
Solution Approach 1:
The patent implements dynamic bandwidth allocation where the distributed data management unit adjusts the bandwidth allocated to each storage node based on real-time performance monitoring. The system continuously monitors I/O performance metrics of each storage node and dynamically modifies bandwidth allocation to optimize data read efficiency. This dynamic adjustment allows the system to adapt to changing conditions and storage node performance variations, improving overall productivity while maintaining operational stability through controlled adaptation.
Solution Approach 2:
The patent incorporates feedback mechanisms where the distributed data management unit monitors the performance of each storage node during data read operations and uses this feedback information to adjust bandwidth allocation. The system measures actual I/O performance, compares it against allocated bandwidth, and modifies future bandwidth distribution accordingly. This feedback loop enables the system to optimize data read efficiency by directing more bandwidth to high-performance storage nodes while reducing allocation to slower nodes, all while maintaining system stability through gradual, monitored adjustments.
Data Source
AI summary
A data management system and method for processing distributed data, which adaptively adjusts the amounts of data to be read from a plurality of storages, depending on I/O performances of the respective storages. The data management system includes a plurality of storage nodes each storing duplicate data of original data, and a distributed data management unit providing read requests for parts of the duplicate data to the respective storage nodes, combining the read parts into a complete version of the data, and outputting the complete version of the data.