Distributed Storage Data Node Asynchronous Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, maintaining availability and improving both write and read performance is challenging due to the need for data conversion to different data structures, which increases processing load and delay, and requires significant storage for holding different data structures.
Innovation Solution
A distributed storage system that temporarily stores updated data in an intermediate structure and asynchronously converts it to a target data structure based on access history, using an access history recording unit to trigger the conversion, thereby optimizing data structure conversion and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is converted to different data structures for various applications, then data availability and versatility are improved, but processing load and conversion delay increase
Solution Approach 1:
The system pre-converts data to multiple data structures in advance and stores them in separate storage locations. When a request arrives, the appropriate pre-converted data structure is directly retrieved without real-time conversion, thereby improving processing speed while maintaining data versatility
Solution Approach 2:
The system segments data storage by creating multiple copies of the same data in different data structures, storing each segment in separate storage locations. This allows different applications to access data in their required formats simultaneously without interfering with each other's performance
2Adaptability or versatility
If data is converted to different data structures for various applications, then data availability is improved, but storage requirements increase
Solution Approach 1:
The system applies different data structures to different portions of data based on local access patterns and application requirements. Frequently accessed data or data with specific access patterns is converted to optimized structures, while less critical data maintains its original format, thereby reducing overall storage requirements while maintaining necessary adaptability
3Reliability
If synchronous data conversion is performed upon update, then data consistency is improved, but write performance deteriorates
Solution Approach 1:
The system performs data conversion in advance during off-peak periods or asynchronously in the background, so that when write operations occur, the conversion has already been completed or is being handled without blocking the main write path, thus maintaining both consistency and write performance
Solution Approach 2:
The system introduces an intermediary conversion layer or buffer that handles data structure transformations separately from the main data write path. This intermediary component manages the conversion process asynchronously or semi-synchronously, ensuring data consistency while preventing write operations from being blocked by conversion delays
Data Source
AI summary
Provided a plurality of data nodes connected in a network, each including a data storage unit. The data node of data replication destination temporarily stores data to be updated in an intermediate data structure, and converts asynchronously with respect to the update request to a target data structure to store the converted data in the data storage unit. Based on access history information stored in an access history recording unit, trigger information, concerning timing for execution of conversion to the target data structure performed asynchronously by the data node is changed.