Key-Value Data Structure Information for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed key-value store systems, there is a lack of association between the file system managing the data source and the distributed KVS management module, leading to information loss and increased communication load between servers when reflecting updates in the key-value data.
Innovation Solution
A mechanism is introduced to manage the data source in association with key-value data, allowing updates to be reflected in the data source through a file I/O API or key-value data API, and reducing communication load by integrating storage areas across multiple computers and managing key-value data structures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If key-value data is generated from file records and stored in distributed KVS, then processing speed is improved through parallelization, but information about the record structure and file organization is lost
Solution Approach 1:
The patent introduces key-value data structure information as an intermediary that bridges the file system and distributed KVS. This intermediary retains essential structural information (record positions, key-value mappings) while enabling the distributed KVS to operate with simplified key-value pairs, thus preserving necessary information without sacrificing processing speed
Solution Approach 2:
The patent segments the data management system into distinct components: the file system managing original files, the distributed KVS managing key-value data, and key-value data structure information managing the correspondence between them. This segmentation allows each component to operate independently with optimized performance while maintaining information integrity through the structure information
2Adaptability or versatility
If key-value data and file data are stored in separate locations without association, then storage flexibility is improved, but communication load between servers increases
Solution Approach 1:
The key-value data structure information acts as an intermediary that associates file data locations with key-value data locations. This intermediary contains mapping information that enables servers to directly locate both file data and corresponding key-value data without requiring frequent inter-server communications, thus reducing communication load while maintaining storage flexibility
Solution Approach 2:
The patent performs preliminary action by pre-establishing the association between file data and key-value data through key-value data structure information during the data loading phase. This preliminary association enables subsequent operations to proceed with reduced communication overhead, as the mapping information is already available locally
3Device complexity
If the file system and distributed KVS management module operate independently, then system modularity is improved, but coordination for reflecting updates becomes difficult
Solution Approach 1:
The key-value data structure information serves as an intermediary that enables coordination between the independent file system and distributed KVS management module. It contains the necessary mapping and structural information that allows either system to initiate updates and ensures the other system can reflect these updates correctly, thus maintaining modularity while improving ease of operation
Data Source
AI summary
A computer system comprising computers, each the computers is coupled to a storage apparatus storing at least one file including records, each the computers includes a file system, a key-value data management module, and a loader, each the computers holds key-value data structure information, the loader being configured to identify a file to be processed in which the at least one piece of the updated key-value data is to be reflected; and transmit a request to update the file including the at least one piece of the updated key-value data; and the file system being configured to identify the position of one of the records corresponding to the piece of updated key-value data within the file based on the key-value data structure information, and update the file by writing the at least one piece of the updated key-value data.