Distributed Storage Nodes with Heterogeneous Data Structures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, as the number of nodes increases, the metaserver's processing performance becomes a bottleneck, and existing methods for data replication lead to inefficiencies in storage utilization and response performance due to the need for identical data structures across nodes, which complicates data management and migration.
Innovation Solution
A distributed storage system where data nodes hold logically identical but physically different data structures, allowing asynchronous conversion to a target structure during updates and using an intermediate retention structure to prioritize response performance, enabling efficient data replication and management across multiple nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated to multiple nodes using traditional methods with identical data structures, then availability is ensured, but storage utilization efficiency decreases and response performance is lowered due to redundant data structure conversions
Solution Approach 1:
The patent applies local quality by allowing each data node to store data in different physical data structures (row-store, column-store, key-value) according to its local characteristics and access patterns, while maintaining logical consistency across the distributed system. This enables optimized storage and access at each node without requiring identical structures system-wide.
Solution Approach 2:
The patent changes the parameter of data structure physical form while maintaining logical identity. Different nodes can use different data structure types (row, column, key-value) and storage formats, allowing the system to adapt to varying access patterns and hardware characteristics, thereby improving storage utilization and response performance.
2Stability of the object's composition
If data structure conversion is performed synchronously during data updates, then data consistency is maintained, but response performance deteriorates due to conversion bottlenecks
Solution Approach 1:
The patent performs data structure conversion in advance or in preparation for future operations. When data is written, it is stored in an appropriate structure at the source node, and conversion to other structures at replica nodes is prepared or performed asynchronously, so that read operations can access pre-converted data without waiting for conversion during the operation.
Solution Approach 2:
The patent introduces dynamic behavior to data structure conversion by making it asynchronous and event-driven rather than synchronous and mandatory. Conversion operations can be triggered by specific events (data writes, reads, or system conditions) and executed at optimal times, allowing the system to balance consistency requirements with performance needs dynamically.
3Device complexity
If replicas are held in the same physical structure, then data consistency is simplified, but storage capacity is wasted and hardware resources are excessively required
Solution Approach 1:
The patent allows each replica to be stored in the most appropriate physical structure for its location and access pattern. Instead of forcing all replicas to use the same structure, each node can optimize its local storage format, improving space utilization and reducing the need for excessive hardware resources while maintaining logical consistency through the common data model.
4Ease of operation
If a metaserver is used to manage location information of data, then data location identification is simplified, but processing performance becomes a bottleneck as system size increases
Solution Approach 1:
The patent segments the centralized metaserver functionality and distributes it across multiple nodes in the system. Each node maintains its own data structure information and can independently process location queries for its local data, eliminating the single-point bottleneck while maintaining the ability to identify data locations through the logical data model.
Data Source
AI summary
A distributed storage system of the present invention includes a plurality of data nodes coupled via a network and respectively including data storage units. At least two of the data nodes hold in the respective data storage units thereof replicas of a plurality of types of data structures that are logically identical but are physically different between the data nodes.