Separating Storage and Computation for Elastic Database Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in elasticity, particularly in cloud-native databases, where adding or removing computing nodes leads to delayed query responses due to the need for significant data redistribution, causing temporary unavailability of nodes.
Innovation Solution
Implementing a data storage system with computing nodes that have cache storage, where data is mapped dynamically across nodes using consistent hashing, allowing changes in the number of nodes without disrupting data transfer, and using non-volatile memory to maintain data integrity during node shutdowns and startups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If computing nodes are added or removed in a traditional database system, then system scalability is improved, but data redistribution time increases and query response is delayed
Solution Approach 1:
The patent segments the database system into separate storage devices and computing nodes. Storage devices hold the actual data while computing nodes perform computation and caching. This segmentation allows computing nodes to be added or removed without requiring data redistribution, as each node independently accesses data from storage devices through a unified namespace.
Solution Approach 2:
The patent introduces storage devices as intermediaries between computing nodes and persistent data. The storage devices maintain a unified namespace that abstracts the physical location of data, allowing computing nodes to access data without needing to know or care about the number or configuration of other computing nodes. This intermediary layer eliminates the need for data redistribution when nodes are added or removed.
2Stability of the object's composition
If data is redistributed across computing nodes when nodes are added or removed, then data balance is improved, but system availability deteriorates due to temporary unavailability
Solution Approach 1:
By separating storage functions from computation functions, the patent allows storage devices to maintain data balance independently while computing nodes remain available for queries. The storage devices handle data redistribution transparently without affecting computing node availability, as nodes continue to serve requests while data is being rebalanced in the background.
Solution Approach 2:
The system performs data redistribution in advance or in the background without requiring computing nodes to be taken offline. The unified namespace allows data to be pre-positioned on storage devices according to the current node configuration, and when nodes are added or removed, the namespace is updated to reflect the new configuration without requiring immediate data movement that would disrupt service.
3Speed
If data is cached in computing nodes for faster access, then query speed is improved, but data transfer overhead increases when nodes are added or removed
Solution Approach 1:
The unified namespace provided by storage devices acts as an intermediary that tracks the logical-to-physical mapping of data. When computing nodes are added or removed, the namespace is updated to redirect queries to the correct storage devices and caching nodes, eliminating the need for manual data transfer between nodes. The namespace absorbs the overhead of tracking data locations, allowing caches to be optimized for speed without incurring redistribution overhead.
4Device complexity
If traditional database architecture is used with tight coupling between storage and computation, then system simplicity is maintained, but elasticity is reduced
Solution Approach 1:
The patent segments the traditionally coupled storage-computation architecture into independent storage devices and computing nodes. This segmentation increases elasticity by allowing independent scaling of storage and computation resources, while the unified namespace maintains architectural simplicity by providing a single, consistent interface for data access regardless of the underlying physical configuration.
Solution Approach 2:
The unified namespace provides universal access to data across different storage devices and computing nodes. It serves multiple functions: abstracting physical data locations, enabling flexible node addition/removal, supporting data caching strategies, and maintaining consistent data access patterns. This multi-functional namespace enables elasticity without requiring complex architecture changes at the application level.
Data Source
AI summary
A device such as a data storage system comprises a non-transitory memory storage comprising instructions, and one or more processors in communication with the memory. The one or more processors execute the instructions to: map a different portion of data in a storage device to each of different caches, wherein each cache is in a computing node with a processor; change a number of the computing nodes; provide a modified mapping in response to the change; and pass queries to the computing nodes. The computing nodes can continue to operate uninterrupted while the number of computing nodes is changed. Data transfer between the nodes can also be avoided.