Distributed Virtual Array Host Storage Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional distributed storage systems face challenges with scalability and flexibility, leading to performance degradation due to resource contention and the 'noisy neighbor' problem, where busy virtual machines can slow down others, and it is difficult to independently scale storage and compute resources.
Innovation Solution
A distributed virtual array (DVA) system where hosts and storage nodes form a single, distributed storage system, leveraging the computational power of hosts for storage processing with minimal inter-server communication, using non-volatile memory for low-latency data storage and caching, and employing DVA components like cache managers, file managers, and data processing engines to handle storage tasks without relying on shared resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional storage controllers are made more powerful to handle compute-intensive storage processing functions, then storage processing capability is improved, but system cost and complexity increase significantly
Solution Approach 1:
The patent segments the storage processing functions by separating compute-intensive tasks (compression, encryption, deduplication) from the storage controller and relocating them to host systems. The storage controller retains only basic data movement and management functions, while hosts perform advanced processing using their existing computational resources. This segmentation reduces controller complexity while maintaining or improving overall storage processing capability.
Solution Approach 2:
The patent leverages the universal computational resources already present in host systems to perform storage processing functions. Instead of requiring specialized, high-power storage controllers, the system uses the general-purpose CPUs and memory of existing servers to handle compression, encryption, and other compute-intensive tasks, thereby avoiding the need for increasingly complex and expensive dedicated storage controllers.
2Productivity
If multiple servers share storage controller resources, then resource utilization is improved, but performance degradation occurs due to contention
Solution Approach 1:
The patent segments storage processing tasks so that compute-intensive operations are performed independently on each host rather than being centralized in shared storage controllers. Each host handles its own compression, encryption, and deduplication tasks locally, eliminating contention for processing resources while still achieving high resource utilization across the distributed system.
Solution Approach 2:
Each host system performs its own storage processing functions independently without requiring services from other hosts or centralized controllers. Hosts autonomously handle compression, encryption, and data management tasks using their own computational resources, thereby avoiding the performance degradation caused by resource contention in shared architectures.
3Loss of time
If compute servers include non-volatile memory for buffering, then write latency is reduced, but server complexity and cost increase
Solution Approach 1:
The patent leverages the universal presence of non-volatile memory (such as SSDs or persistent memory) in modern server systems to provide buffering capabilities. Rather than adding specialized components, the system uses existing NVM resources in hosts to buffer data during processing, reducing write latency while avoiding significant increases in server complexity since NVM is already a standard component in contemporary servers.
4Speed
If storage and compute resources are tightly coupled in traditional systems, then data access speed is improved, but independent scalability is lost
Solution Approach 1:
The patent segments the storage system into independent compute nodes and storage nodes that communicate over a network. Compute-intensive tasks remain on host systems while storage operations occur on dedicated storage nodes, allowing each component to be scaled independently. Data access speed is maintained through efficient network communication and localized processing, while the system gains the flexibility to scale storage and compute resources separately based on different demands.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A data storage system includes a plurality of hosts (100), each of which includes at least one processor (111) and communicates over a network (200) with a plurality of storage nodes (300), at least one of which has at least one storage device (340-1, ..., 340-N), at least one storage controller 9320-1, 320-2, ..., 320-N), and at least one non-volatile memory (330). At least one process within a host issues data storage read/write requests. At least one of the hosts has a cache (113) for caching data stored in at least one of the storage nodes. The host writes data corresponding to a write request to at least one remote non-volatile memory (330) and carries out at least one storage processing function; data in the written-to node may then be made available for subsequent reading by a different one of the hosts. Examples of the storage processing function include compression, ECC computation, deduplicating, garbage collection, write logging, reconstruction, rebalancing, and scrubbing.