Distributed Storage with Local Cache and Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems face performance bottlenecks due to network latency and complexity in data access, especially in multi-tenant cloud environments, where centralized storage options like NAS and SAN systems are slower than local storage and often incompatible with various operating systems, and distributed file systems incur additional latency from spreading data across multiple physical hosts.
Innovation Solution
Implementing a high performance space efficient distributed storage system with a local cache on each host, where a storage controller redirects access to a distributed storage volume, allowing lazy updates and performing compression and deduplication in the background to reduce data access latency and increase storage efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in centralized storage systems (NAS/SAN), then storage capacity is consolidated, but data access latency increases and compatibility with various operating systems is reduced
Solution Approach 1:
The storage system is segmented into distributed storage nodes across multiple hosts, where each host maintains local storage resources. This segmentation allows data to be physically distributed while maintaining centralized management capabilities, thereby reducing access latency by keeping data closer to where it is needed while preserving overall storage capacity.
Solution Approach 2:
A storage controller acts as an intermediary between the distributed storage nodes and the hosts. It manages data placement, replication, and access coordination across the distributed system, enabling centralized storage capabilities to be achieved without the performance penalties of traditional centralized NAS/SAN systems.
2Quantity of substance
If data is spread across multiple physical hosts in a distributed file system, then storage capacity is increased and compatibility is improved, but data access latency increases due to network communication
Solution Approach 1:
Each host in the distributed system is equipped with local cache memory that provides high-speed access to frequently accessed data. This local quality enhancement allows the system to maintain distributed storage capacity while reducing network latency by serving common data access requests from local caches rather than remote storage nodes.
Solution Approach 2:
The system performs preliminary actions by pre-caching frequently accessed data blocks in the local memory of each host before they are actually needed. This anticipatory caching strategy reduces data access latency by having data ready locally before access requests arrive, while still maintaining the benefits of distributed storage capacity.
3Reliability
If duplicate copies of files are stored across multiple hosts, then data redundancy and reliability are improved, but storage space efficiency is reduced
Solution Approach 1:
Instead of storing complete duplicate copies of files across multiple hosts, the system stores only unique data blocks and maintains references or pointers to these blocks from multiple locations. This copying strategy achieves data redundancy and reliability through reference management while dramatically improving storage space efficiency by eliminating actual data duplication.
Solution Approach 2:
The distributed storage system implements a universal reference mechanism that can represent multiple data copies through a single stored block. This multi-functional approach allows the same data block to serve multiple hosts and purposes simultaneously, achieving both redundancy for reliability and space efficiency through shared storage resources.
Data Source
AI summary
Space efficient distributed storage systems are disclosed. For example, A system comprising a distributed storage volume (DSV) deployed on a plurality of hosts, the DSV comprising logical volumes, the logical volumes deployed on physical storage devices; and a first host of the plurality of hosts with a local cache, and a storage controller, the storage controller executing on a processor to receive a request relating to a first file; query the DSV to determine whether a second file that is a copy of the first file is stored in the DSV; and based on determining from the querying that the second file resides in a logical volume of the logical volumes in the DSV, store a separate reference to the second file in at least one logical volume of the DSV, wherein the separate reference is a virtual reference or link to the second file.