Distributed Data Delivery with Quorum Reads and Immutable Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed data storage systems face challenges in efficiently managing heavy read access data that is rarely modified, particularly in ensuring fast and reliable delivery to client-side applications while minimizing computational resources and storage capacity.
Innovation Solution
A distributed computer system that stores immutable data blocks on a database distant to the client, using a data distribution entity to split data into immutable blocks, replicating them across multiple storage nodes, and employing a fuse daemon to perform quorum reads, fetch the fastest responses, and generate a virtual file for direct client access, while utilizing a common middleware client library for efficient communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored on a distant database and accessed over a network, then data reliability and scalability are improved, but data access speed and computational efficiency deteriorate
Solution Approach 1:
The system performs preliminary actions by caching frequently accessed immutable data blocks in local storage before they are actually needed by client applications. The fuse daemon pre-loads data blocks into the page cache based on access patterns, so that when client applications request data, it can be served from local cache rather than fetching from the distant database, thus resolving the contradiction between remote storage reliability and local access speed.
2Reliability
If data is replicated across multiple storage nodes, then data availability and reliability are improved, but storage capacity requirements and system complexity increase
Solution Approach 1:
The system segments data into immutable data blocks that can be independently replicated across multiple storage nodes. Each data block is treated as a separate unit that can be cached and managed independently in the fuse daemon's page cache. This segmentation allows the system to replicate only necessary data blocks across nodes rather than entire files, reducing overall storage complexity while maintaining data availability through selective replication of hot data blocks.
3Speed
If immutable data blocks are cached in page cache, then data access speed is improved, but memory consumption increases
Solution Approach 1:
The system changes the parameter of data block immutability to enable efficient caching. By declaring data blocks as immutable once written, the fuse daemon can cache them in the page cache without worrying about modifications, allowing the operating system's page cache mechanisms to efficiently manage memory usage. This immutability parameter change enables the system to cache frequently accessed data blocks in memory, improving access speed while the OS handles memory reclamation when needed.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A distributed computer system for delivering data to client-side application(s) is provided. The system includes a database configured to store immutable data blocks, a data distribution entity configured to split source-data into immutable data blocks and metadata. The data distribution entity is configured to replicate and store the data blocks on different storage nodes(s) of the database. The metadata comprises values referencing the data blocks for a key-value database call. The system further comprises a data fetching/delivering entity with a fuse-daemon configured to form a quorum read request for data block(s) out of a client-side request for a certain data range. The quorum-read request is a bundle of parallel requests to different storage nodes. The fuse-daemon is configured to fetch the data blocks delivered in the fastest response and to discard the rest. The fuse-daemon generates a virtual file comprising the corresponding range of data from the fetched data blocks.