Distributed Database Memory Eviction to File System
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed database systems face challenges in data synchronization, low latency over wide area networks, scalability, and managing different user schemas, particularly when handling large volumes of data that exceed memory constraints.
Innovation Solution
A distributed parallel processing database system persists table data in memory and a distributed file system, allowing data records and their history to be stored in memory and evicted as log files, with options for write-only or read-write access, enabling efficient handling of large datasets beyond conventional memory limits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data records are stored in memory for fast processing, then processing speed is improved, but memory capacity is limited and cannot handle very large datasets
Solution Approach 1:
The patent segments the database storage into multiple components: hot data stored in memory for fast access, warm data in local disk buffers, and cold data in remote storage. This segmentation allows the system to handle very large datasets by distributing them across different storage layers with varying access speeds and capacities.
Solution Approach 2:
The patent introduces a new dimension of storage capacity by utilizing remote storage systems connected via networks, extending the database system's data holding capacity beyond local memory and disk constraints. This allows handling of datasets that would otherwise exceed available local storage.
2Quantity of substance
If distributed file system is used for storing large datasets, then data capacity is improved, but access latency increases
Solution Approach 1:
The patent implements preliminary actions by pre-fetching data from remote storage to local buffers before it is needed for processing. Data is loaded into memory and local buffers in advance, so when queries need to access the data, it is already available locally, reducing access latency.
Solution Approach 2:
The patent introduces local disk buffers as intermediary storage between remote file systems and memory. These buffers act as mediators that cache frequently accessed data, reducing the frequency and volume of remote storage accesses, thereby lowering overall access latency.
3Loss of time
If data is kept in memory for quick access, then query response time is improved, but memory constraints prevent handling of hundreds of terabytes
Solution Approach 1:
The patent applies local quality by optimizing storage characteristics for different data access patterns. Frequently accessed data (hot data) is kept in memory with high-speed access characteristics, while less frequently accessed data is stored in local buffers or remote storage. This allows the system to handle very large tables while maintaining fast query response times for relevant data.
4Adaptability or versatility
If conventional RDBMS is replaced with distributed system, then scalability is improved, but data synchronization complexity increases
Solution Approach 1:
The patent merges multiple database nodes into a unified distributed system with a common storage layer. By combining local buffers and remote storage into a shared infrastructure, the system achieves scalability while reducing synchronization complexity, as the storage layer handles coordination and data consistency across nodes.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for a distributed parallel processing database system that persists table data in memory to a distributed file system. A distributed parallel processing database system persists table data in memory to a distributed file system. A parameter of creating a database table specifies that data records in the database table and history of changes to the data records can be stored in memory as well as in the distributed file system. When the database table is populated or otherwise modified, data records and the history in memory are evicted to the distributed file system as log files and removed from memory. The log files can be designated as write-only, where the data records, once written, cannot be read by structured query language (SQL) queries, or as read-write, where the data records, once written, can be read by SQL queries.