Distributed Memory Layer for Parallel Database Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In clusters of machines operating in parallel, determining the optimal storage and replication of data for query execution plans is complex, requiring consideration of data locality, transfer costs, and node load, while existing systems often rely on disk storage and lack efficient data placement strategies.
Innovation Solution
Implementing a distributed memory layer that allows data to be stored in main memory, enabling on-demand data movement and replication, with explicit data placement and intra-memory communication, and supporting DAG execution models for efficient job scheduling and data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored on disk storage facilities in distributed systems, then data capacity and persistence are improved, but data access speed and query execution efficiency deteriorate
Solution Approach 1:
The patent segments data storage into two distinct layers: a distributed file system layer for persistent long-term storage, and a distributed memory layer for high-speed access. This segmentation allows data to be stored on disk for capacity while maintaining fast access paths through memory replication, resolving the contradiction between storage capacity and access speed.
Solution Approach 2:
The distributed memory layer acts as an intermediary between the disk storage and the compute nodes. It buffers and caches frequently accessed data in memory, mediating between the slow but high-capacity disk storage and the fast but limited memory resources of individual nodes, thus improving access speed without sacrificing data capacity.
2Reliability
If data is replicated across multiple nodes for fault tolerance and parallel access, then system reliability and query parallelism are improved, but network communication overhead and data transfer costs increase
Solution Approach 1:
The patent implements local quality by allowing each node to maintain local copies of frequently accessed data in its distributed memory layer. This enables nodes to access data locally without network communication, improving reliability and parallelism while minimizing network overhead. Data is replicated selectively based on access patterns rather than uniformly across all nodes.
Solution Approach 2:
Instead of fully replicating all data across all nodes, the system performs partial replication only for hot data that requires fast access. This selective replication approach provides sufficient reliability and parallelism for frequently accessed data while avoiding the excessive network communication costs of complete replication.
3Productivity
If a distributed memory layer is implemented for fast data access, then query execution efficiency is improved, but system complexity and data management overhead increase
Solution Approach 1:
The distributed memory layer is designed with multi-functionality to handle various operations uniformly: data storage, data retrieval, data replication, and coordination with the underlying file system. This universal interface simplifies the complexity by providing a single abstraction layer that handles multiple functions rather than requiring separate mechanisms for each operation.
Solution Approach 2:
The system implements self-service through automatic data placement and replication policies. The distributed memory layer autonomously manages which data to replicate, where to place it, and when to evict it, based on access patterns and system state. This self-management reduces the complexity of manual data administration and simplifies the overall system operation.
Data Source
AI summary
A method and system, the system including a plurality of machines each having a processor and a main memory component; a shared distributed storage facility storing a set of data and accessible by the plurality of machines over a communication network; a controller to select, in response to a state of a query execution plan comprising a plurality of executable jobs for the set of data, which one of a set of scheduling algorithms to execute; an execution engine to execute the selected scheduling algorithm to determine, for each job in the plurality of jobs, which server to schedule to execute the respective job; and providing an indication of the scheduling of the servers determined to be schedules for the execution of the jobs.