Distributed Database File Slicing for Parallel Page-Server Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed database systems face scalability and performance issues due to the close coupling of logical and physical file storage, particularly in scan-intensive analytical workloads, leading to inefficient data movement and storage I/O limitations.
Innovation Solution
Decouple logical database files from physical storage by slicing them into slices and distributing these slices across multiple page servers, using endpoint mappings to manage storage and enabling parallel I/O operations through striped and strided cells.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If logical and physical storage are closely coupled, then storage simplicity is maintained, but scalability and performance in scan-intensive analytical workloads deteriorate
Solution Approach 1:
The patent segments the database file into multiple slices that are distributed across different page servers. Each slice can be independently stored and accessed, enabling parallel I/O operations and improving scalability for scan-intensive workloads while maintaining manageable complexity through structured distribution.
Solution Approach 2:
The patent introduces an intermediary layer between logical and physical storage by using page servers as mediators. The compute node stores only the logical database file and endpoint mappings, while page servers handle the physical storage and data retrieval, decoupling the logical and physical layers to enable independent scaling and optimization.
2Productivity
If data is stored on page servers for analytical workloads, then storage accessibility is improved, but data movement overhead increases
Solution Approach 1:
By segmenting the database into slices distributed across multiple page servers, the system enables parallel data retrieval and processing. This segmentation allows scan-intensive operations to access data from multiple locations simultaneously, improving transaction throughput while reducing the data movement overhead associated with consolidating all data at a single location.
3Reliability
If replica re-creation is performed on entire database files, then storage redundancy is ensured, but system downtime increases
Solution Approach 1:
The patent segments the database file into smaller slices that can be independently replicated across page servers. When a replica re-creation is needed, only the affected slices are copied rather than the entire database file, significantly reducing system downtime while maintaining storage redundancy through distributed slice replication.
Solution Approach 2:
The system maintains endpoint mappings that track which page servers store which slices, enabling rapid replica re-creation by pre-establishing the mapping relationships. When replication is needed, the system can quickly identify and copy only the necessary slices to the appropriate page servers based on these pre-configured mappings, reducing the time required for replica re-creation.
Data Source
AI summary
Distributed database systems including compute nodes and page servers are described herein that enable separating logical and physical storage of database files in a distributed database system. A distributed database system includes a page server and a compute node, and is configured to store a logical database file that includes data and is associated with a file identifier. Each page server is configurable to store slices (i.e., subportions) of the logical database file. The compute node is coupled to the plurality of page servers and configured to store the logical database file responsive to a received command. In an aspect, such storage may comprise slicing the data comprising the logical database file into a set of slices with each being associated with a respective page server, maintaining an endpoint mapping for each slice of the first set of slices, and transmitting each slice to the associated for storage thereby.