Distributed Database Split Volume Redo Change Log Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In modern distributed cloud services, the I/O bottleneck shifts from individual disks and nodes to the network between the database tier and the storage tier, causing performance issues due to amplified traffic and variability in storage node performance.
Innovation Solution
Implementing a distributed database system that utilizes split volume types for redo log records and change log records, where redo log data is stored separately from change log data, allowing for efficient transactional semantics and eliminating bottlenecks by reducing network traffic and enabling scalable database operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If redo log data and change log data are stored together in the same volume, then storage management is simplified, but network traffic is amplified and performance bottlenecks occur due to the need to retrieve both types of data for transaction operations
Solution Approach 1:
The patent divides the storage system into separate volumes: one dedicated to redo log data and another to change log data. This segmentation allows independent management and retrieval of each log type, reducing network traffic amplification when retrieving logs for transaction operations, as only the necessary log type needs to be accessed from its dedicated volume.
2Ease of operation
If the database engine handles all backup, restore, and space management operations, then operational control is centralized, but the database engine becomes a bottleneck and scalability is limited
Solution Approach 1:
The patent extracts backup, restore, and space management operations from the database engine and transfers them to the storage tier. The storage tier autonomously manages these operations using metadata stored in the change log, eliminating the database engine as a bottleneck and enabling independent scaling of storage capacity and performance.
Solution Approach 2:
The storage tier is designed to autonomously perform backup, restore, and space management operations without requiring database engine intervention. The storage system uses metadata from change logs to self-manage these operations, allowing the storage tier to serve itself and reducing dependency on the database engine for scalability.
3Reliability
If the database engine intervenes in every data retrieval operation to ensure transactional semantics, then data consistency is maintained, but network traffic increases and response time deteriorates
Solution Approach 1:
The patent implements preliminary action by having the storage tier pre-process and validate log retrieval operations using metadata from change logs. The storage tier autonomously determines which logs need to be retrieved and ensures transactional semantics are maintained before data is requested by the database engine, eliminating the need for database engine intervention in every retrieval operation and reducing response time.
Data Source
AI summary
Techniques for distributed database storage utilizing split volume types for redo log records and change log records are described. A database engine sends redo log data to a first set of page storage nodes and change log data to a second set of tuple storage nodes, where the change log data is stored in an indexed manner Client access to change log data may be provided through a separate set of streaming storage servers instead of through the database engine.