Database Disk Data Movement Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for migrating large databases across hosts result in significant downtime and revenue loss due to inefficient data movement and synchronization processes, particularly when contemporaneous reading and writing to disks reduce throughput and fail to utilize available network resources effectively.
Innovation Solution
A method and system that restrict disks to perform only one operation at a time, utilizing configuration information to plan and execute data movement efficiently by prioritizing smaller files, estimating network throughput, and utilizing available subnets, with a master coordinating data transfer through worker threads and NIC queues to ensure parallel data movement without simultaneous reading and writing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is migrated from host to host with synchronization, then reliability is improved, but migration time increases significantly
Solution Approach 1:
The data migration process is segmented into distinct phases: initial data copy, incremental synchronization, and final verification. This allows the system to maintain reliability through multiple copies while reducing total migration time by parallelizing operations and avoiding full synchronization of every data block.
Solution Approach 2:
The system performs preliminary data copying and establishes mirror copies before final synchronization is complete. This preliminary action allows the new primary host to be prepared in advance, reducing the actual downtime when data movement is finalized.
2Productivity
If disks perform simultaneous reading and writing operations, then productivity is improved, but data throughput decreases
Solution Approach 1:
Disk operations are segmented into dedicated read phases and write phases. During data migration, disks perform only write operations to destination locations, while source disks perform only read operations. This segmentation eliminates the throughput penalty of simultaneous read-write operations while maintaining high productivity through coordinated parallel operations.
Solution Approach 2:
The system uses periodic action by alternating between read-heavy and write-heavy phases in a coordinated manner. Source disks alternate between reading data to be migrated and waiting for write completion, while destination disks alternate between receiving data and preparing for the next transfer cycle. This periodic coordination maintains high throughput.
Data Source
AI summary
A method, article of manufacture, and apparatus for processing information are disclosed. In some embodiments, this includes collecting information about a database, determining a first disk to read from based on the collected information, determining a second disk to write to based on the collected information, and performing a write operation on the second disk. In some embodiments, a read operation may be performed on the first disk contemporaneously with the write operation on the second disk, and the read operation may be the only operation performed on the first disk at a time.