LSM-Tree SSTable Compaction Offload for CPU Contention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Log Structured Merge Tree (LSM-Tree) data storage structure causes significant reading and writing amplification, CPU contention, and data transfer delays due to frequent compaction operations, degrading application performance and shortening the lifetime of storage devices like SSDs.
Innovation Solution
A data compaction method and device that performs merge sorting and index reconstruction on SSTable files stored based on LSM-Tree, offloading CPU-intensive tasks to storage devices, reducing data transfer and CPU resource requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If compaction operations are performed frequently on the CPU to maintain LSM-Tree data structure, then data storage consistency is improved, but CPU resource usage increases and data transfer delay occurs
Solution Approach 1:
The patent introduces an intermediary mechanism where the storage device's processor acts as a mediator between the host CPU and the compaction operation. The storage device receives compaction instructions from the host, executes the merge sorting and data block reconstruction locally using its own processor, and returns the compacted results to the host. This intermediary approach allows the host CPU to offload compaction tasks, reducing CPU resource usage and data transfer delays while maintaining data storage consistency through coordinated control.
2Reliability
If compaction operations are performed frequently on the CPU to maintain LSM-Tree data structure, then data storage consistency is improved, but CPU contention increases
Solution Approach 1:
The storage device's processor serves as an intermediary that handles compaction operations locally, eliminating the need for the host CPU to directly perform merge sorting and data block reconstruction. The host CPU only needs to send compaction instructions and receive results, significantly reducing CPU contention and system complexity while maintaining data storage consistency through the coordinated intermediary mechanism.
Solution Approach 2:
The storage device performs compaction operations autonomously using its own processor, implementing self-service functionality. The storage device receives compaction instructions from the host, executes the merge sorting algorithm and data block reconstruction independently, and returns the compacted SSTable files to the host. This self-service approach eliminates CPU contention on the host while maintaining data storage consistency through structured communication protocols.
3Reliability
If compaction operations are performed frequently on the CPU to maintain LSM-Tree data structure, then data storage consistency is improved, but transfer resources are consumed
Solution Approach 1:
The storage device's processor acts as an intermediary that performs compaction operations locally, reducing the need for extensive data transfer between the host CPU and storage device. The host CPU sends only compaction instructions and receives final compacted results, minimizing transfer resource consumption while maintaining data storage consistency through coordinated control.
Solution Approach 2:
The storage device autonomously performs compaction operations using its own processor, executing merge sorting and data block reconstruction without requiring the host CPU to process the data. This self-service mechanism significantly reduces transfer resources by keeping the compaction workload localized to the storage device, while data storage consistency is maintained through structured instruction and result exchange.
Data Source
AI summary
Provided is a data compaction method that includes: determining, by an electronic device, a plurality of Sorted String Table (SSTable) files on which a compaction operation is to be performed among SSTable files stored in a storage device connected to the electronic device, the data in the SSTable files stored in the storage device is stored based on a Log Structured Merge Tree (LSM-Tree) mechanism; based on receiving the plurality of SSTable files from the storage device, performing, by the electronic device, merge sorting on data in data blocks of the plurality of SSTable files based on indexes of the data in the data blocks of the plurality of SSTable files, to obtain reconstructed data blocks; and sending, by the electronic device, the reconstructed data blocks to the storage device, the reconstructed data blocks being used by the storage device to obtain reconstructed SSTable files.