Merge-Based Tile Architecture for Database Compaction Offload

Solution Overview

Problem

State-of-the-art compaction schemes for large databases, such as those using the RocksDB key-value data storage engine, are computationally intensive, requiring significant processing resources and taking days or weeks to complete, thereby diverting resources from primary jobs.

Innovation Solution

A hardware-based compaction accelerator utilizing a merge tile architecture, comprising decoders, a merge iterator, and a compaction module, operates independently to reduce the computational burden on the database system by converting SST files to a key-value format, merging and compacting data, and encoding back into SST files, using hardware components like FPGAs or ASICs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If software-based compaction schemes are used, then the database system can perform compaction operations, but the processing resources are significantly consumed and compaction takes days or weeks

Engineering Contradiction:

Improvecompaction speedVSAvoidprocessing resource consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent replaces software-based compaction processing with a hardware-based compaction accelerator that uses dedicated circuitry (FPGA or ASIC) to perform compaction operations. This substitution of hardware for software execution dramatically increases processing speed and reduces resource consumption, as the hardware accelerator can perform compaction in parallel with database operations without diverting processor assets.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent introduces a hardware compaction accelerator as an intermediary component between the database system and storage devices. This accelerator includes dedicated modules for reading SST files, performing compaction operations, and writing compacted files, thereby offloading the computationally intensive compiction task from the main database processing system.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Loss of substance

If compaction operations are performed on large databases, then data redundancy is reduced, but the operation may take days or weeks completing

Engineering Contradiction:

Improvedata redundancyVSAvoidcompaction completion time

Core Design Contradiction:

Loss of substanceVSLoss of time

Solution Approach 1:

The patent replaces software-based compaction processing with a hardware-based compaction accelerator that uses dedicated circuitry (FPGA or ASIC) to perform compaction operations. This substitution of hardware for software execution dramatically increases processing speed and reduces resource consumption, as the hardware accelerator can perform compaction in parallel with database operations without diverting processor assets.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The hardware compaction accelerator is pre-configured with dedicated functional units for reading SST files, performing compaction, and writing output files. This preliminary setup of hardware pathways and logic enables immediate high-speed processing when compaction is initiated, eliminating the need for software interpretation and setup overhead.

Inventive Principle:

Principle #10Preliminary action

3Productivity

If processor assets are redirected to facilitate compaction, then compaction can be performed, but resources are diverted from primary jobs

Engineering Contradiction:

Improvecompaction capabilityVSAvoidresource availability for primary jobs

Core Design Contradiction:

ProductivityVSEase of operation

Solution Approach 1:

The patent segments the compaction function into a separate, independent hardware accelerator unit that operates independently from the main database processing system. This segmentation allows the database system to continue its primary operations while the hardware accelerator handles compiction in parallel, ensuring that processor assets are not diverted from primary jobs.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a hardware compaction accelerator as an intermediary component between the database system and storage devices. This accelerator includes dedicated modules for reading SST files, performing compaction operations, and writing compacted files, thereby offloading the computationally intensive compiction task from the main database processing system.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12380072B1Method and system for performing a compaction/merge job using a merge based tile architecture

Publication Date: 2025.08.05 MARVELL ASIA PTE LTD

AI summary

A hardware-based compaction accelerator may comprise two or more decoders, a merge iterator, a compaction module, and an encoder. Each of the decoders converts a sorted string table (SST) files into a corresponding key-value (KV) format data stream. The merge iterator receives a KV format data stream from each of the decoders, and combines the KV format data streams into a single KV format data stream. The compaction module receives the composite KV format data stream and produces a compacted data stream. The compacted data stream contains less data that is in the composite KV format data stream. The encoder converts the composite KV format data stream back into one or more output SST files. The compaction accelerator may be configured to perform only a subset of the processing available from the decoders, merge iterator, compaction module, and encoder, and may be configured through the Internet using a cloud-based processor.