Data Maintenance Transaction Rollbacks During Compaction

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Cloud storage systems face inefficiencies and increased transaction failures due to inefficient data storage practices, such as duplicative files and fragmented storage, which are exacerbated by cleanup processes that can conflict with ongoing transactions.

Innovation Solution

A system that rolls back or reverses transactions that conflict with data compaction or re-arrangement operations, using a log to track transactions and determine conflicts, and applies rollback policies to prioritize transactions based on their importance and impact.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of substance

If storage cleanup processes are implemented to improve storage efficiency, then storage efficiency is improved, but transaction failure rate increases

Engineering Contradiction:

Improvestorage efficiencyVSAvoidtransaction failure rate

Core Design Contradiction:

Loss of substanceVSReliability

Solution Approach 1:

The system performs preliminary identification of transactions that are in progress before executing the cleanup process. By detecting ongoing transactions and pausing the cleanup operation for those specific files, the system prevents transaction failures while still allowing cleanup to proceed for files not involved in active transactions. This preliminary detection and selective pausing resolves the contradiction by maintaining both storage efficiency improvements and transaction reliability.

Inventive Principle:

Principle #10Preliminary action

2Volume of stationary object

If data compaction is performed to reduce storage space, then storage space utilization is improved, but transaction conflicts increase

Engineering Contradiction:

Improvestorage spaceVSAvoidtransaction conflicts

Core Design Contradiction:

Volume of stationary objectVSObject-affected harmful factors

Solution Approach 1:

The cleanup process applies different treatment to different files based on their transaction status. Files involved in ongoing transactions are excluded from cleanup operations, while files without active transactions undergo compaction and cleanup. This local differentiation allows the system to achieve storage space reduction through compaction while minimizing transaction conflicts by selectively applying cleanup only where safe.

Inventive Principle:

Principle #3Local quality

3Loss of substance

If cleanup processes operate on all files to improve storage efficiency, then storage efficiency is improved, but data loss risk increases

Engineering Contradiction:

Improvestorage efficiencyVSAvoiddata loss risk

Core Design Contradiction:

Loss of substanceVSObject-generated harmful factors

Solution Approach 1:

The system continuously monitors transaction status and provides feedback to the cleanup process. When a transaction is detected on a file slated for cleanup, the system pauses the cleanup operation for that file. This feedback mechanism ensures that data is not lost due to premature cleanup, while still allowing efficient cleanup of files that are safe to process, thus resolving the contradiction between storage efficiency and data loss risk.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS12430294B1Data maintenance transaction rollbacks

Publication Date: 2025.09.30 DATABRICKS INC

AI summary

The present application discloses a method, system, and computer system for managing a data in a storage system. The method includes receiving a first transaction that modifies or deletes first data stored in a storage system, determining that the first data is subject to an intervening re-arrangement transaction, and in response to determining that the first data is subject to the intervening re-arrangement transaction, rolling back the re-arrangement transaction at least with respect to the first data and committing the first transaction.