Targeted Sweep Thread Scaling for Stale Version Deletion

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data management systems face inefficiencies in deleting stale data versions, leading to excessive storage requirements, memory usage, and performance degradation, particularly when dealing with large amounts of short-lived data and long-running transactions.

Innovation Solution

A system is implemented to manage a targeted sweep of stale data across distributed computing nodes, dynamically determining the number of threads for deletion based on historical performance and resource availability, allowing parallel deletion of stale data across fine partitions and reducing redundant deletion attempts.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If multiple versions of data are stored separately without overwriting old versions, then database access contention is reduced and data integrity is improved, but storage requirements continuously increase and can get out of control

Engineering Contradiction:

Improvedata integrityVSAvoidstorage requirements

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent implements a version retention policy that automatically discards stale data versions after they have served their purpose. The system tracks version ages and selectively removes versions that are no longer needed for consistency or recovery, thereby recovering storage space while maintaining data integrity for active versions.

Inventive Principle:

Principle #34Discarding and recovering

Solution Approach 2:

The system dynamically adjusts version retention based on activity patterns and storage conditions. Rather than using a static retention policy, the system adapts the number of retained versions and their lifecycle based on actual usage, allowing optimal balance between reliability and storage efficiency under varying conditions.

Inventive Principle:

Principle #15Dynamics

2Quantity of substance

If a mechanism is implemented to remove stale versions of data, then storage requirements are controlled, but the mechanism may fall behind when waiting for long running transactions or database unavailability

Engineering Contradiction:

Improvestorage requirementsVSAvoiddeletion efficiency

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The system performs preliminary identification of stale versions before actual deletion occurs. By pre-marking versions for deletion based on retention policies and transaction status, the system prepares deletion candidates in advance, allowing faster execution when resources become available without compromising data integrity.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The deletion mechanism operates continuously in the background rather than waiting for transactions to complete. The system maintains a continuous sweep process that identifies and deletes stale versions as soon as they become safe to remove, ensuring storage management keeps pace with data generation without blocking operational transactions.

Inventive Principle:

Principle #20Continuity of useful action

3Reliability

If the deletion mechanism waits for transactions to complete before removing stale versions, then data consistency is maintained, but memory and disk usage increase and performance degrades

Engineering Contradiction:

Improvedata consistencyVSAvoidperformance

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system implements feedback mechanisms that monitor transaction status, version age, and storage conditions to dynamically adjust deletion timing. By continuously gathering feedback on system state, the deletion mechanism can determine the optimal moment to remove stale versions, maintaining consistency while preventing excessive resource consumption from retained versions.

Inventive Principle:

Principle #23Feedback

4Productivity

If parallel deletion threads are increased to improve deletion speed, then storage cleanup efficiency is improved, but system complexity and resource management become more difficult

Engineering Contradiction:

Improvedeletion speedVSAvoidthread management complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The deletion workload is segmented into independent tasks that can be executed in parallel. By dividing the version space into partitions or ranges assigned to different threads, the system achieves parallel deletion efficiency while keeping each thread's management simple and manageable through clear task boundaries.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12561300B1Automatic scaling for targeted sweep

Publication Date: 2026.02.24 PALANTIR TECHNOLOGIES INC

AI summary

A system for scaling parallelism in sweeping stale data associated with multi-version data management is disclosed. The system is programmed to manage a targeted sweep of stale data stored on distributed storage associated with a set of computing nodes, where the stale data includes metadata for database transactions. The system is programmed to dynamically determine, on a computing node, how many threads to simultaneously perform deletion for the targeted sweep based on historical performance of the computing nodes, historical performance of the distributed storage associated with the computing nodes, or other factors. The system is programmed to then cause a certain number of threads to run on the computing node, where each thread identifies and deletes specific metadata for data transactions within the scope of the targeted sweep.