Lock-Free Data Retention Index for Concurrent Storage Management

Solution Overview

Problem

In multi-core processing systems, managing data retention and removal efficiently while handling concurrent access by multiple clients or processes is challenging due to dependencies on data objects, leading to storage space issues and performance bottlenecks.

Innovation Solution

Implementing a lock-free data structure for a data retention index that allows multiple processes to independently and concurrently access and update the index, ensuring no process blocks access, thereby reducing processing time and improving storage reclamation efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If traditional locking mechanisms are used to manage data retention index updates in multi-core systems, then data consistency is maintained, but processing speed and system performance deteriorate due to blocking operations

Engineering Contradiction:

Improvedata consistencyVSAvoidprocessing speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent replaces traditional mechanical locking mechanisms with a lock-free data structure implementation. Instead of using mutexes or locks to ensure data consistency during index updates, the system employs atomic operations and memory barriers provided by the hardware memory model to achieve the same consistency guarantees without blocking other processes. This substitution eliminates the performance bottleneck while maintaining reliability.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent introduces an intermediary layer in the form of a specially designed data structure that mediates access to the retention index. This data structure uses atomic compare-and-swap operations as intermediaries to coordinate updates from multiple processes simultaneously, allowing concurrent access without traditional locks while ensuring that only valid updates are applied to the index.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If concurrent access to data retention index is allowed without locking, then processing time is reduced, but data consistency and reliability worsen due to potential conflicts

Engineering Contradiction:

Improveprocessing timeVSAvoiddata consistency

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent replaces mechanical locking mechanisms with a lock-free data structure implementation. Instead of using mutexes or locks to ensure data consistency during index updates, the system employs atomic operations and memory barriers provided by the hardware memory model to achieve the same consistency guarantees without blocking other processes. This substitution eliminates the performance bottleneck while maintaining reliability.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent introduces an intermediary layer in the form of a specially designed data structure that mediates access to the retention index. This data structure uses atomic compare-and-swap operations as intermediaries to coordinate updates from multiple processes simultaneously, allowing concurrent access without traditional locks while ensuring that only valid updates are applied to the index.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Quantity of substance

If storage space is aggressively reclaimed in multi-core systems, then storage efficiency is improved, but system performance worsens due to increased contention and locking overhead

Engineering Contradiction:

Improvestorage efficiencyVSAvoidsystem performance

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The patent replaces mechanical locking mechanisms with a lock-free data structure implementation. Instead of using mutexes or locks to ensure data consistency during index updates, the system employs atomic operations and memory barriers provided by the hardware memory model to achieve the same consistency guarantees without blocking other processes. This substitution eliminates the performance bottleneck while maintaining reliability.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent introduces an intermediary layer in the form of a specially designed data structure that mediates access to the retention index. This data structure uses atomic compare-and-swap operations as intermediaries to coordinate updates from multiple processes simultaneously, allowing concurrent access without traditional locks while ensuring that only valid updates are applied to the index.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS10872073B1Lock-free updates to a data retention index

Publication Date: 2020.12.22 AMAZON TECH INC

AI summary

A lock-free data structure may be used to provide concurrent access to a data retention index for a data store. Different processes may obtain keys to the data retention index from the lock-free data structure. The different processes may update entries in the data retention index corresponding to the keys to identify data objects to retain in the data store. The data retention index may be evaluated to determine data objects not identified in the data retention index and reclaim portions of the data store that store the determined data objects for storing other data.