Distributed Deletion Verification for Data Integrity

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Asynchronous distributed storage systems face challenges in ensuring data integrity during migration, particularly in verifying the deletion of data from the source data store before copying it to the target data store, due to the lack of trust among entities involved in the verification process.

Innovation Solution

Implementing a distributed deletion request processing system that involves an auditor, a deleter, and a listener to verify and authorize deletion requests, ensuring that multiple entities individually approve the deletion, using verification checks such as data set validation and metadata consistency, before executing the deletion.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is deleted from the source data store before verification of successful copying to the target data store, then storage efficiency and migration speed are improved, but data integrity and reliability deteriorate

Engineering Contradiction:

Improvemigration speedVSAvoiddata integrity

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system performs preliminary actions by copying data to the target data store before deleting from the source. The deletion is conditioned on successful verification of the copy, ensuring that data integrity is maintained while enabling efficient migration. This resolves the contradiction by establishing a sequence where the protective action (verification) occurs before the productivity-enhancing action (deletion).

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces an intermediary verification mechanism that mediates between the source and target data stores. This intermediary process validates the copied data and provides authorization for deletion, creating a trusted bridge that enables both high migration speed and data integrity. The intermediary acts as a gatekeeper that ensures reliability while facilitating productivity.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Device complexity

If verification checks are centralized in a single entity, then the verification process is simplified and faster, but trust and security among distributed entities deteriorate

Engineering Contradiction:

Improveverification process complexityVSAvoidtrust among entities

Core Design Contradiction:

Device complexityVSReliability

Solution Approach 1:

The verification process is segmented and distributed across multiple entities including the source data store, target data store, and intermediate verification services. Each entity performs specific verification checks independently, and their results are aggregated to form the overall verification outcome. This segmentation maintains security and trust while managing complexity through modular design.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system merges multiple verification results from different entities into a unified authorization decision. By combining the verification outcomes from the source data store, target data store, and intermediate services, the system achieves both distributed trust validation and streamlined decision-making, resolving the contradiction between complexity and reliability.

Inventive Principle:

Principle #5Merging (Combining)

3Reliability

If multiple entities perform verification checks before deletion, then data integrity and security are improved, but the verification process time and system complexity increase

Engineering Contradiction:

Improvedata integrityVSAvoidverification process time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The verification checks are performed continuously and asynchronously during the data migration process rather than as a sequential bottleneck. Multiple entities verify data integrity in parallel, and the useful action of verification continues throughout the migration, reducing overall verification time while maintaining comprehensive data integrity checks.

Inventive Principle:

Principle #20Continuity of useful action

Solution Approach 2:

The system performs verification checks at multiple levels and stages, applying partial verification early in the process and more comprehensive verification as needed. This staged approach ensures data integrity through multiple checks while minimizing the time impact by not requiring all verification steps to complete before migration begins.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11573925B2Techniques for distributing verification checks incident to data deletion

Publication Date: 2023.02.07 AMAZON TECH INC

AI summary

Techniques described and suggested herein include distributed deletion request processing and verification. For example, incident to migration of original data from a first data store to a second data store, verifications and confirmations related to removing the original data from the first data store may be performed so as to ensure the integrity of the original data represented on the second data store prior to removing the actual original data on the first data store. In some embodiments, the verifications and confirmations performed in connection with a deletion request may be apportioned to multiple entities, each of which may not fully trust the others. As a result, in some embodiments, a given deletion request may only be fulfilled if all of the entities involved in the verification process individually provide authorization to execute the deletion request.