VLDB Data Scrubbing Using Chunked Parallel Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data scrubbing tools struggle with the scale and complexity of very large databases (VLDBs) in regulated industries, leading to performance bottlenecks, incomplete scrubbing, and compliance issues, while custom scripts lack robustness, consistency, and integration with industry-approved tools.

Innovation Solution

A method and system that segments VLDBs into manageable chunks based on row identifiers for parallel processing, using industry-approved scrubbing engines to ensure compliance, consistency, and efficient data scrubbing while maintaining data integrity and structure.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If existing data scrubbing tools are used on VLDBs, then data scrubbing can be performed, but performance bottlenecks occur and scrubbing becomes incomplete

Engineering Contradiction:

Improvedata scrubbing speedVSAvoidscrubbing completeness

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent divides the VLDB into multiple manageable chunks based on row identifiers, allowing parallel processing of multiple chunks simultaneously. This segmentation enables the scrubbing system to handle extremely large datasets without performance bottlenecks while maintaining complete coverage through systematic processing of all chunks.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces parallel processing across multiple chunks as an additional dimension to the traditional sequential scrubbing approach. By distributing the scrubbing workload across multiple concurrent instances processing different chunks, the system achieves both high speed and complete scrubbing coverage.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Adaptability or versatility

If custom scripts are used for data scrubbing, then flexibility is achieved, but robustness and consistency are insufficient

Engineering Contradiction:

Improvescrubbing flexibilityVSAvoidscrubbing robustness

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent introduces industry-approved scrubbing engines as intermediary components that bridge the flexibility of custom scripts with the robustness of validated tools. These engines provide consistent, reliable scrubbing operations while still allowing customization through configuration, ensuring both adaptability and robustness.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Object-affected harmful factors

If data scrubbing is performed on VLDBs, then sensitive information can be protected, but compliance with regulatory requirements becomes difficult to ensure

Engineering Contradiction:

Improvesensitive information protectionVSAvoidregulatory compliance

Core Design Contradiction:

Object-affected harmful factorsVSReliability

Solution Approach 1:

The patent uses industry-approved scrubbing engines as intermediaries that are specifically designed to meet regulatory requirements. These engines provide compliant scrubbing operations that protect sensitive information while ensuring adherence to regulatory standards through their validated methodologies and documentation.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Productivity

If parallel processing is implemented for VLDB scrubbing, then processing speed increases, but system complexity increases

Engineering Contradiction:

Improvescrubbing throughputVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments the database into chunks that can be processed in parallel, which increases throughput. The segmentation is based on row identifiers and is managed through a control component that coordinates the parallel instances, making the complexity manageable through structured organization rather than uncontrolled parallelization.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a control component as an intermediary that manages the parallel processing coordination. This control component handles the complexity of distributing and coordinating scrubbing operations across multiple instances, allowing the system to achieve high throughput without unmanageable complexity.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20260073079A1Data scrubbing for very large databases

Publication Date: 2026.03.12 WELLS FARGO BANK NA

AI summary

This disclosure relates to a method and system for data scrubbing in very-large-databases (VLDB) within regulated industries. The method involves segmenting a copy of a production database into manageable chunks based on row identifiers, allowing for parallel processing without database contention. Each chunk is processed by concurrently executing instances of a data scrubbing component, which replaces sensitive data with anonymous data of the same type using specified scrubbing algorithms. The system ensures that scrubbed data maintains the same structure and statistical properties as the original data while preventing the restoration of sensitive information. The process is meticulously documented to meet regulatory standards and facilitate audits, making it a robust solution for data scrubbing in environments with stringent compliance requirements.