VLDB Data Scrubbing Using ROWID Chunking and Parallel Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data scrubbing tools struggle with the scale and complexity of very large databases (VLDBs) in regulated industries, leading to performance bottlenecks, incomplete scrubbing, and challenges in compliance, consistency, scalability, robustness, and integration with industry-approved tools.
Innovation Solution
A method and system that segments VLDBs into manageable chunks based on row identifiers (ROWIDs) for parallel processing, using industry-approved data scrubbing engines like Delphix, ensuring consistent data scrubbing across tables, maintaining data integrity, and comprehensive documentation for compliance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data scrubbing is applied to very large databases, then data security and compliance are improved, but processing time and system resources increase significantly
Solution Approach 1:
The patent divides the VLDB into multiple manageable chunks based on row identifiers (ROWIDs), allowing parallel processing of these chunks. This segmentation enables the scrubbing process to handle large datasets by processing smaller portions simultaneously, thereby reducing overall processing time while maintaining comprehensive data security coverage
Solution Approach 2:
The patent introduces parallel processing as an additional dimension to the traditional sequential scrubbing approach. By executing multiple scrubbing operations concurrently on different chunks of data, the system increases throughput and reduces processing time without compromising the thoroughness of security checks
2Reliability
If data scrubbing is applied to very large databases, then data security is improved, but data integrity and consistency may be compromised
Solution Approach 1:
By segmenting the database into chunks with specific ROWID ranges, the patent ensures that each chunk can be scrubbed independently while maintaining referential integrity through foreign key constraints. This segmentation approach allows thorough scrubbing of sensitive data while preserving the structural integrity and consistency relationships within the database
Solution Approach 2:
The patent implements validation mechanisms that provide feedback during the scrubbing process to ensure data integrity is maintained. The system validates scrubbed data against original data structures, constraints, and relationships, allowing corrections to be made if integrity issues are detected, thereby ensuring both security and integrity
3Reliability
If data scrubbing is applied to very large databases, then compliance with regulatory standards is improved, but complexity of the scrubbing process increases
Solution Approach 1:
The patent creates a universal scrubbing framework that can handle multiple data types and regulatory requirements through a single system. The framework provides standardized interfaces for different scrubbing algorithms and compliance validations, allowing the same infrastructure to serve multiple compliance purposes without increasing operational complexity
Solution Approach 2:
The patent works on copies of the production database rather than the original, allowing scrubbing operations to be performed on replicated data. This copying approach enables comprehensive compliance checking and scrubbing without affecting the original production systems, simplifying the overall process by isolating compliance operations from production workloads
4Object-affected harmful factors
If data scrubbing is applied to very large databases, then sensitive information protection is improved, but scalability of the solution decreases
Solution Approach 1:
The patent segments the database by ROWID ranges, which naturally scales with database size. This segmentation strategy allows the scrubbing system to handle increasingly large datasets by simply adding more chunks to the parallel processing pool, maintaining scalability while effectively protecting sensitive information across the entire VLDB
Solution Approach 2:
The patent leverages parallel processing as a scalability dimension, where additional computational resources can be allocated to process more chunks simultaneously. This approach allows the system to scale horizontally by adding parallel processing capacity rather than requiring proportional increases in sequential processing time or memory resources
Data Source
AI summary
This disclosure relates to a method and system for data scrubbing in very-large-databases (VLDB) within regulated industries. The method involves segmenting a copy of a production database into manageable chunks based on row identifiers, allowing for parallel processing without database contention. Each chunk is processed by concurrently executing instances of a data scrubbing component, which replaces sensitive data with anonymous data of the same type using specified scrubbing algorithms. The system ensures that scrubbed data maintains the same structure and statistical properties as the original data while preventing the restoration of sensitive information. The process is meticulously documented to meet regulatory standards and facilitate audits, making it a robust solution for data scrubbing in environments with stringent compliance requirements.