Background Dataset Rewrites for Compliant Data Deletion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems face challenges in efficiently and compliantly handling large datasets for physical deletion in multi-vendor environments, particularly in adhering to regulations like GDPR and CCPA, due to the immutability of storage systems and the operational difficulties of continuous background maintenance.
Innovation Solution
A data access platform that automates the physical deletion of data by queuing files based on staleness tolerance, prioritizing treatments, and performing background maintenance to ensure compliance with regulations, using a planner-worker system to manage and prioritize file rewrites.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is physically deleted from storage systems to comply with GDPR and CCPA regulations, then regulatory compliance is achieved, but storage system immutability prevents complete data removal
Solution Approach 1:
The patent segments the data deletion process into multiple independent components: (1) identifying files containing PII through scanning and metadata analysis, (2) queueing identified files for treatment, (3) rewriting files to remove PII while preserving structure, and (4) verifying deletion completeness. This segmentation allows each component to address specific challenges of storage system immutability independently.
Solution Approach 2:
The patent implements preliminary actions by continuously scanning and identifying files containing PII before regulatory deadlines, pre-queueing these files for treatment, and proactively rewriting them to remove sensitive information. This preliminary action ensures compliance is achieved before requests are made, rather than reacting to deletion requests after data has been collected.
2Adaptability or versatility
If manual configuration and rebuilding of application services is performed to adapt to different deployment parameters, then customization for each service is achieved, but time consumption and operational complexity increase
Solution Approach 1:
The patent creates a universal data access platform that serves multiple application services simultaneously. The platform implements a common interface layer that handles data access, PII identification, and deletion operations for diverse services including batch processing, stream processing, and interactive query services. This universal platform eliminates the need to manually configure and rebuild separate data management systems for each application service.
Solution Approach 2:
The patent introduces an intermediary data access platform layer between storage systems and application services. This intermediary handles all data access operations, PII identification, and deletion tasks, allowing application services to focus on their core functionality without needing to implement their own data management and compliance mechanisms. The intermediary translates diverse service requirements into unified data access operations.
3Reliability
If continuous background maintenance is implemented to physically delete data, then regulatory compliance is maintained, but operational burden and system complexity increase
Solution Approach 1:
The patent implements self-service mechanisms where the system automatically scans for PII, identifies files requiring deletion, queues them for treatment, and executes rewriting operations without human intervention. The platform monitors its own compliance status and autonomously maintains data deletion schedules, reducing operational burden while ensuring continuous regulatory compliance.
Solution Approach 2:
The patent establishes continuous background maintenance processes that continuously scan for PII, identify files requiring treatment, and rewrite them as they are discovered or according to scheduled intervals. This continuous action ensures compliance is maintained at all times rather than being periodic, while the automated nature of the process reduces operational complexity compared to manual batch operations.
4Reliability
If large datasets are processed for physical deletion, then complete data removal is achieved, but processing time and computational resources increase
Solution Approach 1:
The patent segments large datasets into smaller manageable files and processes them independently through the queueing system. Each file is scanned, evaluated for PII content, and rewritten separately, allowing parallel processing of multiple files simultaneously. This segmentation enables complete data removal from large datasets while maintaining practical processing speeds through concurrent operations.
Solution Approach 2:
The patent implements partial action by scanning file metadata and content samples to identify files containing PII before committing to full rewriting operations. This preliminary identification step filters out files that do not require treatment, avoiding unnecessary processing of large datasets that contain no sensitive information, thereby improving overall processing efficiency while maintaining complete removal of PII from relevant files.
Data Source
AI summary
Various embodiments of the present technology generally relate to management of big data storage and the physical removal of data via data access systems for large data processing environments having multiple application services and multiple storage services. In some embodiments, a method of physically removing data from a storage system provides for identifying one or more files needing data removal treatment, determining that a file needing data removal treatment should be queued, and populating a queue with the file. Determining that a file should be queued is based, at least in part, on a staleness tolerance. The method further provides for treating the file and replacing a previous version of the file in storage with the updated file. In some implementations, treating the file includes removing data from the file to create an updated file and may further include additional changes to the file.