Background Dataset Maintenance for Automated Data Deletion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems face challenges in efficiently and efficiently managing large-scale data processing across multiple application and storage services, particularly in adhering to regulations like GDPR and CCPA, due to the need for manual configuration and the operational complexity of physically deleting user data from large datasets, which can take weeks to complete and require significant compute resources.
Innovation Solution
A data access platform that automates the physical deletion of user data by queuing files based on staleness tolerance, prioritizing treatments, and performing background maintenance, allowing users to specify treatment transformations and staleness requirements, thereby reducing operational burden and ensuring compliance with regulations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual configuration and physical deletion methods are used for data management, then data can be removed from storage, but the process is time-consuming and requires significant operational resources
Solution Approach 1:
The system performs preliminary actions by identifying and queuing files for treatment before actual data deletion occurs. Files are prioritized and prepared in advance based on staleness tolerance criteria, so that when deletion is needed, the system can process them efficiently without delays caused by real-time identification and manual configuration.
Solution Approach 2:
The background dataset maintenance system operates autonomously to identify, queue, and treat files requiring data removal. The system self-manages the entire workflow from file identification through treatment prioritization to execution, eliminating the need for manual configuration and reducing operational resource requirements while maintaining compliance with data retention policies.
2Reliability
If comprehensive data treatment is performed across all files, then regulatory compliance is achieved, but operational complexity and resource consumption increase
Solution Approach 1:
The system applies different treatment priorities to different files based on their specific characteristics and staleness tolerance levels. Rather than treating all files uniformly, the system identifies which files require immediate attention versus those that can wait, applying localized quality control to each file based on its individual needs and compliance requirements.
Solution Approach 2:
The system changes the parameter of staleness tolerance to determine treatment priority. By adjusting this parameter, the system can dynamically prioritize files for treatment based on how long they have been stale, ensuring that files exceeding compliance thresholds are handled first while reducing operational complexity through automated parameter-based decision-making.
3Reliability
If files are processed immediately for data removal, then compliance is maintained, but system performance and other operations are impacted
Solution Approach 1:
The system performs data treatment operations periodically through background maintenance processes rather than immediately when compliance thresholds are crossed. Files are queued for treatment and processed at scheduled intervals, allowing the system to maintain compliance while minimizing disruption to primary operations and overall system performance.
Solution Approach 2:
The queue system acts as an intermediary between file identification and actual treatment execution. Files waiting for treatment are held in the queue, decoupling the compliance monitoring function from the data treatment function. This allows the system to maintain compliance tracking while scheduling treatments during periods of lower system load, preserving operational efficiency.
Data Source
AI summary
Various embodiments of the present technology generally relate to management of big data storage and the physical removal of data via data access systems for large data processing environments having multiple application services and multiple storage services. In some embodiments, a method of physically removing data from a storage system provides for identifying one or more files needing data removal treatment, determining that a file needing data removal treatment should be queued, and populating a queue with the file. Determining that a file should be queued is based, at least in part, on a staleness tolerance. The method further provides for treating the file and replacing a previous version of the file in storage with the updated file. In some implementations, treating the file includes removing data from the file to create an updated file and may further include additional changes to the file.