Lineage-Aware Data Retention for Granular Deletion Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems and methods struggle to efficiently implement personal data deletion policies in large-scale data systems, particularly due to the complexity of managing data lineage and compliance with regulations like GDPR and CCPA, where data transformations complicate deletion periods.
Innovation Solution
A computing system implementing lineage-aware data retention, which tracks parent-child relationships between datasets to ensure granular deletion schedules and uses declarative policies with override options to manage data retention and deletion efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data deletion policies are implemented in large-scale data systems, then compliance with regulations like GDPR and CCPA is achieved, but system complexity increases due to data lineage tracking and transformation management
Solution Approach 1:
The system pre-computes and stores deletion dates for all descendant transactions when a parent transaction is initially created or updated. This preliminary action ensures that when deletion is needed, the system can immediately execute without complex real-time lineage traversal, thus maintaining compliance while reducing operational complexity
Solution Approach 2:
The patent segments the data system into transactions with explicit parent-child relationships and assigns deletion dates at each level. By dividing the monolithic deletion problem into discrete transaction-level units with predefined deletion schedules, the system manages complexity through structured segmentation rather than holistic management
2Adaptability or versatility
If data is retained longer to maintain flexibility in deletion scheduling, then compliance options are preserved, but storage costs and data management overhead increase
Solution Approach 1:
The system dynamically determines retention periods by calculating deletion dates based on parent transaction timestamps and lineage depth. Rather than using fixed retention periods, the deletion schedule adapts to each transaction's specific position in the lineage graph, optimizing storage by deleting data as soon as compliance requirements are met while maintaining flexibility for different compliance scenarios
3Measurement precision
If granular deletion schedules are implemented for each transaction, then precise compliance control is achieved, but computation overhead for tracking and managing lineage increases
Solution Approach 1:
The system performs preliminary computation of deletion dates for all descendant transactions when a parent transaction is created or its deletion date is updated. This pre-computation stores the results in the transaction metadata, eliminating the need for expensive real-time lineage traversal during deletion operations, thus achieving precise compliance control with reduced computational overhead
Solution Approach 2:
The patent creates and stores copies of deletion date information in the metadata of each transaction. Rather than computing lineage relationships on-demand, the system maintains copied deletion date data that can be quickly accessed and enforced, reducing computation overhead while preserving granular compliance control
Data Source
AI summary
Systems and methods for lineage-aware data retention are provided. An example method includes receiving information of a committed transaction. The committed transaction is configured to add or change data to a dataset. The example method further includes receiving one or more lineages for the committed transaction, determining one or more parent transactions based at least in part on the one or more lineages, obtaining one or more parent retention dates that correspond to the one or more parent transactions respectively, and determining a transaction retention date for the committed transaction based at least in part on the one or more parent retention dates.