Lineage-Aware Data Retention for Granular Deletion Compliance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems struggle to efficiently implement personal data deletion policies in large-scale data systems, particularly due to the complexity of managing data lineage and compliance with regulations like GDPR and CCPA, where data transformations complicate deletion periods.
Innovation Solution
A lineage-aware data retention system that tracks parent-child relationships between datasets, assigns granular deletion dates to transactions, and supports override policies to ensure compliance and efficient data deletion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data deletion policies are implemented in large-scale data systems, then data compliance with regulations like GDPR and CCPA is achieved, but system complexity increases due to data lineage tracking and transformation management
Solution Approach 1:
The patent segments data deletion management into transaction-level operations, where each data transaction is individually tracked with its own retention date. This allows compliance to be enforced at the granular transaction level rather than requiring complex system-wide lineage tracking, reducing overall system complexity while maintaining reliability.
Solution Approach 2:
The system performs preliminary action by assigning retention dates to data transactions at the time of ingestion or transformation, rather than calculating deletion requirements later based on complex lineage analysis. This advance assignment simplifies subsequent deletion operations and reduces the computational complexity of tracking data relationships.
2Manufacturing precision
If granular deletion dates are assigned to individual transactions, then data retention precision is improved, but processing overhead increases due to tracking multiple parent-child relationships
Solution Approach 1:
The patent implements self-service by having each data transaction automatically inherit its retention date from parent transactions through a automated propagation mechanism. This eliminates the need for manual lineage tracking and reduces processing overhead, as the system autonomously manages the inheritance relationships without requiring complex intervention.
Solution Approach 2:
The system uses copying by replicating retention date information from parent transactions to child transactions through lineage relationships. This allows precise retention tracking without requiring the system to recalculate or re-analyze lineage relationships, significantly reducing processing overhead while maintaining high retention precision.
3Reliability
If data is retained according to strict lineage-aware policies, then compliance accuracy is improved, but storage costs increase due to extended retention periods
Solution Approach 1:
The patent applies local quality by assigning different retention dates to different data transactions based on their specific lineage and compliance requirements, rather than applying a uniform retention policy across all data. This allows the system to retain only the necessary amount of data for each transaction, improving compliance accuracy while minimizing unnecessary storage accumulation.
Solution Approach 2:
The system changes parameters by dynamically adjusting retention dates based on lineage analysis and compliance requirements. Transactions that originate from compliant sources receive appropriate retention dates, while those requiring longer retention are extended only as necessary. This parameter adjustment optimizes the balance between compliance accuracy and storage efficiency.
4Adaptability or versatility
If override policies are implemented for specific transactions, then operational flexibility is improved, but policy management complexity increases
Solution Approach 1:
The patent extracts override capability as a separate, optional feature that can be applied to individual transactions or datasets without affecting the overall retention policy framework. This allows operational flexibility to be introduced selectively, managing complexity by isolating override operations from the core policy management system.
Data Source
AI summary
Systems and methods for lineage-aware data retention are provided. An example method includes receiving information of a committed transaction. The committed transaction is configured to add or change data to a dataset. The example method further includes receiving one or more lineages for the committed transaction, determining one or more parent transactions based at least in part on the one or more lineages, obtaining one or more parent retention dates that correspond to the one or more parent transactions respectively, and determining a transaction retention date for the committed transaction based at least in part on the one or more parent retention dates.