Provenance-Based Data Backup Filtering for Big Data Lakes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data mining processes generate vast collections of temporary data that are not effectively managed for backup operations, particularly due to resource intensity and lack of participation in data backup processes within data lakes.
Innovation Solution
A method and system utilizing data provenance to filter and replicate data from data mining processes, focusing on datasets that have significant reuse potential, low storage requirements, and high resource costs, ensuring efficient backup storage through a provenance-based approach.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data lakes store vast collections of temporary data from data mining processes, then data storage capacity is improved, but backup operation participation and data management efficiency deteriorate
Solution Approach 1:
The patent segments data lakes into multiple data zones based on data characteristics, access patterns, and backup requirements. This segmentation allows different zones to participate in backup operations differently, improving overall backup efficiency while maintaining storage capacity. Hot data zones with high reuse potential are prioritized for backup, while cold data zones are managed differently.
Solution Approach 2:
The patent changes the parameter of data backup participation by introducing provenance-based metadata that tracks data characteristics, processing history, and reuse potential. This metadata enables intelligent differentiation of backup priorities, transforming the uniform backup approach into a selective, efficiency-driven backup strategy that addresses the contradiction between storage capacity and backup efficiency.
2Productivity
If provenance-based filtering is applied to identify datasets for backup, then backup efficiency is improved, but system complexity increases
Solution Approach 1:
The patent implements preliminary action by capturing and storing provenance metadata during data creation and processing operations. This metadata, including data characteristics, processing history, and reuse potential, is recorded in advance, enabling efficient backup decision-making without adding complexity to the actual backup execution phase.
Solution Approach 2:
The patent introduces provenance metadata as an intermediary layer between data storage and backup operations. This metadata acts as a mediator that provides the necessary information for intelligent backup decision-making, simplifying the backup system's complexity by offloading the decision logic to pre-computed metadata rather than requiring complex real-time analysis during backup operations.
3Reliability
If all temporary data from data mining is backed up, then data reliability is improved, but storage resource utilization deteriorates
Solution Approach 1:
The patent applies local quality by differentiating backup priorities across different data zones and datasets based on their specific characteristics, reuse potential, and business value. Instead of uniform backup of all temporary data, the system applies selective backup policies tailored to local data properties, improving storage resource utilization while maintaining reliability for critical datasets.
Solution Approach 2:
The patent implements discarding and recovering by identifying temporary data with low reuse potential and discarding it from backup operations, while recovering and prioritizing backup of data with high reuse potential and business value. This selective approach optimizes storage resource utilization by eliminating unnecessary backup of transient data while ensuring reliability preservation of valuable datasets.
Data Source
AI summary
A method and system for provenance-based data backups. Specifically, the method and system disclosed herein entail generating and, subsequently, using data provenance to filter which collections of data, produced through the data mining of big data, should be replicated and stored in backup storage.