Workflow Data Management via Regeneration Flags
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The accumulation of intermediate data in data analysis processes leads to significant storage capacity issues, as existing methods like compression and deduplication are insufficient, and there is a risk of losing valuable data due to human error in deletion decisions.
Innovation Solution
A computer system that manages data workflow by storing management information with deletion flags, regenerates and re-stores data based on lineage information, and automatically deletes data with high regeneration costs to optimize storage usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If intermediate data is stored to preserve data assets and enable lineage tracking, then data governance and analysis capability are improved, but storage capacity is consumed
Solution Approach 1:
The system performs preliminary actions by storing management information and deletion flags before actual data deletion occurs. This allows the system to proactively manage storage by identifying candidates for deletion based on regeneration cost calculations, while preserving critical data through pre-established lineage tracking mechanisms.
Solution Approach 2:
The system creates a logical copy of data information through management information structures that track lineage, metadata, and deletion flags. This virtual representation allows the system to maintain data governance capabilities without requiring physical storage of all intermediate data, effectively separating the informational value from the physical storage burden.
2Volume of stationary object
If data deletion is performed to reduce storage usage, then storage capacity is freed, but risk of losing valuable intermediate data increases due to human error
Solution Approach 1:
The system implements feedback mechanisms by continuously monitoring data access patterns, lineage relationships, and regeneration costs. Deletion decisions are made based on automated feedback from these metrics rather than human judgment, ensuring that data critical to business operations or expensive to regenerate is preserved while safely deleting redundant data.
Solution Approach 2:
The system performs self-service by automatically evaluating data for deletion based on pre-defined criteria including regeneration cost calculations and lineage analysis. This eliminates human error in deletion decisions while maintaining data governance, as the system independently determines which data can be safely removed without business impact.
3Volume of stationary object
If regeneration cost is calculated to determine deletion candidates, then storage optimization is improved, but processing time and computational resources are consumed
Solution Approach 1:
The system applies partial action by calculating regeneration costs only for data that meets specific deletion candidate criteria rather than analyzing all stored data. This selective approach reduces computational overhead while still achieving effective storage optimization by focusing resources on evaluating only the most promising deletion candidates.
Solution Approach 2:
The system performs preliminary calculations of regeneration costs and stores this information in management structures for future reference. This allows deletion decisions to be made more efficiently by reusing previously computed cost data rather than recalculating from scratch, reducing processing time for subsequent deletion evaluations.
Data Source
AI summary
A computer system stores management information that manages a workflow and a deletion flag that indicates deletion of data in the workflow hidden from a user. The computer system executes a workflow that includes one or more processes that convert input data into output data. The computer system includes a lineage of the executed workflow including information of the input data and the output data in the management information. The computer system deletes data selected from data in the executed workflow, and sets the deletion flag of the selected data in the management information. The computer system, in response to an access to first data to which the deletion flag is set, regenerate the first data based on the management information and removes the deletion flag of the first data in the management information.