Data Storage Management With Asynchronous Index Garbage Collection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems fail to efficiently manage storage by removing non-committed data items, leading to resource consumption and processing slowdowns due to accumulation of auto-committed data.
Innovation Solution
A data storage management system (DMS) that performs asynchronous garbage collection and auto-commits non-committed data during object creation, using a cleanup mechanism to remove auto-committed data after transaction completion, ensuring efficient storage allocation and usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If auto-commit mechanism is used during object creation, then data availability for new objects is improved, but storage resource waste increases due to accumulation of non-committed data
Solution Approach 1:
The system performs preliminary auto-commit of data items before the parent transaction completes, allowing new objects to be created with immediately available data. This preliminary action resolves the contradiction by making data available earlier without waiting for the full transaction to complete.
Solution Approach 2:
The system implements a cleanup mechanism that identifies and removes data items that were auto-committed but whose parent transactions ultimately failed or rolled back. This discarding of useless auto-committed data prevents storage resource waste while maintaining the benefits of early data availability for successful transactions.
2Reliability
If data items are retained in storage, then data availability is maintained, but processing speed decreases due to accumulation of unnecessary data
Solution Approach 1:
The cleanup mechanism continuously identifies and removes data items that are no longer needed (where parent transactions have failed or rolled back). This selective discarding maintains data availability for valid items while removing unnecessary data that would slow down processing, thus resolving the speed-reliability contradiction.
Solution Approach 2:
The system implements self-service through automatic detection and cleanup of obsolete data items. The cleanup mechanism autonomously identifies data items whose parent transactions have completed or failed and removes them without manual intervention, maintaining processing speed while preserving data availability for active transactions.
3Productivity
If asynchronous garbage collection is implemented, then storage efficiency is improved, but system complexity increases
Solution Approach 1:
The asynchronous garbage collection system operates autonomously to manage storage efficiency. It automatically tracks transaction states, identifies obsolete data items, and performs cleanup without requiring complex manual management or intervention. This self-service approach improves storage efficiency while keeping the added system complexity manageable through automation.
Solution Approach 2:
The system introduces a cleanup mechanism as an intermediary component that bridges the gap between auto-commit operations and final transaction completion. This intermediary asynchronously monitors transaction states and coordinates the removal of obsolete data, improving storage efficiency while managing complexity through a dedicated mediation layer rather than embedding complexity throughout the entire system.
Data Source
AI summary
System, method, and various embodiments for a data storage management system are described herein. An embodiment operates by receiving a request to create an index based on a portion of a database, the portion comprising one or more entries that correspond to one or more ongoing transactions. The index is generated based on the portion of the database, and a subset of entries that correspond to one or more ongoing transactions are auto-committed prior to a completion of the one or more ongoing transactions. A command to rollback the request to create the index is detected. The index is scheduled for asynchronous garbage collection, that remove information associated with the generated index from both memory and disk based upon a completion of one or more parallel transactions. The information associated with the generated index is removed from both the memory and disk in accordance with the asynchronous garbage collection.