Database Index Rollback With Asynchronous Garbage Collection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems struggle with efficiently managing storage by removing data items that are no longer useful, leading to accumulation in memory and disk, which slows down processing and consumes unnecessary resources.
Innovation Solution
Implementing a data storage management system (DMS) that performs auto-commit on non-committed data during object creation, uses asynchronous garbage collection (AGC) to clean up auto-committed data, and employs a cleanup manager to ensure data is removed only after ongoing transactions are completed, thereby optimizing storage allocation and usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data items are automatically committed to disk during object creation, then data availability is improved, but storage space is consumed by potentially obsolete data
Solution Approach 1:
The system performs preliminary auto-commit of data items to disk during object creation, ensuring data is available before the creating transaction completes. This preliminary action improves data availability while the subsequent garbage collection process removes obsolete committed data to free storage space.
Solution Approach 2:
The system implements asynchronous garbage collection that identifies and removes data items that have been committed but are no longer needed by any active transaction. This discarding process recovers storage space while maintaining data availability for transactions that still need it.
2Quantity of substance
If garbage collection is performed synchronously, then storage space is freed quickly, but transaction processing speed decreases
Solution Approach 1:
The system schedules garbage collection operations in advance as asynchronous tasks that execute independently of the main transaction processing flow. This preliminary scheduling ensures storage space is freed without blocking or slowing down transaction processing, as the GC operations run in parallel.
Solution Approach 2:
The system dynamically manages garbage collection execution, allowing it to run asynchronously when storage resources permit and adjusting its intensity based on system load. This dynamic approach maintains storage efficiency while preserving transaction processing speed under varying conditions.
3Quantity of substance
If data is removed immediately after commit, then storage efficiency is improved, but data consistency for parallel transactions is compromised
Solution Approach 1:
The system uses feedback mechanisms to track which transactions are still active and which data items they depend on. Garbage collection is triggered only when feedback indicates that no active transactions need the data, ensuring data consistency is maintained while achieving storage efficiency through timely removal of obsolete data.
Solution Approach 2:
The system introduces an intermediary reference counting mechanism that tracks dependencies between transactions and committed data. This intermediary prevents premature removal of data by maintaining references until all dependent transactions complete, ensuring consistency while enabling efficient storage management.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
System, method, and various embodiments for a data storage management system are described herein. An embodiment operates by receiving a request to create an index based on a portion of a database, the portion comprising one or more entries that correspond to one or more ongoing transactions. The index is generated based on the portion of the database, and a subset of entries that correspond to one or more ongoing transactions are auto-committed prior to a completion of the one or more ongoing transactions. A command to rollback the request to create the index is detected. The index is scheduled for asynchronous garbage collection, that remove information associated with the generated index from both memory and disk based upon a completion of one or more parallel transactions. The information associated with the generated index is removed from both the memory and disk in accordance with the asynchronous garbage collection.