Storage Object Transaction Offloading for Capacity Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage systems face size limitations, which can lead to insufficient space for new transactions, necessitating the removal of processed transactions to accommodate incoming data, while maintaining data integrity and performance.
Innovation Solution
The storage system implements an update process that moves processed transactions from objects to a transaction datastore, creating space by storing them as historical transactions, and uses optimistic locking to ensure data integrity during updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If processed transactions are stored in memory locations with size limitations, then data access performance is maintained, but storage capacity becomes insufficient for new transactions
Solution Approach 1:
The storage system is divided into two distinct segments: memory locations (e.g., object storage) for storing current/active transactions with fast access, and a transaction datastore (e.g., database) for storing historical transactions. This segmentation allows the system to maintain high performance for active transactions while offloading historical data to a storage mechanism with greater capacity but potentially slower access, thereby resolving the contradiction between speed and storage capacity.
Solution Approach 2:
The invention extracts historical transactions from the memory locations and moves them to the transaction datastore. By removing processed/completed transactions from the object storage, the system frees up space in the size-limited memory locations to accommodate new transactions, thus maintaining storage capacity without sacrificing performance for active data.
2Quantity of substance
If processed transactions are removed from memory locations, then storage space is created for new transactions, but data integrity may be compromised
Solution Approach 1:
The transaction datastore acts as an intermediary that ensures data integrity during the transition of transactions from memory locations to permanent storage. The system uses optimistic locking mechanisms and transactional operations to ensure that data is consistently transferred and that no data is lost or corrupted during the move operation. This intermediary layer protects data integrity while enabling space management.
Solution Approach 2:
The system performs preliminary actions by copying transactions to the transaction datastore before removing them from memory locations. This preliminary copy operation ensures that data exists in the new location before the old location is cleared, preventing data loss. The use of atomic operations ensures that the transfer is completed successfully or rolled back if errors occur, maintaining data integrity.
3Reliability
If optimistic locking is used during updates, then data integrity is ensured, but system complexity increases
Solution Approach 1:
The optimistic locking mechanism enables each transaction operation to self-verify data consistency without requiring complex external locking protocols. The system uses version checking and conditional updates where transactions include their own integrity verification logic. This self-service approach to data integrity reduces the need for complex centralized locking mechanisms, thereby managing system complexity while maintaining reliability.
Data Source
AI summary
Described herein are storage systems, computer readable mediums, and methods for overcoming size limit storage limitations associated with storage systems. Processed transactions may be removed from an object so that input data may be stored at the object. In some implementations, the storage system removes the processed transactions from the object once a section of the object includes a threshold amount of data, and stores the processed transactions at a transaction datastore. Once the processed transactions have been removed from the object, the storage system may store the input data at the object.