Automated Data Recreation in Limited Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In systems with limited storage capacity, researchers face challenges in managing large datasets, such as satellite imagery, where storage space is often depleted, leading to errors or the need to delete data, which can be time-consuming and difficult due to coordination issues and lack of information on regenerating derived data.
Innovation Solution
A computer-implemented method and system that uses a catalog to automatically recreate data at a first storage location by identifying and retrieving it from a second storage location, allowing seamless data access and storage management without user intervention, utilizing a processor to update the catalog and facilitate data recreation based on retrieval history and scripts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is continuously stored in a system with limited storage capacity, then data availability is improved, but storage space is depleted
Solution Approach 1:
The system creates a catalog that stores metadata and location information as a copy of the actual data. When data is needed, the system retrieves it from the catalog rather than storing the full data set, significantly reducing storage requirements while maintaining data availability
Solution Approach 2:
The system extracts only the essential metadata and location information from the complete data sets and stores them in the catalog. This extraction allows the system to maintain data accessibility while occupying minimal storage space
2Quantity of substance
If data is deleted to free storage space, then storage capacity is improved, but data loss occurs and regeneration is difficult
Solution Approach 1:
The system performs preliminary actions by storing metadata and location information in the catalog before actual data deletion occurs. This preliminary cataloging enables rapid data regeneration from backup sources without losing critical information
Solution Approach 2:
The system discards full data sets from primary storage to free space, but recovers them on-demand by retrieving location information from the catalog and fetching data from backup sources, achieving both space efficiency and data preservation
3Ease of operation
If manual coordination is required to delete data, then storage management control is improved, but time consumption increases
Solution Approach 1:
The system provides self-service storage management by automatically tracking data locations in the catalog and handling data retrieval and regeneration without requiring manual user coordination. The catalog serves as an automated index that guides data management operations
Solution Approach 2:
The system implements feedback mechanisms where the catalog continuously tracks data locations and availability, automatically adjusting storage management decisions based on this feedback information, eliminating the need for manual intervention
Data Source
AI summary
Techniques for automatically recreating data removed from storage are disclosed. A request to access data at a first storage location is received. It is determined, based on a catalog and using a computer processor, that the data has been removed from the first storage location. In response the data at the first storage location is automatically recreated using the computer processor. The recreating includes identifying a second storage location for the data, based on the catalog, retrieving the data from the second storage location, and storing the data at the first storage location. The catalog is updated to reflect recreating the data at the first storage location.