Memory-Efficient Data Unarchiving via Secondary Store Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data archiving methods require reloading entire archive blocks into primary memory to access specific data, leading to inefficient use of scarce and expensive memory resources and limiting the establishment of shorter data retention periods.
Innovation Solution
Implement a memory-efficient unarchiving mechanism that loads data of interest into a secondary data store, allowing selective extraction and reloading into primary memory without reloading additional data, thereby reducing memory usage and enabling shorter retention periods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If entire archive blocks are reloaded into primary memory to access specific data, then data accessibility is improved, but memory usage increases significantly
Solution Approach 1:
The patent segments archive blocks into smaller granularities (e.g., 64KB or 1MB chunks) and selectively loads only the specific segments containing the requested data into primary memory, rather than loading entire archive blocks. This segmentation enables precise control over memory usage while maintaining data accessibility.
Solution Approach 2:
The patent extracts and loads only the specific data portions needed from archive blocks into primary memory, leaving the rest of the archive block in secondary storage. This extraction approach eliminates the need to load unnecessary data, directly reducing memory usage while preserving access to required data.
2Quantity of substance
If data is archived to limit memory usage, then memory efficiency is improved, but data access complexity increases
Solution Approach 1:
The patent introduces an intermediary layer (archive management system with mapping structures) between primary memory and secondary storage. This intermediary automatically handles the complexity of locating and loading specific data segments from archives, presenting a simple interface to applications while managing the underlying access complexity.
Solution Approach 2:
The patent performs preliminary actions by pre-establishing mapping structures and indexes that track data locations within archive blocks. When data is needed, the system quickly resolves the location through these pre-computed mappings and loads only the necessary segments, reducing access complexity compared to searching through entire archive blocks.
3Duration of action of stationary object
If longer data retention periods are established, then data availability is improved, but memory consumption increases
Solution Approach 1:
The patent segments retained data into archive blocks organized by retention periods, allowing selective loading of only those segments whose retention period has not expired. This enables the system to maintain long retention policies while loading only necessary data segments into memory, rather than keeping all retained data in primary memory.
Solution Approach 2:
The patent applies different storage qualities to different data segments based on their retention requirements and access patterns. Frequently accessed or recently expired data segments are loaded into primary memory, while older or less frequently accessed segments remain in secondary storage, optimizing memory usage while maintaining required retention periods.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems, methods, and computer media are described for memory-efficient unarchiving. Data items can be archived and removed from primary data store over time. An archive block containing both a data item of interest and additional data items can be located. The data item of interest, as well as the additional data, can be loaded into a secondary data store. The data item of interest can then be extracted from the secondary data store and reloaded into the primary data store. The secondary data store can then be cleared. Memory-efficient unarchiving more efficiently uses computing resources and unarchives only the data that is needed.