Archival Data Stream Segmentation and On-Demand Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data archiving and retrieval systems face inefficiencies in preserving chain of custody and data characteristics, particularly in real-time search and retrieval, and often result in data loss during restoration due to the deletion of stub changes.
Innovation Solution
A method that separates data into streams based on criteria, indexes only relevant data items and properties upon query, and restores data items by combining stubs with archived data while preserving changes, using a record manifest for metadata and transferring data between storage locations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is archived by creating stubs and storing only metadata in the archive, then storage space is conserved, but data integrity is compromised during restoration due to loss of stub changes
Solution Approach 1:
The data item is segmented into two parts: a stub containing metadata and a pointer stored in the archive, and the actual data content stored separately. This segmentation allows the archive to consume less storage space while preserving the ability to restore complete data items by combining stubs with archived content.
Solution Approach 2:
A pointer acts as an intermediary between the stub and the archived data content. The pointer stores information about the original data item's location and characteristics, enabling the system to retrieve and reconstruct complete data items during restoration by linking stubs with their corresponding archived content.
2Speed
If indexing is performed on the entire data stream before search, then search speed is improved, but computing resources are wasted on irrelevant data items
Solution Approach 1:
Instead of indexing the entire data stream, the system performs partial indexing only on data items that match the search criteria. The indexing process is applied selectively to relevant portions of the data, reducing computing resource consumption while maintaining search efficiency for the specific query at hand.
Solution Approach 2:
The system performs preliminary filtering of data items based on search criteria before applying indexing operations. By pre-identifying relevant data items that match search parameters, the system prepares only the necessary portions for indexing, thereby conserving computing resources while ensuring fast search performance on relevant data.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Data items are archived by separating them into two or more data streams according to common characteristics or categories. Data item properties, including custodian and date properties, are defined for the items in each stream. A record manifest, including metadata corresponding to the data item properties for the stream, is created. The data items and the manifest are stored. The data items are indexed only on demand, and only to the extent necessary to satisfy the demand. When data is restored from archival storage, it is combined with the stub in a manner that treats the stub and stored data as complementary parts, thus preserving any changes to the stub that are not reflected in the archive copy.