Snapshot File Indexing via Metadata Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies in indexing and searching snapshots due to their large size, making it impractical to scan every snapshot for file management and retrieval, especially when dealing with billions of files and terabytes of data.
Innovation Solution
A method is implemented to generate and maintain file records and snapshot lists, allowing for efficient indexing and searching by scanning the storage system once and using notifications to update records, enabling quick retrieval of active, deleted, or modified files across snapshots.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If every snapshot is scanned to maintain file index, then file retrieval completeness is improved, but computational overhead and time consumption increase significantly
Solution Approach 1:
The patent extracts only the necessary file metadata information from snapshots rather than scanning entire snapshot contents. The indexing system pulls out file names, paths, and timestamps from snapshot metadata structures, avoiding the need to read and process actual file data, thereby reducing indexing time while maintaining retrieval completeness
Solution Approach 2:
The system performs preliminary indexing by creating file records from snapshot metadata during snapshot creation or immediately afterward. This preliminary action prepares the index structure in advance, so that when file retrieval is needed, the pre-indexed data can be quickly accessed without performing full snapshot scans at retrieval time
2Measurement precision
If snapshot data is fully scanned for file management, then file search accuracy is improved, but system resource consumption increases
Solution Approach 1:
The patent introduces an intermediary indexing layer that sits between the snapshot storage and the file search functionality. This index structure acts as a mediator, containing pre-extracted file metadata that enables accurate file searches without requiring direct access to or scanning of the actual snapshot data, thus reducing system resource consumption while maintaining search accuracy
Solution Approach 2:
The system creates simplified copies of file metadata information from snapshots and stores them in an index structure. These copies contain essential file identification data (names, paths, timestamps) that enable accurate file searches without requiring the original snapshot data to be scanned, thereby reducing computational resources while preserving search accuracy
3Loss of information
If comprehensive snapshot scanning is performed, then file version tracking is improved, but device complexity increases
Solution Approach 1:
The patent segments the file version tracking functionality into distinct components: snapshot metadata collection, file record generation, and version history maintenance. Each component handles a specific aspect of version tracking independently, reducing overall system complexity while maintaining comprehensive version information. The indexing system processes snapshots in discrete units and maintains version records in a structured, modular manner
Data Source
AI summary
Indexing files to account for snapshots can include generating, based on a scan of the storage system, one or more file records. Each file record of the one or more file records can correspond to a file stored on the storage system at the time of the scan. The file records can be maintained based on one or more notifications received from the storage system. A snapshot list of the storage system can be maintained, the snapshot list having snapshot data corresponding to one or more snapshots stored on the storage system. A search result can be generated that satisfies a search parameter based at least on a) the one or more file records and/or b) the snapshot list.