File and Backup Indexing for Efficient Garbage Collection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data storage systems face inefficiencies in indexing large numbers of backed-up files, particularly due to high resource consumption and slow performance during garbage collection, as they require traversing all file identifiers to determine expired files and consume significant storage space with redundant data.

Innovation Solution

The implementation of a File index and Backup index with additional 'FirstValid' and 'LastValid' properties to efficiently determine file inclusion in backups, allowing for faster garbage collection and reduced storage needs by eliminating file identifiers from the Backup index.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If all files in large backups are indexed to ensure search availability, then search completeness is improved, but system resource consumption increases significantly

Engineering Contradiction:

Improvesearch availabilityVSAvoidsystem resource consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent extracts only the essential information needed for search operations (file metadata and backup associations) from the complete file set, storing only unique file entries in the index while maintaining backup-level tracking through the Backup index. This selective extraction reduces indexing resource consumption while preserving search capability.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

Instead of indexing all files and then filtering during search, the patent inverts the approach by maintaining a Backup index that tracks file associations at the backup level, allowing the system to determine file availability through backup metadata rather than exhaustive file indexing.

Inventive Principle:

Principle #13The other way round (Inversion)

2Reliability

If the Backup index stores identifiers for all files belonging to each backup, then file-to-backup mapping is complete, but storage space consumption increases significantly

Engineering Contradiction:

Improvefile-to-backup mapping accuracyVSAvoidstorage space

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent removes redundant file identifiers from the Backup index by extracting only the essential backup metadata (creation time, expiration time, and unique file references). The File index stores unique file entries, while the Backup index stores backup-level information, eliminating duplicate file identifier storage across multiple backups.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent merges file identification and backup association functions into a coordinated two-index system where the File index handles unique file metadata and the Backup index handles backup-level tracking. This combination allows the system to maintain complete file-to-backup mapping without storing all file identifiers in every backup record.

Inventive Principle:

Principle #5Merging (Combining)

3Reliability

If the garbage collection thread traverses all file identifiers in valid backups to determine expired files, then deletion accuracy is improved, but processing time increases significantly

Engineering Contradiction:

Improvedeletion accuracyVSAvoidgarbage collection processing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent performs preliminary organization of file and backup metadata in the two-index structure before garbage collection is needed. The File index pre-processes unique file entries with their metadata, and the Backup index pre-organizes backup-level information. During garbage collection, the system can efficiently query this pre-organized data without traversing all file identifiers, reducing processing time while maintaining deletion accuracy.

Inventive Principle:

Principle #10Preliminary action

4Measurement precision

If search results are cross-referenced with backup specifications to find relevant files, then search accuracy is improved, but search performance decreases due to reprocessing

Engineering Contradiction:

Improvesearch accuracyVSAvoidsearch performance

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent pre-processes and stores file-to-backup associations in the two-index structure during the indexing phase. The File index contains unique file metadata, and the Backup index contains backup-level information including file associations. When a search is performed, the system can directly query the pre-organized indexes without reprocessing search results, maintaining search accuracy while improving performance.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11614999B2Efficient method to index scheduled backup of same target and the corresponding files

Publication Date: 2023.03.28 EMC IP HLDG CO LLC

AI summary

An apparatus, method, and system for indexing backed up files is disclosed. The operations comprise: maintaining a File index and a Backup index, the File index comprising entries each associated with an individual file backed up, the Backup index comprising entries each associated with a particular backup; receiving a request to determine whether or not a first file is included in a first backup; determining whether or not the first file is included in the first backup based on the File index entry corresponding to the first file and the Backup index entry corresponding to the first backup; and returning a result of the determination of whether or not the first file is included in the first backup.