Append Log Metadata Store with Garbage Collection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional storage systems face challenges in efficiently storing metadata due to overhead complexities, unreliability, and speed issues, with existing solutions either being too complex or lacking in reliability and atomicity.

Innovation Solution

The use of an append log for metadata storage, which allows for quick addition of entities and provides reliability through error correction, combined with dedicated storage sections for efficient monitoring and garbage collection, and redundancy across multiple storage devices.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If traditional databases are used for metadata storage, then reliability is improved, but device complexity and overhead increase

Engineering Contradiction:

Improvemetadata storage reliabilityVSAvoidstorage system complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent extracts the metadata storage function from complex traditional databases and implements it using simple append-only log files. This separation allows the metadata store to be simplified while maintaining reliability through the append-only nature of the logs, which prevent corruption and enable easy recovery.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent uses simple, disposable append log files instead of complex persistent database structures. These log files are easy to create, modify, and replace, providing a simple yet reliable metadata storage mechanism that can be quickly rewritten if corruption occurs.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

2Device complexity

If simple files are used for metadata storage, then device complexity is reduced, but reliability and atomicity deteriorate

Engineering Contradiction:

Improvestorage system complexityVSAvoidmetadata storage reliability

Core Design Contradiction:

Device complexityVSReliability

Solution Approach 1:

The patent segments metadata storage into multiple append-only log files, each handling specific metadata types or time periods. This segmentation allows for atomic operations within each log file while maintaining overall system reliability through isolated failure domains.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements preliminary actions by pre-creating the append-only log structure and establishing clear append operations before any metadata needs to be stored. This preliminary setup ensures atomicity and reliability without requiring complex transaction management during operation.

Inventive Principle:

Principle #10Preliminary action

3Reliability

If traditional databases are used for metadata storage, then reliability is improved, but storage speed deteriorates

Engineering Contradiction:

Improvemetadata storage reliabilityVSAvoidmetadata storage speed

Core Design Contradiction:

ReliabilityVSSpeed

Solution Approach 1:

The patent uses append-only log files that allow continuous writing without interruption or locking. New metadata is continuously appended to the end of the log, enabling high-speed storage operations while maintaining reliability through the immutable append-only nature of the logs.

Inventive Principle:

Principle #20Continuity of useful action

Solution Approach 2:

Instead of using traditional database insert/update operations that require locking and transaction management, the patent inverts the approach by using append-only operations that naturally provide atomicity and reliability without sacrificing speed.

Inventive Principle:

Principle #13The other way round (Inversion)

4Speed

If append logs are used for metadata storage, then storage speed is improved, but reliability may deteriorate without error correction

Engineering Contradiction:

Improvemetadata storage speedVSAvoidmetadata storage reliability

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

The patent implements error correction and validation mechanisms beforehand in the append log structure, including checksums and validation routines that prevent corruption before it affects data integrity. This cushioning allows high-speed appending while maintaining reliability.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

Data Source

PatentUS11928084B2Metadata store in multiple reusable append logs

Publication Date: 2024.03.12 NVIDIA CORP

AI summary

A storage system (100) stores metadata using an append log to provide speed and reliability. Appending entities to the end of a metadata store (154) and provides reliability in the event of failure because a failed append operation at worst corrupts the end of the append log, which is easily detected and corrected. A metadata store (154) can be split into sections (158). A used section (158) may be identified as garbage when the stored metadata in other sections (158) make the all of its stored entity sets stale. A used section (158) can be made garbage by storing the entity sets from the section (158) in another section (158). Sections (158) containing garbage thereby change to unused allow replacement and removal of previous sections (158) of metadata.