Archival Storage Snapshot Indexing With Deferred B+ Trees

The patent segments the indexing process from the storage process by using log-structured formats. Data is written sequentially to logs without immediate indexing, and indexing is performed separately in the background. This segmentation allows rapid data intake while deferred indexing operations build the metadata structures needed for efficient retrieval, resolving the contradiction between fast writing and fast reading regardless of snapshot quantity.

The patent performs preliminary actions by pre-computing and storing index metadata in log-structured formats during the data ingestion phase. Instead of indexing on-demand during retrieval, the system prepares index structures in advance as data is written to logs, enabling constant-time retrieval operations later independent of the number of snapshots.

The patent replaces the traditional mechanical file system metadata traversal mechanism with a log-structured indexing system. Instead of traversing directory structures and metadata links through multiple snapshots, the system uses sequential log entries with embedded index information that can be directly queried, substituting the mechanical traversal process with a more efficient logging and indexing approach that scales independently of snapshot quantity.

The patent changes the fundamental parameter of how metadata is organized and accessed. Rather than using traditional file system metadata structures that require traversal, the system transforms metadata into log-structured format with direct indexing capabilities. This parameter change in the data organization structure enables retrieval efficiency to remain constant even as the number of snapshots increases to meet archival requirements.

The patent performs preliminary indexing actions in the background after data is written to logs, rather than concurrently during the write operation. Data is first written sequentially to log files at high speed, and then indexing operations are performed separately on the logged data. This preliminary action separation eliminates indexing overhead from the critical write path, maximizing write throughput while still achieving fast retrieval through pre-computed indexes.

The patent segments the data processing workflow into distinct phases: a write phase where data is rapidly appended to logs without indexing overhead, and an indexing phase where metadata structures are built from the logged data. This segmentation of concerns allows the write operation to achieve maximum throughput by avoiding concurrent indexing, while the separate indexing phase ensures retrieval efficiency is maintained.