Transaction Log Index Generation in Enterprise Backup Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Information management systems face bottlenecks in resource availability due to communication bottlenecks during backup and restore processes, particularly when dealing with a large number of client devices, leading to reduced scalability and increased resource usage.

Innovation Solution

Implementing a distributed backup management system where clients store job metadata locally and communicate directly with cloud or network storage, reducing queries to the storage manager and enabling clients to monitor commit cycles for efficient backup and restore operations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If clients communicate with the storage manager during each backup operation, then backup management can be centralized and controlled, but communication overhead increases and scalability decreases

Engineering Contradiction:

Improvecentralized backup managementVSAvoidbackup processing speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system performs preliminary actions by having clients obtain job metadata from the storage manager before backup operations begin. This metadata includes configuration information and resource allocation details that remain relatively static. By preparing these parameters in advance and caching them locally, the system reduces the need for repeated communication during actual backup operations, thus improving productivity while maintaining centralized control.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The backup management process is segmented into distinct phases: configuration retrieval phase (communication with storage manager) and execution phase (local operation with cached metadata). This segmentation allows centralized management to occur only when necessary (at the beginning), while the bulk of backup operations proceed independently at clients, resolving the contradiction between centralized control and processing speed.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If the information management system supports a large number of client devices, then system coverage and utility increase, but resource availability and system performance decrease

Engineering Contradiction:

Improvenumber of supported clientsVSAvoidcomputing resource consumption

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

Clients are empowered to self-serve by caching job metadata locally and using it for multiple backup operations without requiring continuous interaction with the storage manager. This self-service capability allows clients to autonomously perform backups using cached configuration information, significantly reducing the computing resources consumed by the central storage manager and enabling the system to support more clients efficiently.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system extracts the static configuration information and resource allocation parameters from the centralized storage manager and places them locally in the client's cache. By taking out these parameters that do not need frequent updates, the system reduces the communication and processing burden on the storage manager, allowing it to scale to support more clients without proportionally increasing resource consumption.

Inventive Principle:

Principle #2Taking out (Extraction)

3Reliability

If the storage manager handles all backup coordination for multiple clients, then centralized control is maintained, but communication bottlenecks increase and system scalability is limited

Engineering Contradiction:

Improvebackup coordination controlVSAvoidcommunication overhead

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The storage manager performs preliminary coordination actions by providing job metadata to clients before backup operations commence. This metadata encapsulates the coordination information needed for backup execution. By completing the coordination work in advance and caching it locally at clients, the system maintains centralized control over backup policies and resource allocation while dramatically reducing communication overhead during actual backup operations.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11263173B2Transaction log index generation in an enterprise backup system

Publication Date: 2022.03.01 COMMVAULT SYSTEMS INC

AI summary

Certain embodiments disclosed herein reduce or eliminate a communication bottleneck at the storage manager by reducing communication with the storage manager while maintaining functionality of an information management system. In some implementations, operations performed as part of a backup process may be stored in transaction logs. These transaction logs may include information about a transaction performed between the client computing system and the network storage that hosts the backup of the client computing system. The transaction logs may be provided to a secondary storage system that can be used to form a backup index. The backup index may be used to facilitate accessing the data stored at the network storage. Advantageously, generating the transaction logs and separating the generation of the backup index from the backup process can reduce resource usage during performance of the backup and speed up the backup process while further reducing interaction with the storage manager.