Distributed Content Indexing System Parallel Task Splitting

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional content indexing systems face scalability issues due to their single-device architecture, require reconfiguration when backup data formats change, and necessitate synchronization between metadata and content index databases, leading to increased computing costs and potential errors.

Innovation Solution

A distributed content indexing system that combines backup and content index databases into a single entity, allowing parallel task execution across multiple servers and media agents, enabling content indexing regardless of backup data format and reducing synchronization needs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If a single computing device or server is used for content indexing, then the system is simpler to manage, but scalability is limited

Engineering Contradiction:

Improvesystem complexityVSAvoidscalability

Core Design Contradiction:

Device complexityVSProductivity

Solution Approach 1:

The patent divides the content indexing system into multiple independent computing devices (indexing servers, worker nodes) that can operate in parallel. Each node handles specific indexing tasks independently, allowing the system to scale horizontally by adding more nodes without increasing overall system complexity significantly.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent transitions from a single-device vertical architecture to a multi-device horizontal architecture. By introducing the dimension of distributed computing across multiple networked devices, the system achieves scalability while maintaining manageable complexity through standardized communication protocols and task distribution mechanisms.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Adaptability or versatility

If backup data format changes or application provides data in different format, then the system needs to be reconfigured, but maintaining fixed format configuration simplifies the system

Engineering Contradiction:

Improveformat compatibilityVSAvoidconfiguration complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent implements a universal content indexing approach that can handle multiple backup data formats through a common interface. The indexing system is designed to work with restored data from various backup formats without requiring format-specific configuration, making the system adaptable to different data sources while maintaining simple unified configuration.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent introduces a data restoration layer as an intermediary between the backup storage and content indexing components. This intermediary handles format conversion and standardization, allowing the indexing system to receive uniformly formatted data regardless of the original backup format, thus providing format compatibility without increasing indexing configuration complexity.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Ease of manufacture

If backup metadata database and content index database are separated, then each database can be optimized for its specific function, but synchronization between them requires additional computing resources and may cause errors

Engineering Contradiction:

Improvedatabase optimizationVSAvoidcomputing resources for synchronization

Core Design Contradiction:

Ease of manufactureVSUse of energy by stationary object

Solution Approach 1:

The patent combines the backup metadata database and content index database into a unified database system. This eliminates the need for synchronization between separate databases, reducing computing resource consumption for data consistency maintenance while still allowing each functional area (metadata management and content indexing) to operate with its required optimizations within the unified system.

Inventive Principle:

Principle #5Merging (Combining)

4Device complexity

If conventional content indexing systems are used, then the initial configuration is simple, but reconfiguration is needed when data format changes

Engineering Contradiction:

Improveinitial configuration simplicityVSAvoidformat adaptability

Core Design Contradiction:

Device complexityVSAdaptability or versatility

Solution Approach 1:

The patent implements a dynamic content indexing system that can automatically adapt to different data formats without requiring manual reconfiguration. The system dynamically adjusts its processing based on the input data format through automated format detection and adaptation mechanisms, maintaining simple initial configuration while providing high format adaptability.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS12182092B2Distributed framework for data proximity-based task splitting in a content indexing system

Publication Date: 2024.12.31 COMMVAULT SYSTEMS INC

AI summary

An improved content indexing (CI) system is disclosed herein. For example, the improved CI system may include a distributed architecture of client computing devices, media agents, a single backup and CI database, and a pool of servers. After a file backup occurs, the backup and CI database may include file metadata indices and other information associated with backed up files. Servers in the pool of servers may, in parallel, query the backup and CI database for a list of files assigned to the respective server that have not been content indexed. The servers may then request a media agent to restore the assigned files from secondary storage and provide the restored files to the servers. The servers may then content index the received restored files. Once the content indexing is complete, the servers can send the content index information to the backup and CI database for storage.