File System Indexing via Parallel Subdirectory Traversal

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Implementing information lifecycle management (ILM) in a large namespace file system without disrupting ongoing data processing operations is challenging due to the time-consuming nature of classification and indexing processes.

Innovation Solution

Executing utility programs on multiple host processors to traverse subdirectory trees, collect file-specific information, and store it in logs or databases, facilitating efficient storage and retrieval by accessing this information during the ILM process.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Extent of automation

If classification and indexing is performed on all files in a large namespace, then information lifecycle management is enabled, but processing time and disruption to ongoing operations increase significantly

Engineering Contradiction:

Improveinformation lifecycle managementVSAvoidprocessing time

Core Design Contradiction:

Extent of automationVSLoss of time

Solution Approach 1:

The file system namespace is divided into multiple subdirectory trees, and the classification indexing process is segmented across multiple host processors. Each processor handles a specific subset of files, enabling parallel processing that reduces total processing time while maintaining comprehensive ILM coverage across the entire namespace.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

File-specific information is collected and stored in logs during a preliminary traversal phase before the main ILM processing begins. This preliminary action prepares indexing data in advance, allowing the subsequent ILM operations to proceed faster with reduced disruption to ongoing data processing activities.

Inventive Principle:

Principle #10Preliminary action

2Productivity

If multiple host processors are used to traverse subdirectory trees, then processing speed increases, but system complexity increases

Engineering Contradiction:

Improveprocessing speedVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The workload is segmented across multiple host processors, with each processor responsible for traversing specific subdirectory trees. This segmentation enables parallel processing that increases productivity while keeping the complexity of individual processor tasks manageable through clear division of labor.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Logs serve as intermediary structures that store file-specific information collected during traversal. These logs act as intermediaries between the multiple host processors and the final ILM processing, simplifying coordination and data management across the distributed system without requiring complex inter-process communication protocols.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS7672981B1Object classification and indexing of very large name spaces using grid technology

Publication Date: 2010.03.02 EMC IP HLDG CO LLC

AI summary

For migration or de-duplication of a file system having a large number of files, a utility program traverses the file system to create a log of file-specific information about the file system. For identification of duplicates, the utility program produces a signature for each file. Respective instances of the utility program are started on multiple nodes upon which the file system is mounted. A fully qualified pathname is compiled during transfer of the log to a database. Multiple databases can be produced for the file system such that each database contains the file-specific information for a specified range of inode numbers. The database also maintains classification state for each file. For example, for a migration or replication process, the classification state identifies whether or not the file has been untouched, copied, linked, secondary-ized, source deleted, or modified.