Super File Indexing for Similar Data Search

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data indexing methods are inefficient due to large index sizes and cluttered search results from similar data, particularly in environments like data protection and email servers, where conventional approaches fail to leverage data similarity effectively.

Innovation Solution

The technique involves pre-processing similar data to group it into super files, which are then indexed and queried, allowing for efficient retrieval and presentation of search results, leveraging existing index and query engines while relaxing query semantics and enabling incremental updates.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional indexing methods are used to store all files individually, then complete search coverage is achieved, but index size becomes very large and search results are cluttered with similar data

Engineering Contradiction:

Improvesearch coverageVSAvoidindex size

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent merges multiple similar files into a single super file that contains representative terms from all files in the group. This combining approach reduces the total number of indexed entities while preserving search capability, directly addressing the contradiction between complete search coverage and large index size.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The super file serves multiple functions: it represents an entire group of similar files for indexing purposes, provides search results for all member files simultaneously, and enables efficient querying without storing redundant information. This multi-functionality resolves the contradiction by making the index more compact while maintaining search effectiveness.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Measurement precision

If all similar files are indexed separately, then accurate individual search results are obtained, but query processing time increases and results are cluttered

Engineering Contradiction:

Improvesearch result accuracyVSAvoidquery processing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent performs preliminary grouping of similar files into super files during the indexing phase, before query processing occurs. This pre-processing organizes files by similarity, so that during query processing, the system can efficiently retrieve super files representing groups of similar files, significantly reducing query processing time while maintaining accurate results through post-processing expansion.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The super file acts as an intermediary between the index and individual files. Instead of directly indexing and querying each individual file, the system queries super files that represent groups of similar files, then expands results to individual files through post-processing. This intermediary structure reduces query processing time while preserving search accuracy.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Quantity of substance

If virtual sub files are used to partition original files, then index size is reduced, but query engine complexity increases and incremental updates are disabled

Engineering Contradiction:

Improveindex sizeVSAvoidquery engine complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

Instead of partitioning files into virtual sub files as the patent describes, this invention inverts the approach by grouping multiple files into super files. This reversal simplifies the query engine because it works with consolidated file groups rather than fragmented sub-files, and enables incremental updates by allowing individual files to be added or removed from super files without requiring complete re-indexing.

Inventive Principle:

Principle #13The other way round (Inversion)

4Quantity of substance

If identical files are detected and skipped, then index size is reduced for exact duplicates, but near identical files are not addressed and index remains large

Engineering Contradiction:

Improveindex sizeVSAvoidsimilarity handling capability

Core Design Contradiction:

Quantity of substanceVSAdaptability or versatility

Solution Approach 1:

The patent changes the parameter of file identification from exact matching to similarity-based grouping. Instead of treating files as identical only when exactly the same, the system groups near-identical and similar files together based on content similarity metrics, then represents these groups with super files. This parameter change reduces index size for both exact duplicates and near-duplicates while maintaining appropriate search results.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS8898138B2Efficiently indexing and searching similar data

Publication Date: 2014.11.25 DELL EMC

AI summary

Techniques for efficiently indexing and searching similar data are described herein. According to one embodiment, in response to a query for one or more terms received from a client, a query index is accessed to retrieve a list of one or more super files. Each super file is associated with a group of similar files. Each super file includes terms and/or sequences of terms obtained from the associated group of similar files. Thereafter, the super files representing groups of similar files are presented to the client, where each of the super files includes at least one of the queried terms. Other methods and apparatuses are also described.