Fuzzy Hashing for Encrypted Document Classification

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing document search and classification techniques face limitations, including inability to operate on encrypted documents and high computational costs due to the need for rendering and comparing multiple documents at runtime, especially with large datasets.

Innovation Solution

The use of fuzzy hashing based on the structural characteristics of Portable Document Format (PDF) documents, where statistics about the COS object tree are compiled and input into a rolling hash function to generate a digital fingerprint, allowing for classification and search of structurally similar documents, including encrypted ones, without the need for rendering.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If text extraction or rendering is performed on documents, then document content can be analyzed and classified, but encrypted documents cannot be processed

Engineering Contradiction:

Improvecapability to process encrypted documentsVSAvoidinability to access document content

Core Design Contradiction:

Adaptability or versatilityVSLoss of information

Solution Approach 1:

Instead of extracting text or rendering documents to analyze content, the patent inverts the approach by analyzing the document structure directly from the binary file format. This allows processing of encrypted documents because the structural metadata (COS object tree, page count, font information, etc.) remains accessible without decryption, while still enabling document classification and homology detection.

Inventive Principle:

Principle #13The other way round (Inversion)

2Measurement precision

If rendering and comparison of multiple documents is performed at runtime, then document similarities can be identified, but computational costs increase significantly with large numbers of documents

Engineering Contradiction:

Improvedocument similarity detection accuracyVSAvoidcomputational cost

Core Design Contradiction:

Measurement precisionVSPower

Solution Approach 1:

The patent pre-computes structural fingerprints for documents during indexing rather than at runtime. By extracting structural features (COS object tree statistics, page information, font descriptors) and generating fingerprint hashes in advance, the system avoids expensive rendering and comparison operations during search. This preliminary processing significantly reduces runtime computational costs while maintaining similarity detection accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Instead of rendering full document representations for comparison, the patent creates simplified structural copies in the form of fingerprints. These fingerprints capture essential structural characteristics (object counts, hierarchy relationships, metadata) in a compact format that enables efficient comparison without the computational overhead of full document rendering.

Inventive Principle:

Principle #26Copying

3Adaptability or versatility

If structural characteristics are used for document classification, then encrypted documents can be processed, but the ability to detect content-based similarities is reduced

Engineering Contradiction:

Improveability to process encrypted documentsVSAvoidcontent similarity detection accuracy

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent combines multiple structural features into a composite fingerprint representation. Instead of relying on a single structural attribute, it integrates COS object tree statistics, page information, font descriptors, and metadata into a comprehensive structural fingerprint. This composite approach captures richer structural patterns that correlate with content similarity, partially compensating for the inability to access actual text content in encrypted documents.

Inventive Principle:

Principle #40Composite materials

Data Source

PatentUS9996603B2Detecting homologies in encrypted and unencrypted documents using fuzzy hashing

Publication Date: 2018.06.12 ADOBE INC

AI summary

Techniques are provided for automatically detecting homologies between documents based on structural characteristics. Various statistics relating to the COS structure of a PDF document are compiled. The statistics are input into a rolling hash function to generate a digital fingerprint of the document. Fingerprints from two similar documents will have small edit distances between them, and can therefore be classified similarly or provided as results to a fingerprint-based search. For example, an unclassified document may be classified in the same class as a representative document where the fingerprints of the two documents have a small edit distance between them. Since the structure of the document is used instead of the text content or renderings, it is possible to operate on encrypted documents. Further, representative elements of a particular class of documents can be selected for comparison against a target document for improved resolution of the results.