File Pre-processing for Document Fingerprinting Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Document fingerprinting techniques struggle to reliably identify protected information when it is copied from a known file and pasted into a different file, leading to potential data leakage.
Innovation Solution
A system and method for pre-processing files by determining classification parameters, such as file format and language, to standardize data representation, involving steps like text extraction, Unicode normalization, symbol replacement, and decompilation, to ensure that fingerprints of fixed-size fragments can accurately match common data strings across files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If document fingerprinting is performed on copied and pasted content, then data leakage detection capability is improved, but false negatives occur when content is reformatted
Solution Approach 1:
The patent applies preliminary normalization processing to both the reference document and the inspected document before generating fingerprints. This pre-processing step standardizes formatting, encoding, and structure, ensuring that identical content regardless of format differences will produce matching fingerprints. The normalization is performed in advance to eliminate formatting variations that would otherwise cause false negatives in detection.
Solution Approach 2:
The patent changes the parameters of document representation by normalizing various attributes such as text encoding, whitespace, punctuation, and structural elements. By transforming documents into a standardized parameter space, the system ensures that semantically identical documents have identical fingerprint representations, resolving the contradiction between detecting reformatted content and maintaining matching accuracy.
2Measurement precision
If pre-processing is applied to standardize file formats, then fingerprint matching accuracy is improved, but processing complexity increases
Solution Approach 1:
The patent segments the pre-processing task into distinct, modular normalization steps including text extraction, encoding standardization, whitespace normalization, punctuation handling, and structural normalization. Each step addresses a specific aspect of formatting variation independently, making the overall complex process manageable and systematic while achieving high fingerprint matching accuracy.
3Reliability
If comprehensive pre-processing is performed on all files, then detection reliability is improved, but processing time increases
Solution Approach 1:
The patent applies partial normalization by focusing pre-processing efforts only on the specific elements that impact fingerprint matching (text content, basic formatting, encoding) while skipping or simplifying other processing steps. This selective approach achieves sufficient detection reliability without the full overhead of comprehensive pre-processing, thereby reducing processing time while maintaining effectiveness.
Data Source
AI summary
The present disclosure relates to pre-processing of files to better prepare them for the process of comparing fingerprints of fragments of fixed size N from these files to other files that contain similar information but may be structured differently. The pre-processing method and system are applied to files with known protected data before fingerprints of some of the N-fragments from these files are added to the digital fingerprint library and to the unknown files before the fingerprints of some of their N-fragments are compared to the fingerprints of other N-fragments of data stored in the digital fingerprint library.