Office File Deduplication Using Metadata-Independent Content Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication systems fail to effectively identify and remove duplicate files across different repositories due to metadata manipulation by web-based document sharing or hosting platforms, leading to inefficient storage and resource usage.
Innovation Solution
A system that classifies file components into metadata and non-metadata portions, generates a manifest for each non-metadata portion, normalizes the output data stream, and uses a hash code comparison to identify and remove duplicate files, regardless of metadata changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If files are uploaded to a web-based document sharing platform, then the files can be accessed and shared through the platform, but the metadata is changed or modified making deduplication impossible
Solution Approach 1:
The file is segmented into metadata portions and non-metadata portions. The system generates separate hash codes for each non-metadata portion, allowing deduplication to focus on the actual content while ignoring modified metadata. This segmentation enables the system to maintain deduplication capability even when metadata changes during platform operations.
Solution Approach 2:
The system extracts and excludes metadata portions from the deduplication process by generating hash codes only for non-metadata portions. This extraction approach removes the interfering element (metadata) that causes deduplication failures, allowing the system to reliably identify duplicate files based on their actual content.
2Quantity of substance
If duplicate files are not identified and removed, then all files can be retained in the database, but storage space is wasted and resources are burdened
Solution Approach 1:
The system replaces traditional full-file comparison methods with hash code comparison. By generating hash codes from normalized non-metadata portions and comparing these compact representations, the system efficiently identifies duplicates without burdening resources with extensive file I/O and processing, enabling scalable deduplication of large datasets.
3Device complexity
If traditional deduplication methods are used, then the process is simple, but it fails when metadata is manipulated by another system
Solution Approach 1:
The system performs preliminary actions by classifying file portions into metadata and non-metadata categories before generating hash codes. This preliminary classification ensures that only relevant content portions are hashed, making the subsequent deduplication process both simple and reliable, as it systematically handles metadata manipulation issues before they affect duplicate detection.
Data Source
AI summary
A system includes a processor and a memory. When executed by the processor, the processor is caused to receive a first file including one or more components, parse the first file into a metadata portion and one or more non-metadata portions, generate a manifest for each of the one or more non-metadata portions, generate an output data stream including component manifest and data pairs for each of the one or more non-metadata portions, normalize the output data stream, generate a first hash code corresponding to the normalized output data stream and compare the first hash code to a plurality of hash codes. If the first hash code matches any hash code of the plurality of hash codes, the processor is caused to prevent the first file from being stored in the database or automatically remove the corresponding file associated with the hash code matched with the first hash code.