Email Attachment De-duplication via Reference Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database and memory management systems face inefficiencies in managing and reducing duplicate files (blobs) across computer systems, leading to wasted disk space, I/O bandwidth, and network bandwidth.
Innovation Solution
A system and method for de-duplicating files and attachments within a file system by checking for existing data, managing attachment states, and sharing references, which includes detecting and repairing corrupted attachments to prevent duplicate storage and ensure data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the system stores all attachments individually without de-duplication, then data integrity and accessibility are maintained, but disk space is wasted and storage efficiency deteriorates
Solution Approach 1:
The patent merges duplicate attachments by storing only one copy in the attachment store while maintaining multiple references to it. When an attachment is uploaded, the system checks if identical attachments already exist in the store. If duplicates are found, the system consolidates them into a single stored copy with multiple reference pointers, thereby saving disk space while preserving accessibility for all users.
Solution Approach 2:
The patent uses reference copying instead of physical copying. Instead of storing multiple physical copies of identical attachments, the system creates reference pointers that point to the single stored attachment. This allows multiple users to access the same attachment without duplicating the actual data, resolving the contradiction between space efficiency and accessibility.
2Quantity of substance
If the system performs comprehensive checks for duplicate attachments, then storage efficiency is improved, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary actions by generating and storing hash values of attachments during the upload process. Before storing an attachment, the system computes its hash value and searches the attachment store for matching hashes. This preliminary hashing action enables快速 identification of duplicates without requiring full file comparisons, thereby improving storage efficiency while minimizing processing time overhead.
3Loss of energy
If the system shares attachment references across multiple users, then network bandwidth usage is reduced, but data security and state management complexity increase
Solution Approach 1:
The patent introduces an intermediary attachment store that acts as a centralized repository for shared attachments. Instead of allowing direct peer-to-peer sharing that could lead to inconsistent states, the system uses the attachment store as an intermediary. All attachment operations go through this centralized store, which maintains the canonical version of each attachment and manages reference counts, ensuring state consistency while enabling efficient reference sharing across users.
4Reliability
If the system automatically repairs corrupted attachments, then data reliability is improved, but additional processing steps and time are required
Solution Approach 1:
The patent implements self-service repair mechanisms where the system automatically detects and repairs corrupted attachments without requiring manual intervention. When an attachment is retrieved or accessed, the system checks its integrity using hash verification. If corruption is detected, the system automatically repairs the attachment by regenerating it from the stored hash or requesting a fresh copy from the source, thereby maintaining high data reliability while minimizing the impact on processing throughput through automated operations.
Data Source
AI summary
Systems and techniques of de-duplicating file and/or blobs within a file system are presented. In one embodiment, an email system is disclosed wherein the email system receives email messages comprising a set of associated attachments. The system determines whether the associated attachments have been previously stored in the email system, the state of the stored attachment, and if the state of the attachment is appropriate for sharing copies of the attachment, then providing a reference to the attachment upon a request to share the attachment. In another embodiment, the system may detect whether stored attachments are corrupted and, if so, attempt to repair the attachment, and possibly, prior to sharing references to the attachment.