Deduplication Repository Iterator Interface for Shared File Regions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file system deduplication processes do not provide external applications with information about shared file regions, leading to inefficient processing and increased resource usage as each application must independently track and process duplicate data segments multiple times.
Innovation Solution
A deduplication repository with an iterator interface that returns file regions and associated objects, along with a data structure that tracks checksum values and timestamps, allowing external applications to efficiently identify and process shared regions without maintaining their own processing state.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If file system deduplication processes keep internal information about duplicate files and portions, then storage requirements are reduced, but external applications cannot access this information and must independently track processed data
Solution Approach 1:
The patent introduces an intermediary component (deduplication information provider/interface) that mediates between the file system deduplication process and external applications. This intermediary exposes the internal deduplication information through standardized interfaces, allowing applications to query and utilize shared region information without directly accessing the file system's internal structures.
2Reliability
If each application independently maintains its own index to track processed data, then applications can avoid reprocessing duplicate data, but resource usage increases and processing efficiency decreases
Solution Approach 1:
The patent merges the tracking functionality of multiple applications into a single shared deduplication information store. Instead of each application maintaining separate indexes, they all query the same deduplication information provided by the file system, consolidating the tracking effort and eliminating redundant maintenance overhead.
Solution Approach 2:
The deduplication information provider serves multiple applications simultaneously with a single interface, making the deduplication tracking capability universal across different applications. This multi-functional interface allows various applications (backup, antivirus, search, etc.) to benefit from the same deduplication information without requiring application-specific implementations.
3Reliability
If applications process duplicate segments multiple times, then comprehensive processing is ensured, but bandwidth consumption and processing time increase
Solution Approach 1:
The system performs preliminary deduplication analysis at the file system level before applications process the data. By pre-identifying and marking duplicate regions, the system enables applications to skip already-processed segments, avoiding redundant processing while ensuring comprehensive coverage through the shared deduplication state.
Data Source
AI summary
An interface is disclosed that makes information obtained from a file deduplication process available to an application for the efficient operation thereof. A data deduplication repository is scanned to determine a plurality of file segments and respective checksum values associated with the segments. A data structure is generated that allows shared segments to be identified by indexing using a common checksum value. The segments also indicate the file to which they belong and may also include a timestamp value. This data structure is updated as files are modified, etc. The data structure is accessible to an application program so that the application program can readily determine which segments are shared between multiple files. With this information, the application can efficiently process the segment once rather than multiple times. Timestamps can be used by the application to efficiently identify only those segments that were accessed after a given time.