Adaptive Chunk Deduplication for WORM Lock Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data platforms face challenges in efficiently managing write once, read many (WORM) lock expiration times for chunkfiles, leading to suboptimal storage utilization and increased costs due to redundant chunk storage.
Innovation Solution
The implementation of adaptive deduplication techniques that selectively extend the WORM lock expiration time for chunkfiles based on policies, allowing new references to existing chunks while ensuring storage efficiency and reducing the number of superfluous chunks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If WORM lock expiration time is extended for all chunks in a chunkfile, then data integrity and compliance are maintained, but storage utilization deteriorates due to retention of superfluous chunks
Solution Approach 1:
The patent applies different WORM lock expiration times to different chunks within the same chunkfile based on their individual reference patterns. Chunks that are still referenced by active archives maintain their original expiration times for compliance, while chunks that have become unreferenced or superfluous have their expiration times reduced or are eligible for immediate deletion. This local differentiation resolves the contradiction by maintaining data integrity for necessary chunks while freeing storage space for superfluous ones.
2Quantity of substance
If deduplication is applied to all chunks, then storage efficiency improves, but data lock compliance deteriorates due to premature deletion of locked data
Solution Approach 1:
The patent implements a dynamic deduplication approach where the decision to deduplicate chunks is not static but adapts based on current archive references and WORM lock status. The system continuously evaluates which chunks are still protected by active WORM locks versus which chunks have expired or are no longer referenced. This dynamic evaluation allows deduplication to proceed safely for expired chunks while automatically preventing deduplication of chunks that would violate active data locks, thus resolving the contradiction between storage efficiency and compliance.
3Ease of operation
If new chunkfiles are created for each archive, then data lock management simplifies, but storage utilization deteriorates due to redundant chunk storage
Solution Approach 1:
The patent merges multiple archives into a single shared chunkfile structure where chunks are collectively managed across all archives. Instead of creating separate chunkfiles for each archive, the system consolidates chunks into a unified storage structure with a single WORM lock applied to the entire chunkfile. This merging approach resolves the contradiction by enabling sophisticated reference tracking across all archives simultaneously, allowing the system to identify and retain only the specific chunks actually referenced by each archive while freeing space for unreferenced chunks, thus improving storage utilization without complicating lock management.
Data Source
AI summary
Techniques are described for selectively extending a WORM lock expiration time for a chunkfile. An example method comprises identifying, by a data platform implemented by a computing system, a chunkfile that includes a chunk that matches data for an object of a file system; determining, by the data platform after identifying the chunkfile, whether to deduplicate the data for the object of the file system by adding a reference to the matching chunk, wherein determining whether to deduplicate the data comprises applying a policy to at least one of a property of the chunkfile or properties of one or more of a plurality of chunks included in the chunkfile; and in response to determining to not deduplicate the data for the object of the file system, causing a new chunk for the data for the object of the file system to be stored in a different, second chunkfile.