Deduplicated Data Management Using Boolean Flags
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing deduplicated data in highly dynamic large scale distributed computing and storage systems is inefficient due to the high computational and networking resources required for maintaining accurate reference counts, often leading to bottlenecks and user experience delays, especially when relying on semaphores and atomic operations for concurrent access.
Innovation Solution
Implementing collision-free deduplication data usage indicators using Boolean deletion and usage flags, which reduce the need for semaphores and atomic operations by periodically scanning for references and updating metadata only when necessary, thereby minimizing resource consumption and improving data management efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If reference counts are continually updated to maintain accurate tracking of deduplicated data access, then data management accuracy is improved, but computing and networking resource consumption increases
Solution Approach 1:
The patent implements periodic scanning of metadata stores instead of continuous updating of reference counts. The system performs scans at scheduled intervals to detect changes in data usage patterns, eliminating the need for constant computational updates while maintaining accurate tracking of deduplicated data access over time.
Solution Approach 2:
The system uses the natural lifecycle events of data operations (writes, reads, deletions) to automatically trigger metadata updates without requiring active monitoring or continuous computational intervention. The metadata store self-updates when data is accessed or modified, reducing the burden on dedicated resource tracking mechanisms.
2Reliability
If atomic operations and semaphores are used to handle concurrent access to reference counts, then data consistency is improved, but user experience deteriorates due to delays
Solution Approach 1:
The patent extracts the concurrent access problem from the data path by separating reference count tracking from the primary data operations. Instead of using atomic operations for every data access, the system uses periodic scans that periodically check metadata, eliminating blocking operations and improving user experience while maintaining consistency through periodic synchronization.
Solution Approach 2:
The system dynamically adjusts the scanning frequency and metadata update strategies based on system conditions and data access patterns. This allows the system to maintain data consistency without rigid, blocking operations, enabling more flexible and responsive data management that adapts to varying workload conditions.
3Measurement precision
If metadata is updated frequently to track all references to deduplicated data, then reference tracking accuracy is improved, but device complexity increases
Solution Approach 1:
The patent segments the metadata management function into separate periodic scanning processes rather than a single monolithic update mechanism. Different metadata stores can be scanned independently at different frequencies, allowing the system to track references accurately while managing complexity through modular, independent management of different data types and access patterns.
Data Source
AI summary
Facilitating deduplication of data in a computing system without managing access to reference count variables. A method embodiment commences upon detecting first data unit and calculating a first checksum value. At a later time, a second data unit is received and the system calculates a second checksum value. If the second checksum value is the same as the first checksum value, then the first data unit and the second data unit are the same data and need not be duplicated. In such cases, an entry in the metadata points to the location of the first data unit that is already stored. Additional metadata entries are made in the metadata to associate a Boolean usage state flag and a Boolean deletion state flag with the second checksum value. Periodically scans of the metadata are performed. When both Boolean flags are in a particular state, the deduplicated data is deleted.