User-Level Deduplication via Content Salting for Accurate Billing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication systems struggle to accurately attribute and bill users for deduplication services, as they are not designed to distinguish between individual users, leading to inefficient resource usage and potential exploitation by users who pre-compress data to reduce costs.
Innovation Solution
Implementing user-level deduplication by assigning a unique salt to each user, which is used to hash and identify duplicate data, allowing for user-specific deduplication within a shared environment, enabling accurate measurement and billing of deduplication services.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If global deduplication systems are used to identify and eliminate duplicate data across multiple users, then storage space efficiency is improved, but the ability to accurately attribute and bill services to individual users deteriorates
Solution Approach 1:
The patent segments the global deduplication system into user-level components by introducing unique user identifiers (UIDs) and content salts for each user. Data is hashed with user-specific salts to create user-level deduplication domains, allowing the system to maintain global storage efficiency while enabling precise user-level measurement and attribution of deduplication services.
2Quantity of substance
If users pre-compress data before sending to the deduplication system, then the amount of logical capacity consumed is reduced, but the effectiveness of the deduplication system deteriorates
Solution Approach 1:
The system performs preliminary hashing of data with user-specific content salts before data is sent to the deduplication system. This pre-hashing action creates a unique fingerprint for each user's data that enables accurate measurement of deduplication effectiveness at the user level, preventing users from exploiting the system by pre-compressing data to artificially reduce measured consumption.
3Ease of operation
If all users are treated equally with respect to data reduction techniques, then administrative complexity is reduced, but fairness in cost attribution deteriorates
Solution Approach 1:
The patent implements local quality by applying user-specific content salts and UIDs to each user's data independently. This allows the system to maintain simple global administration while enabling differentiated user-level measurement and fair cost attribution, as each user's deduplication effectiveness is measured independently using their own salt and UID.
Data Source
AI summary
In one example, a method for managing data includes detecting, in a shared multi-tenant deduplication system, a data stream that is specific to a particular user and that includes one or more user objects each having one or more data chunks. The data associated with the user is salted by associating a user-specific salt with the data chunks of the one or more user objects to form user-specific combinations of the form ((user-specific salt)+(user data chunk)). Finally, the user data is processed using the user-specific salt.