Data deduplication method and apparatus, computer device, and storage medium

Abstract

The present application relates to a data deduplication method and apparatus, a computer device, and a storage medium. The method comprises: dividing multiple pieces of data to be deduplicated into at least one data subset (S102); for each data subset, determining, on the basis of a probability filter corresponding to the data subset, a mapping value of each piece of data to be deduplicated in the data subset (S104); on the basis of the mapping value of each piece of data to be deduplicated in the data subset, correspondingly determining a data filtering result of each piece of said data in the data subset (S106); and, on the basis of the data filtering result of each piece of data to be deduplicated in the data subset, performing deduplication on the data subset (S108).

Description

Data deduplication methods, apparatus, computer equipment and storage media

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on November 19, 2024, application number 2024116512253, entitled "Data Deduplication Method, Apparatus, Computer Equipment and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of data processing technology, and in particular to a data deduplication method, apparatus, computer equipment, and storage medium. Background Technology

[0004] With the development of big data technology, data deduplication has become a key step in data preprocessing, which is crucial for reducing storage overhead and improving data deduplication efficiency.

[0005] The core idea of ​​data deduplication is to identify and delete duplicate data using some method. Common data deduplication methods include hash tables, sorting methods, and Bloom filters. Taking the Bloom filter as an example, it is a highly space-efficient probabilistic data structure that can determine whether an element exists in a set with very little memory consumption and time cost, thus quickly searching and deduplicating data. However, the Bloom filter has a high false positive rate, resulting in lower data deduplication accuracy. Summary of the Invention

[0006] According to various embodiments of this application, a data deduplication method, apparatus, computer device, and storage medium are provided.

[0007] Firstly, this application provides a data deduplication method, the method comprising:

[0008] Divide multiple sets of data to be deduplicated into at least one subset; each subset corresponds to an independent probability filter.

[0009] For each data subset, the mapping value of each deduplicated data in the data subset is determined according to the probability filter corresponding to the data subset;

[0010] Based on the mapping values ​​of each deduplicated data in the data subset, the data filtering results for each deduplicated data in the data subset are determined accordingly; the data filtering results are used to indicate whether the deduplicated data exists in the probability filter; and

[0011] Based on the data filtering results of each deduplicated data in the data subset, the data subset is deduplicated.

[0012] In one embodiment, dividing the plurality of data to be deduplicated into at least one subset includes:

[0013] Obtain the data features of each of the data to be deduplicated;

[0014] Based on the data characteristics of each of the deduplicated data, the local sensitivity hash value of each of the deduplicated data is determined using a locality-sensitive hash function; and

[0015] Based on the local sensitive hash value of each of the deduplicated data, each of the deduplicated data is divided into at least one data subset.

[0016] In one embodiment, the probabilistic filter includes a Bloom filter, and the mapping value includes a hash value, wherein determining the mapping value of each deduplicated data in the data subset based on the probabilistic filter corresponding to the data subset includes:

[0017] Based on the Bloom filter corresponding to the data subset, determine the hash function corresponding to the data subset; and

[0018] The hash value of each piece of data to be deduplicated in the data subset is determined according to the hash function.

[0019] In one embodiment, the probability filter includes a bit array, and the step of determining the data filtering result for each deduplicated data in the data subset according to the mapping value of each deduplicated data in the data subset includes:

[0020] Iterate through the corresponding positions of the mapping values ​​of each deduplicated data in the data subset within the data set; and

[0021] For any duplicate data to be deduplicated in the data subset, if there is a 0 among the values ​​corresponding to the duplicate data to be deduplicated obtained through the traversal, it is determined that the data filtering result of the duplicate data to be deduplicated is not present in the probability filter; if there is no 0 among the values ​​corresponding to the duplicate data to be deduplicated obtained through the traversal, it is determined that the data filtering result of the duplicate data to be deduplicated is present in the probability filter.

[0022] In one embodiment, the step of performing deduplication processing on the data subset based on the data filtering results of each deduplicated data in the data subset includes:

[0023] For any one piece of data to be deduplicated in the data subset, if the data filtering result of the data to be deduplicated indicates that it exists in the probability filter, the data to be deduplicated is stored in the duplicate database corresponding to the data subset; and

[0024] If the data filtering result of the data to be deduplicated is that it does not exist in the probability filter, the data to be deduplicated is stored in the result database corresponding to the data subset.

[0025] In one embodiment, the method further includes:

[0026] Based on the result database and duplicate database corresponding to the data subset, the false alarm rate of the probability filter is determined; and

[0027] If the false alarm rate exceeds the allowable error range of the probability filter, the probability filter is adjusted, and the adjusted probability filter is used to determine the data filtering result of the subsequent deduplicated data until the false alarm rate meets the allowable error range.

[0028] In one embodiment, adjusting the probability filter of the data subset includes:

[0029] Adjust the filtering parameters of the probability filter for the data subset, wherein the filtering parameters include at least one of the number of hash functions and the length of the bit array;

[0030] Wherein, when the filtering parameters include the number of hash functions and the length of the bit array, the changing trends of the number of hash functions and the length of the bit array are the same.

[0031] Secondly, this application provides a data deduplication device, the device comprising:

[0032] The partitioning module is used to divide multiple data sets to be deduplicated into at least one subset; each subset corresponds to an independent probability filter.

[0033] The mapping module is used to determine the mapping value of each deduplicated data in the data subset according to the probability filter corresponding to the data subset.

[0034] The filtering module is used to determine the data filtering result for each piece of data to be deduplicated in the data subset based on the mapping value of each piece of data to be deduplicated in the data subset; the data filtering result is used to indicate whether the data to be deduplicated exists in the probability filter; and

[0035] The deduplication module is used to perform deduplication processing on the data subset based on the data filtering results of each deduplicated data in the data subset.

[0036] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0037] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0038] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0039] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the disclosed drawings without creative effort.

[0041] Figure 1 is a flowchart of one of the data deduplication methods in some embodiments;

[0042] Figure 2 is a second schematic flowchart of a data deduplication method in some embodiments;

[0043] Figure 3 is a flowchart of one of the data deduplication methods in some embodiments;

[0044] Figure 4 is a flowchart of one of the data deduplication methods in some embodiments;

[0045] Figure 5 is a flowchart of the data deduplication method in some embodiments;

[0046] Figure 6 is a flowchart of the data deduplication method in some embodiments;

[0047] Figure 7 is a flowchart of the data deduplication method in some embodiments, number seven.

[0048] Figure 8 is a structural block diagram of the data deduplication device in some embodiments;

[0049] Figure 9 is an internal structure diagram of a computer device in some embodiments. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] In one embodiment, as shown in Figure 1, a data deduplication method is provided. This embodiment illustrates the method by applying it to a terminal. It is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps S102 to S108.

[0052] S102: Divide multiple sets of data to be deduplicated into at least one subset of data.

[0053] The data to be deduplicated can be of any type. In this embodiment, the data to be deduplicated refers to data that requires deduplication. For example, the data to be deduplicated may be data collected by an internet platform, data collected by a customer relationship management (CMR) system, data collected by an application, etc. The data to be deduplicated can be determined according to the actual data deduplication scenario, and is not limited here.

[0054] Each data subset includes at least one piece of data to be deduplicated. The pieces of data to be deduplicated within each data subset have a certain degree of similarity; in other words, at least one data feature of the pieces of data to be deduplicated within each data subset has a high degree of similarity or is the same. For example, multiple pieces of data to be deduplicated can be divided into at least one data subset according to preset rules. These preset rules are pre-defined and can be set according to the actual scenario; no specific limitations are imposed here.

[0055] Each data subset corresponds to an independent probability filter. The probability filters for different data subsets may be the same or different. In this embodiment, the probability filter can be customized for each data subset. The probability filter is used to determine whether the data to be deduplicated in the data subset is repeated. A probability filter is a probabilistic data structure that can determine the existence of a piece of data to be deduplicated within the probability filter with very low memory consumption and time cost, thereby quickly searching and deduplicating data. For example, the probability filter includes at least one of a Bloom filter, a counting Bloom filter, a cuckoo filter, a dynamic Bloom filter, and a quotient filter.

[0056] S104: For each data subset, determine the mapping value of each data in the data subset based on the probability filter corresponding to the data subset.

[0057] The mapping value refers to the value obtained after calculating the deduplicated data according to a preset algorithm. This preset algorithm may include a preset function, such as a hash function. The mapping value for each piece of data to be deduplicated is determined by the corresponding probability filter. The mapping values ​​for all data to be deduplicated within the same data subset have the same length. The mapping values ​​for all data to be deduplicated in different data subsets can have the same or different lengths. The mapping value can be determined based on the probability filter corresponding to each data subset, and this is not limited here. A probability filter can be understood as mapping each piece of data to be deduplicated within a data subset to a value with a preset fixed length, i.e., the mapping value.

[0058] S106: Based on the mapping values ​​of each deduplicated data in the data subset, determine the data filtering results for each deduplicated data in the data subset.

[0059] There is a preset mapping relationship between the mapping values ​​and the data filtering results. For example, based on the mapping values ​​and mapping relationships of each piece of data to be deduplicated in the data subset, the data filtering results for each piece of data to be deduplicated in the data subset can be determined respectively. The data filtering results indicate whether the data to be deduplicated exists in the probability filter. The data filtering results include whether the data to be deduplicated exists in the probability filter, or whether the data to be deduplicated does not exist in the probability filter.

[0060] S108: Based on the data filtering results of each deduplicated data in the data subset, perform deduplication processing on the data subset.

[0061] If the data filtering result for the data to be deduplicated shows that the data exists in the probability filter, it means that the data to be deduplicated is duplicated in the data subset and needs to be deleted from the subset. If the data filtering result for the data to be deduplicated shows that the data does not exist in the probability filter, it means that the data to be deduplicated is not duplicated in the data subset and does not need to be deleted from the subset. For example, based on the data filtering results for each data subset, duplicate data in the subset is deleted, resulting in a dataset without duplicate data.

[0062] The aforementioned data deduplication method divides multiple sets of data to be deduplicated into at least one data subset. For each data subset, a mapping value for each set of data to be deduplicated is determined based on the probability filter corresponding to the data subset. Based on the mapping value of each set of data to be deduplicated, a data filtering result is determined for each set of data to be deduplicated. Based on the data filtering result of each set of data to be deduplicated, the data subset is deduplicated. The data deduplication method provided in this application divides the data to be deduplicated into at least one data subset, achieving a more refined distinction between similar data. Furthermore, an independent probability filter is applied to each data subset. That is, an independent probability filter is designed adaptively and specifically for each data subset, avoiding the use of a uniform probability filter for deduplication of different data with large differences. This reduces the probability filter's misjudgment of duplicate data, improving not only the deduplication efficiency but also significantly improving the accuracy of data deduplication.

[0063] In one embodiment, as shown in FIG2, step S102, dividing multiple data to be deduplicated into at least one data subset, includes the following steps S202 to S206.

[0064] S202: Obtain the data characteristics of each piece of data to be deduplicated.

[0065] Data characteristics of the data to be deduplicated are used to describe the attributes and characteristics of the data. For example, data characteristics include at least one of the following: data source, timestamp, data type, and data format. The data source refers to the origin of the data to be deduplicated; for example, the data source is server A. The timestamp records the time of any operation in the generation or modification of the data to be deduplicated. The data type defines the type and scope of the data to be deduplicated; for example, the data type is floating-point. The data format refers to the specific representation of the data to be deduplicated during at least one operation, such as storage or transmission; for example, the data format is binary.

[0066] S204: Based on the data characteristics of each piece of data to be deduplicated, determine the local sensitivity hash value of each piece of data to be deduplicated using the local sensitivity hash function.

[0067] For example, locality-sensitive hash functions include at least one of MinHash, SimHash, LSH Forest, Euclidean LSH, Cosine LSH, and Multi-Probe LSH.

[0068] Taking the data characteristics of the data to be deduplicated, including the data source, as an example, after obtaining the data source of each data to be deduplicated based on the above step S202, the local sensitive hash value of each data to be deduplicated is determined according to the data source of each data to be deduplicated based on the local sensitive hash function.

[0069] It should be noted that in practical applications, an appropriate locality-sensitive hash function can be selected according to the actual application scenario. The above is only an illustrative example and should not be construed as a specific limitation on locality-sensitive hash functions.

[0070] S206: Based on the local sensitive hash value of each piece of data to be deduplicated, divide each piece of data to be deduplicated into at least one subset of data.

[0071] For example, data to be deduplicated with the same or similar locality-sensitive hash values ​​are divided into the same data subset, so that each piece of data to be deduplicated is divided into at least one data subset. Here, "similar locality-sensitive hash values" means that the locality-sensitive hash values ​​meet a preset hash value range, and the hash value ranges corresponding to different data subsets are different.

[0072] The aforementioned data deduplication method obtains the data features of each piece of data to be deduplicated, determines the local sensitive hash value of each piece of data based on the data features of each piece of data based on the local sensitive hash function, and divides each piece of data into at least one data subset based on the local sensitive hash value of each piece of data. This embodiment introduces local sensitive hashing technology to divide the data to be deduplicated into at least one data subset based on the data features of the data to be deduplicated, thereby mapping similar pieces of data to the same or similar data subsets. That is, it quickly finds similar items in a large-scale dataset, and then applies an independent probability filter to each data subset for deduplication processing. This not only maintains the similarity between the pieces of data to be deduplicated, but also achieves a more refined distinction between similar data, which helps to reduce false positives. While ensuring query efficiency, it significantly improves the accuracy of data deduplication.

[0073] In one embodiment, the probabilistic filter includes a Bloom filter, and the mapping value includes a hash value. As shown in Figure 3, step S104, determining the mapping value for each deduplicated data in the data subset based on the probabilistic filter corresponding to the data subset, includes steps S302 and S304.

[0074] S302: Determine the hash function corresponding to the data subset based on the Bloom filter corresponding to the data subset.

[0075] Each Bloom filter has at least one hash function. Different Bloom filters can have the same or different number of hash functions. The number of hash functions can be determined based on the data subset and the data deduplication requirements, and is not limited here.

[0076] For example, the hash function includes at least one of the following: Secure Hash Algorithm 256 (SHA-256), SHA-3, BLAKE2, Murmur Hash, City Hash, FNV-1a, and XX Hash. Different Bloom filters may use the same or different types of hash functions. The type of hash function can be determined based on the data subset and the data deduplication requirements, and is not limited here.

[0077] S304: Determine the hash value of each piece of data to be deduplicated in the data subset according to the hash function.

[0078] In the application, for each piece of data to be deduplicated in each data subset, the hash value of each piece of data to be deduplicated is determined according to the hash function of the corresponding probability filter. Thus, based on the hash value of each piece of data to be deduplicated, the data filtering result of each piece of data to be deduplicated in the data subset is determined. Then, based on the data filtering result of each piece of data to be deduplicated in the data subset, the data subset is deduplicated, thereby achieving data deduplication.

[0079] The above data deduplication method determines the hash function corresponding to the data subset based on the Bloom filter corresponding to the data subset, and determines the hash value of each deduplicated data in the data subset according to the hash function. This implementation uses the Bloom filter to perform hash processing on the data to be deduplicated, which can determine whether a deduplicated data exists in the Bloom filter with very small memory consumption and time cost, thereby quickly searching for and deduplicating data and improving the efficiency of data deduplication.

[0080] In one embodiment, the probabilistic filter includes a bit array. A bit array is a data structure used to store and manipulate bits (0 or 1). Each element in the bit array is a bit, either 0 or 1. For each probabilistic filter, the initial value of each bit in its bit array is 0. The length of the bit array, or the size of the bit array, is preset. The bit array length of each probabilistic filter can be the same or different, and the bit array length can be determined based on the data subset and data deduplication requirements; no limitation is made here.

[0081] Based on the above, as shown in Figure 4, step S106, according to the mapping value of each deduplicated data in the data subset, determines the data filtering result of each deduplicated data in the data subset, including the following steps S402 and S404.

[0082] S402: Iterate through the mapping values ​​of each data to be deduplicated in the data subset and find the corresponding values ​​in the bit array.

[0083] In the application, the value at the corresponding position in the bit array is found based on the value of each bit of the mapping value of each piece of the data to be deduplicated. For example, if the mapping value of the data to be deduplicated, D1, is 256, then the values ​​at the 2nd, 5th, and 6th bits in the corresponding bit array are searched.

[0084] S404: For any duplicate data to be deduplicated in the data subset, if there is a 0 among the values ​​corresponding to the duplicate data to be deduplicated obtained through the traversal, the data filtering result of the duplicate data to be deduplicated is determined to be absent from the probability filter; if there is no 0 among the values ​​corresponding to the duplicate data to be deduplicated obtained through the traversal, the data filtering result of the duplicate data to be deduplicated is determined to be present in the probability filter.

[0085] For any duplicate data point in the data subset, if any of the values ​​corresponding to the duplicate data point are 0, it indicates that the duplicate data point definitely does not exist in the probability filter. Therefore, the data filtering result for the duplicate data point is determined to be absent from the probability filter. For example, if the values ​​of the 2nd, 5th, and 6th bits of the bit array corresponding to the duplicate data point D1 are 101, then the data filtering result for the duplicate data point D1 is determined to be absent from the corresponding probability filter.

[0086] For any duplicate data point in the data subset, if none of the values ​​corresponding to the duplicate data point obtained through iteration are 0, it indicates that the duplicate data point may exist in the probability filter. Therefore, the data filtering result of the duplicate data point is determined to exist in the probability filter. For example, if the values ​​of the 2nd, 5th, and 6th bits of the bit array corresponding to the duplicate data point D1 are 111, then the data filtering result of the duplicate data point D1 is determined to exist in the corresponding probability filter.

[0087] In the application, for data to be deduplicated that does not exist in the probability filter, the corresponding position of the data to be deduplicated is set to 1 in the bit array, so that the data to be deduplicated is stored in the probability filter, and then used to determine whether there is data that is the same as the data to be deduplicated.

[0088] The above data deduplication method iterates through the corresponding positions of the mapping values ​​of each deduplicated data in the data subset. For any deduplicated data in the data subset, if there is a 0 among the values ​​corresponding to the deduplicated data obtained through the iteration, it is determined that the data filtering result of the deduplicated data does not exist in the probability filter; otherwise, if there is no 0 among the values ​​corresponding to the deduplicated data obtained through the iteration, it is determined that the data filtering result of the deduplicated data exists in the probability filter. In other words, by using the mapping values ​​of the deduplicated data to query the data, compared with directly querying the original data, the mapping values ​​reduce the amount of data, thereby reducing the data query complexity, increasing the data query speed, and thus improving the efficiency of deduplicating the data based on the values ​​obtained from the query.

[0089] In one embodiment, as shown in FIG5, step S108, based on the data filtering results of each deduplicated data in the data subset, performs deduplication processing on the data subset, including the following steps S502 and S504.

[0090] S502: For any duplicate data to be deduplicated in the data subset, if the data filtering result of the data to be deduplicated is that it exists in the probability filter, store the data to be deduplicated in the duplicate database corresponding to the data subset.

[0091] S504: If the data filtering result of the data to be deduplicated does not exist in the probability filter, store the data to be deduplicated in the result database corresponding to the data subset.

[0092] Each data subset corresponds to a duplicate database and a result database. Data filtering results in data that exists in the probability filter, indicating that the data to be deduplicated is duplicated within the data subset; therefore, the data to be deduplicated is stored in the duplicate database corresponding to the data subset. Data filtering results in data that does not exist in the probability filter, indicating that the data to be deduplicated is not duplicated within the data subset; therefore, the data to be deduplicated is stored in the result database corresponding to the data subset. The data in the duplicate database is duplicated from the data in the result database; that is, the data in the duplicate database is the set of duplicate data within the data subset, and the result database is the set of data after removing duplicate data from the data subset.

[0093] The above data deduplication method, for any one piece of data to be deduplicated in a data subset, if the data filtering result of the data to be deduplicated is that it exists in the probability filter, then the data to be deduplicated is stored in the duplicate database corresponding to the data subset; if the data filtering result of the data to be deduplicated is that it does not exist in the probability filter, then the data to be deduplicated is stored in the result database corresponding to the data subset. This achieves deduplication processing of duplicate data in the same data subset, so that the deduplicated result database can be applied, avoiding the resource consumption caused by duplicate data, and helping to improve data processing efficiency and reduce resource consumption.

[0094] In one embodiment, as shown in FIG6, the data deduplication method further includes the following steps S602 and S604.

[0095] S602: Determine the false alarm rate of the probability filter based on the result database and duplicate database corresponding to the data subset.

[0096] For example, based on the result database and duplicate database corresponding to the data subset, false positive data in the duplicate database that does not exist in the result database is identified, and the false positive rate of the probability filter is determined based on the false positive data. For instance, if the result database includes {1,2,3,4,5,6,7,8} and the duplicate database includes {0,1}, and the false positive data includes 0, then the false positive rate is 10%.

[0097] S604: If the false alarm rate exceeds the allowable error range of the probability filter, adjust the probability filter and use the adjusted probability filter to determine the data filtering result of subsequent data until the false alarm rate meets the allowable error range.

[0098] The allowable error range is preset and can be set according to the data deduplication accuracy requirements; it is not limited here. The allowable error range for each probability filter can be the same or different. If the false alarm rate exceeds the allowable error range of the probability filter, it indicates that the deduplication accuracy of the probability filter does not meet the data deduplication accuracy requirements, and the probability filter needs to be optimized and adjusted. When the false alarm rate exceeds the allowable error range of the probability filter, the result database and duplicate database can be updated based on the false alarm data determined in step S602 above to obtain accurate result data and duplicate database.

[0099] In the application, the adjusted probability filter is used to process subsequently collected data belonging to the corresponding subset of the probability filter, until the false alarm rate meets the allowable error range. For example, for subsequently collected data to be deduplicated (hereinafter referred to as target data), if the subset of target data is determined to be the subset of the adjusted probability filter, then the mapping value of the target data is determined based on the adjusted probability filter. Based on the mapping value, it is determined whether the target data exists in the data filtering result of the adjusted probability filter. The target data is then stored in an updated duplicate database or an updated result database based on this data filtering result. The false alarm rate of the probability filter is then re-determined based on the duplicate database and the result database. If the false alarm rate does not exceed the allowable error range of the probability filter, the result database can be directly applied. If the false alarm rate exceeds the allowable error range of the probability filter, the probability filter is adjusted further, and the adjusted probability filter is used to determine the data filtering result of subsequent data until the false alarm rate meets the allowable error range.

[0100] It should be noted that in application, the adjusted probability filter can be used to reprocess each piece of data to be deduplicated in the data subset. The false alarm rate of the adjusted probability filter can be determined based on the newly determined result database and duplicate database. If the false alarm rate does not exceed the allowable error range of the probability filter, the result database can be applied directly. If the false alarm rate exceeds the allowable error range of the probability filter, the probability filter should be adjusted again, and the data subset should be processed again using the adjusted probability filter until the false alarm rate meets the allowable error range.

[0101] It should be noted that the above is only an illustrative example. In actual applications, the appropriate method can be selected to optimize and adjust the probability filter according to the application scenario. No further restrictions are imposed here.

[0102] The aforementioned data deduplication method determines the false alarm rate of the probability filter based on the result database and duplicate database corresponding to the data subset. If the false alarm rate exceeds the allowable error range of the probability filter, the probability filter is adjusted, and the adjusted probability filter is used to determine the data filtering result of subsequent data until the false alarm rate meets the allowable error range. This implementation enters a feedback learning mechanism, records and analyzes the false alarm rate of the probability filter, and optimizes and adjusts the probability filter using the false alarm rate and the allowable error range. This reduces the false alarm rate of the probability filter, improves the data deduplication accuracy of the probability filter, and enables the system to continuously optimize its performance during continuous operation, thereby reducing the long-term false positive rate.

[0103] In one embodiment, step S604, adjusting the probability filter of the data subset, may include adjusting the filtering parameters of the probability filter for the data subset. The filtering parameters include at least one of the number of hash functions and the length of the bit array.

[0104] For example, if the false positive rate exceeds the allowable error range of the probability filter, the number of hash functions can be increased or decreased, the bit array length can be increased or decreased, or the number of hash functions can be increased or decreased simultaneously with the bit array length. That is, operations can be performed on at least one of the number of hash functions and the bit array length. It is understood that when the number of hash functions changes, the number of hash functions can be increased or decreased accordingly. The hash functions to be increased or decreased can be determined based on factors such as the data subset, the false positive rate, and the allowable false positive range, and are not limited here.

[0105] When the filtering parameters include the number of hash functions and the length of the bit array, the trends of change for the number of hash functions and the length of the bit array are the same. For example, if the false positive rate exceeds the allowable error range of the probabilistic filter, the number of hash functions and the length of the bit array are increased respectively. Alternatively, if the false positive rate exceeds the allowable error range of the probabilistic filter, the number of hash functions and the length of the bit array are decreased respectively.

[0106] The above data deduplication method, when the false alarm rate exceeds the allowable error range of the probability filter, adjusts the number of hash functions and the length of the bit array of the probability filter simultaneously to ensure that the false alarm rate of the probability filter does not exceed the allowable error range, thereby reducing the false judgment rate and improving the data deduplication accuracy.

[0107] In one embodiment, as shown in FIG7, a data deduplication method is provided, which includes the following steps S702 to S724.

[0108] S702: Obtain the data features of multiple data sets to be deduplicated. The data features include the data source or timestamp.

[0109] S704: Based on the data characteristics of each piece of data to be deduplicated, determine the local sensitivity hash value of each piece of data to be deduplicated using the local sensitivity hash function.

[0110] S706: Based on the local sensitive hash value of each piece of data to be deduplicated, divide each piece of data to be deduplicated into at least one subset of data.

[0111] S708: Initialize the number of hash functions, bit array size, and allowable error range for the Bloom filter of each data subset.

[0112] S710: Determine the hash value of each deduplicated data in the data subset based on the Bloom filter corresponding to each data subset.

[0113] S712: Based on the hash value of each piece of data to be deduplicated in each data subset, determine whether each piece of data to be deduplicated exists in the corresponding Bloom filter. If not, proceed to step S714; if yes, proceed to step S716.

[0114] S714: Store the data to be deduplicated in the results database.

[0115] S716: Store the data to be deduplicated in the duplicate database.

[0116] S718: Determine the false alarm rate of the Bloom filter based on the results database and the duplicate database.

[0117] S720: Determine whether the false alarm rate of the Bloom filter exceeds the allowable error range. If yes, proceed to step S722; otherwise, proceed to step S724.

[0118] S722: Dynamically increase or decrease at least one of the number of Bloom filter hash functions or the length of the bit array by small increments, and continue data collection until the allowable error range is met.

[0119] S724: Record the results database, duplicate database, number of hash functions for Bloom filters, bit array length, false alarm rate, and allowable false alarm range. After multiple data collections, once the false alarm rate stabilizes, statistical analysis of the error can be gradually reduced.

[0120] The aforementioned data deduplication method introduces Locality Sensitive Hashing (LSH) technology, dividing the data to be deduplicated into subsets according to certain rules (such as data source, timestamp, etc.). This enables more refined differentiation of similar data. Furthermore, an independently designed Bloom filter is applied to each subset, reducing false positives and significantly improving the accuracy of data deduplication while maintaining query efficiency. In addition, a feedback learning mechanism is introduced. By recording and analyzing the false positive rate, the choice of hash function and bit array layout are optimized and adjusted based on the false positive rate. This allows the system to continuously optimize its performance during operation, reducing the long-term false positive rate and improving the accuracy and efficiency of data deduplication.

[0121] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0122] Based on the same inventive concept, this application also provides a data deduplication apparatus for implementing the data deduplication method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data deduplication apparatus embodiments provided below can be found in the limitations of the data deduplication method described above, and will not be repeated here.

[0123] In one embodiment, as shown in FIG8, a data deduplication device 800 is provided, including: a partitioning module 801, a mapping module 802, a filtering module 803, and a deduplication module 804.

[0124] The partitioning module 801 divides multiple sets of data to be deduplicated into at least one subset; each subset corresponds to an independent probability filter. The mapping module 802, for each subset, determines the mapping value for each set of data to be deduplicated based on the probability filter corresponding to that subset. The filtering module 803, based on the mapping values ​​of each set of data to be deduplicated, determines the data filtering result for each set of data to be deduplicated; the data filtering result indicates whether the data to be deduplicated exists in the probability filter. The deduplication module 804 performs deduplication processing on the subset based on the data filtering results for each set of data to be deduplicated.

[0125] In one embodiment, the partitioning module 801 is further configured to obtain the data characteristics of each piece of data to be deduplicated, determine the local sensitive hash value of each piece of data to be deduplicated based on the data characteristics of each piece of data to be deduplicated using a local sensitive hash function, and divide each piece of data to be deduplicated into at least one data subset based on the local sensitive hash value of each piece of data to be deduplicated.

[0126] In one embodiment, the mapping module 802 is further configured to determine the hash function corresponding to the data subset based on the Bloom filter corresponding to the data subset, and determine the hash value of each deduplicated data in the data subset according to the hash function.

[0127] In one embodiment, the filtering module 803 is further configured to traverse the mapping values ​​of each deduplicated data in the data subset at the corresponding positions in the bit array, and for any deduplicated data in the data subset, if there is a 0 in the values ​​corresponding to the deduplicated data obtained through traversal, determine that the data filtering result of the deduplicated data does not exist in the probability filter, and if there is no 0 in the values ​​corresponding to the deduplicated data obtained through traversal, determine that the data filtering result of the deduplicated data exists in the probability filter.

[0128] In one embodiment, the deduplication module 804 is further configured to, for any one piece of data to be deduplicated in the data subset, store the data to be deduplicated in the duplicate database corresponding to the data subset if the data filtering result of the data to be deduplicated is that it exists in the probability filter, and store the data to be deduplicated in the result database corresponding to the data subset if the data filtering result of the data to be deduplicated is that it does not exist in the probability filter.

[0129] In one embodiment, the data deduplication device 800 further includes an adjustment module, which is used to determine the false alarm rate of the probability filter based on the result database and the duplicate database corresponding to the data subset, and to adjust the probability filter if the false alarm rate exceeds the allowable error range of the probability filter, and use the adjusted probability filter to determine the data filtering result of the subsequent data to be deduplicated, until the false alarm rate meets the allowable error range.

[0130] In one embodiment, the adjustment module is further configured to adjust the filtering parameters of the probability filter for the data subset, the filtering parameters including at least one of the number of hash functions and the length of the bit array. Wherein, when the filtering parameters include the number of hash functions and the length of the bit array, the number of hash functions and the length of the bit array exhibit the same trend of change.

[0131] The aforementioned data deduplication device 800 divides multiple data sets to be deduplicated into at least one data subset using a partitioning module 801. A mapping module 802 determines the mapping value of each data set to be deduplicated within each subset based on the corresponding probability filter. A filtering module 803 determines the data filtering result for each data set based on its mapping value. Finally, a deduplication module 804 performs deduplication processing on the data subsets based on the data filtering results. This data deduplication device 800, by dividing the data set to be deduplicated into at least one data subset, achieves a more refined distinction between similar data. Furthermore, it applies an independent probability filter to each data subset; that is, it adaptively and specifically designs an independent probability filter for each data subset. This avoids using a uniform probability filter for deduplication of different data with significant differences, reducing the probability filter's misjudgment of duplicate data. This not only improves deduplication efficiency but also significantly enhances the accuracy of data deduplication.

[0132] Each module in the aforementioned data deduplication device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0133] In one embodiment, a computer device, which may be a server, is provided, and its internal structure is shown in Figure 9. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data such as data to be deduplicated, a result database, and a deduplication database. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data deduplication method.

[0134] Those skilled in the art will understand that the structure shown in Figure 9 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0135] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the aforementioned data deduplication method.

[0136] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned data deduplication method.

[0137] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the aforementioned data deduplication method.

[0138] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data deduplication logic devices, etc., and are not limited to these.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A data deduplication method, characterized in that, The method includes: Divide multiple sets of data to be deduplicated into at least one subset; each subset corresponds to an independent probability filter. For each data subset, the mapping value of each deduplicated data in the data subset is determined according to the probability filter corresponding to the data subset; Based on the mapping values ​​of each deduplicated data in the data subset, the data filtering results for each deduplicated data in the data subset are determined accordingly; the data filtering results are used to indicate whether the deduplicated data exists in the probability filter; and Based on the data filtering results of each deduplicated data in the data subset, the data subset is deduplicated.

2. The method according to claim 1, characterized in that, The step of dividing multiple sets of data to be deduplicated into at least one subset includes: Obtain the data features of each of the data to be deduplicated; Based on the data characteristics of each of the deduplicated data, the local sensitivity hash value of each of the deduplicated data is determined using a locality-sensitive hash function; and Based on the local sensitive hash value of each of the deduplicated data, each of the deduplicated data is divided into at least one data subset.

3. The method according to claim 1 or 2, characterized in that, The probabilistic filter includes a Bloom filter, and the mapping value includes a hash value. The step of determining the mapping value of each deduplicated data in the data subset based on the probabilistic filter corresponding to the data subset includes: Based on the Bloom filter corresponding to the data subset, determine the hash function corresponding to the data subset; and The hash value of each piece of data to be deduplicated in the data subset is determined according to the hash function.

4. The method according to claim 1 or 2, characterized in that, The probability filter includes a bit array. The step of determining the data filtering result for each piece of data to be deduplicated in the data subset based on the mapping value of each piece of data to be deduplicated in the data subset includes: Iterate through the corresponding positions of the mapping values ​​of each deduplicated data in the data subset within the data set; and For any duplicate data to be deduplicated in the data subset, if there is a 0 among the values ​​corresponding to the duplicate data to be deduplicated obtained through the traversal, it is determined that the data filtering result of the duplicate data to be deduplicated is not present in the probability filter; if there is no 0 among the values ​​corresponding to the duplicate data to be deduplicated obtained through the traversal, it is determined that the data filtering result of the duplicate data to be deduplicated is present in the probability filter.

5. The method according to claim 1 or 2, characterized in that, The step of performing deduplication processing on the data subset based on the data filtering results of each deduplicated data in the data subset includes: For any one piece of data to be deduplicated in the data subset, if the data filtering result of the data to be deduplicated indicates that it exists in the probability filter, the data to be deduplicated is stored in the duplicate database corresponding to the data subset; and If the data filtering result of the data to be deduplicated is that it does not exist in the probability filter, the data to be deduplicated is stored in the result database corresponding to the data subset.

6. The method according to claim 5, characterized in that, The method further includes: Based on the result database and duplicate database corresponding to the data subset, the false alarm rate of the probability filter is determined; and If the false alarm rate exceeds the allowable error range of the probability filter, the probability filter is adjusted, and the adjusted probability filter is used to determine the data filtering result of the subsequent deduplicated data until the false alarm rate meets the allowable error range.

7. The method according to claim 6, characterized in that, The adjustment of the probability filter for the data subset includes: Adjust the filtering parameters of the probability filter for the data subset, wherein the filtering parameters include at least one of the number of hash functions and the length of the bit array; Wherein, when the filtering parameters include the number of hash functions and the length of the bit array, the changing trends of the number of hash functions and the length of the bit array are the same.

8. A data deduplication device, characterized in that, The device includes: The partitioning module is used to divide multiple data sets to be deduplicated into at least one subset; each subset corresponds to an independent probability filter. The mapping module is used to determine the mapping value of each deduplicated data in the data subset according to the probability filter corresponding to the data subset. The filtering module is used to determine the data filtering result for each piece of data to be deduplicated in the data subset based on the mapping value of each piece of data to be deduplicated in the data subset; the data filtering result is used to indicate whether the data to be deduplicated exists in the probability filter; and The deduplication module is used to perform deduplication processing on the data subset based on the data filtering results of each deduplicated data in the data subset.

9. The apparatus according to claim 8, characterized in that, The partitioning module is further configured to acquire the data characteristics of each of the deduplicated data to be deduplicated; and determine the local sensitive hash value of each of the deduplicated data to be deduplicated based on the local sensitive hash function according to the data characteristics of each of the deduplicated data to be deduplicated. And based on the local sensitive hash value of each of the deduplicated data, each of the deduplicated data is divided into at least one data subset.

10. The apparatus according to claim 8 or 9, characterized in that, The probability filter includes a Bloom filter, and the mapping value includes a hash value; The mapping module is further configured to determine the hash function corresponding to the data subset based on the Bloom filter corresponding to the data subset; and to determine the hash value of each piece of data to be deduplicated in the data subset based on the hash function.

11. The apparatus according to claim 8 or 9, characterized in that, The probability filter includes a bit array; The filtering module is further configured to iterate through the mapping values ​​of each piece of data to be deduplicated in the data subset and the corresponding values ​​in the bit array; and for any piece of data to be deduplicated in the data subset, if there is a 0 among the values ​​corresponding to the deduplicated data obtained by the iteration, determine that the data filtering result of the deduplicated data does not exist in the probability filter. If none of the values ​​corresponding to the data to be deduplicated obtained through iteration are 0, it is determined that the data filtering result of the data to be deduplicated exists in the probability filter.

12. The apparatus according to claim 8 or 9, characterized in that, The deduplication module is further configured to, for any one piece of data to be deduplicated in the data subset, store the data to be deduplicated in the duplicate database corresponding to the data subset if the data filtering result of the data to be deduplicated is that it exists in the probability filter; and store the data to be deduplicated in the result database corresponding to the data subset if the data filtering result of the data to be deduplicated is that it does not exist in the probability filter.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images