Method and apparatus for verifying consistency of synchronized data, and computing device cluster

Abstract

A method for verifying the consistency of synchronized data. A source database comprises a first table, a target database comprises a second table, and data in the second table is obtained by means of performing data synchronization on data in the first table. The method comprises: acquiring a plurality of first data files; acquiring a plurality of second data files; and when a hash value in a first data file is the same as a hash value in a second data file having the same identifier, determining that some data in a first table corresponding to the hash value in the first data file is the same as some data in a second table corresponding to the hash value in the second data file having the same identifier. The method in the present application requires no data sorting mode, thereby reducing the consumption of various resources and saving on time. In the method, when data consistency verification is performed, data comparison can be performed according to the granularity of data files, the number of data files corresponding to each table is flexibly set, and the memory occupancy of a data comparison process is effectively controlled, thereby preventing excessive memory occupancy and avoiding the problem of OOM.

Description

A method, apparatus, and computing device cluster for synchronizing data consistency verification

[0001] This application claims priority to Chinese Patent Application No. 202510062544.9, filed on January 14, 2025, entitled "A Method, Apparatus and Computing Device Cluster for Synchronizing Data Consistency Verification", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to the field of data synchronization technology, and in particular to a method, apparatus, and computing device cluster for verifying the consistency of synchronized data. Background Technology

[0003] In database synchronization, the synchronization of tables without primary keys and data consistency verification are two crucial steps. Data consistency is essential for ensuring the accuracy and reliability of data synchronization. However, data consistency verification of tables without primary keys is indeed a challenge faced by the industry. The main difficulty lies in the lack of unique identifiers for records in tables without primary keys, leading to the potential for duplicate data across multiple rows, and these tables typically contain large amounts of data. Directly comparing data at the table level consumes significant memory resources, putting pressure on existing systems. Furthermore, resuming the comparison from the breakpoint if an interruption occurs during synchronization is also a technical challenge, as there is no clear starting point to continue the unfinished synchronization task. Therefore, special strategies and techniques are needed to effectively handle the synchronization and consistency verification of tables without primary keys to ensure the integrity and continuity of data synchronization. Summary of the Invention

[0004] This application provides a method for verifying the consistency of synchronized data. This method reduces resource consumption and saves time during the consistency verification process, and avoids OutOfMemoryError (OOM) issues. Furthermore, this application also provides a device for verifying the consistency of synchronized data and a cluster of computing devices corresponding to this method.

[0005] Therefore, the following technical solutions are adopted in the embodiments of this application:

[0006] In a first aspect, this application provides a method for verifying the consistency of synchronized data. The source database includes a first table, and the target database includes a second table. The data in the second table is obtained by synchronizing the data in the first table. The method includes: acquiring multiple first data files; each first data file has an identifier, and each first data file includes hash values ​​of some data in the first table, with each hash value in the first data file associated with the identifier of the corresponding data file; acquiring multiple second data files; each second data file has an identifier, and each second data file includes hash values ​​of some data in the second table, with each hash value in the second data file associated with the identifier of the corresponding data file, and the identifiers of the multiple second data files are the same as the identifiers of the multiple first data files; when the hash values ​​in the first data files with the same identifier are the same as the hash values ​​in the second data files, it is determined that the partial data in the first table corresponding to the hash values ​​in the first data files with the same identifier are the same as the partial data in the second table corresponding to the hash values ​​in the second data files.

[0007] In this implementation, when partitioning tables in the source and target databases that require data consistency verification, the method converts each piece of data in the table into a hash value. After the hash values ​​are transformed according to set rules, they can be allocated to multiple data files with the same identifier as the transformed data, thus distributing data from a single table across multiple data files. This method eliminates the need for data sorting, reducing resource consumption and saving time.

[0008] Furthermore, this method allows for data consistency checks by comparing data files at a granular level. It allows for flexible setting of the number of data files for each table, effectively controlling memory usage during the data comparison process and preventing excessive memory consumption and OutOfMemoryError (OOM) issues.

[0009] In one implementation, before acquiring multiple first data files, the method further includes: splitting the hash values ​​of each data in the first table to obtain at least one array of each data in the first table; performing integer conversion on the at least one array of each data in the first table to obtain at least one integer type value of each data in the first table; performing an XOR operation on the at least one integer type value of each data in the first table to obtain a single value of each data in the first table; performing a modulo operation on the single value of each data in the first table to obtain a conversion result of each data in the first table; and assigning the hash values ​​of each data in the first table to a first data file with the same identifier as the conversion result of each data.

[0010] In this embodiment, the method can transform the hash value of each data to reduce a relatively large number to a relatively small number that is within the sequence number range of the data file, so that each data in the table can be allocated to different data files according to the transformation result, thereby achieving partitioning of the data in the table.

[0011] In one implementation, before acquiring multiple first data files, the method further includes: concatenating each row of data in the first table to obtain a string of each row of data in the first table; and performing a hash calculation on the string of each row of data in the first table to obtain a hash value corresponding to each data in the first table.

[0012] In this implementation, when partitioning a table, the method converts each row of data into a hash value and writes the corresponding hash value to a data file for subsequent data comparison. Compared to related technologies, since data sorting requires significant resource consumption and time, this method reduces resource consumption and saves sorting time by using hash values ​​instead of traditional data sorting.

[0013] In one implementation, before acquiring multiple first data files, the method further includes: writing the hash values ​​corresponding to each piece of data in the first table to multiple data files through at least one write thread.

[0014] In this embodiment, when writing data, the method can split the write thread that performs the write operation into multiple write threads, allowing different write threads to write data to different data files concurrently, thereby improving the data writing speed and reducing the total time required for data comparison.

[0015] In one implementation, the method further includes: creating a first Merkle tree and a second Merkle tree; the leaf nodes of the first Merkle tree store data from a first data file, and the nodes of the first Merkle tree other than the leaf nodes store hash values ​​obtained by hashing the data stored in the two next-level nodes connected to the first node; the leaf nodes of the second Merkle tree store data from a second data file, and the nodes of the second Merkle tree other than the leaf nodes store hash values ​​obtained by hashing the data stored in the two next-level nodes connected to the second node; the identifier of the first data file stored in the first Merkle tree is the same as the identifier of the second data file stored in the second Merkle tree; when the hash value stored in the root node of the first Merkle tree is the same as the hash value stored in the root node of the second Merkle tree, it is determined that the data in the first data file stored in the first Merkle tree is the same as the data in the second data file stored in the second Merkle tree.

[0016] In one embodiment, the method further includes: when the hash value stored in the root node of the first Merkle tree is different from the hash value stored in the root node of the second Merkle tree, comparing the hash values ​​stored in the nodes of the next level of the first Merkle tree with the same identifier with the hash values ​​stored in the nodes of the next level of the second Merkle tree, determining the target leaf node in the first Merkle tree and the second Merkle tree, wherein the data stored in the target leaf node in the first Merkle tree is different from the data stored in the target leaf node in the second Merkle tree; and modifying the row data stored in the target leaf node in the second Merkle tree to the row data stored in the target leaf node in the first Merkle tree.

[0017] Secondly, this application provides a data consistency verification device, comprising: a first processing unit for acquiring multiple first data files; a source database including a first table, each first data file having an identifier, each first data file including a hash value of a portion of data in the first table, and the hash value in each first data file being associated with the identifier of the corresponding data file; a second processing unit for acquiring multiple second data files; a target database including a second table, the data in the second table being obtained by synchronizing data from the first table, each second data file having an identifier, each second data file including a hash value of a portion of data in the second table, the hash value in each second data file being associated with the identifier of the corresponding data file, and the identifiers of the multiple second data files being the same as the identifiers of the multiple first data files; and a third processing unit for determining that, when the hash value in the first data file with the same identifier is the same as the hash value in the second data file, the portion of data in the first table corresponding to the hash value in the first data file with the same identifier is the same as the portion of data in the second table corresponding to the hash value in the second data file.

[0018] In one embodiment, before acquiring multiple first data files, the first processing unit is further configured to: split the hash values ​​of each data in the first table to obtain at least one array of each data in the first table; perform integer conversion on the at least one array of each data in the first table to obtain at least one integer type value of each data in the first table; perform an XOR operation on the at least one integer type value of each data in the first table to obtain a single value of each data in the first table; perform a modulo operation on the single value of each data in the first table to obtain a conversion result of each data in the first table; and assign the hash values ​​of each data in the first table to a first data file with the same identifier as the conversion result of each data.

[0019] In one embodiment, before acquiring multiple first data files, the first processing unit is further configured to concatenate each row of data in the first table to obtain a string of each row of data in the first table; and perform hash calculation on the string of each row of data in the first table to obtain the hash value corresponding to each data in the first table.

[0020] In one embodiment, before acquiring multiple first data files, the first processing unit is further configured to write the hash values ​​corresponding to each piece of data in the first table to the multiple data files respectively through at least one write thread.

[0021] In one embodiment, the third processing unit is further configured to create a first Merkle tree and a second Merkle tree; the leaf nodes of the first Merkle tree store data from a first data file, and the nodes of the first Merkle tree other than the leaf nodes store hash values ​​obtained by hashing the data stored in the two next-level nodes connected to the first node; the leaf nodes of the second Merkle tree store data from a second data file, and the nodes of the second Merkle tree other than the leaf nodes store hash values ​​obtained by hashing the data stored in the two next-level nodes connected to the second node; the identifier of the first data file stored in the first Merkle tree is the same as the identifier of the second data file stored in the second Merkle tree; when the hash value stored in the root node of the first Merkle tree is the same as the hash value stored in the root node of the second Merkle tree, it is determined that the data in the first data file stored in the first Merkle tree is the same as the data in the second data file stored in the second Merkle tree.

[0022] In one embodiment, the third processing unit is further configured to, when the hash value stored in the root node of the first Merkle tree is different from the hash value stored in the root node of the second Merkle tree, compare the hash values ​​stored in the nodes of the next level of the first Merkle tree with the same identifier with those stored in the nodes of the next level of the second Merkle tree, determine the target leaf node in the first Merkle tree and the second Merkle tree, and determine that the data stored in the target leaf node in the first Merkle tree is different from the data stored in the target leaf node in the second Merkle tree; and modify the row data stored in the target leaf node in the second Merkle tree to the row data stored in the target leaf node in the first Merkle tree.

[0023] Thirdly, embodiments of this application provide a computing device, including: at least one memory; and at least one processor, the processor being configured to execute instructions stored in the memory to cause the computing device to perform the various possible implementations of the first aspect.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium including computer program instructions, which, when executed by a computing device, perform the various possible implementations of the first aspect.

[0025] Fifthly, this application provides a computer program product containing instructions, characterized in that the computer program product stores instructions that, when executed by a computing device, cause the computing device to implement various possible implementations of the first aspect.

[0026] In a sixth aspect, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the various possible implementations of the first aspect.

[0027] In a seventh aspect, embodiments of this application provide a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the various possible implementations of the first aspect.

[0028] Eighthly, this application provides a computer program product containing instructions, characterized in that the computer program product stores instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to implement various possible implementations of the first aspect. Attached Figure Description

[0029] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.

[0030] Figure 1 is a schematic diagram of the structure of a data synchronization system provided in an embodiment of this application;

[0031] Figure 2 is a schematic diagram of the process of converting the hash value of each data according to the set rules provided in the embodiment of this application;

[0032] Figure 3(a) is a schematic diagram of the process of writing data from a table in the source database to different data files according to an embodiment of this application;

[0033] Figure 3(b) is a schematic diagram of the process of writing data from a table in the target database to different data files according to an embodiment of this application;

[0034] Figure 4 is a schematic diagram of the process of comparing data in source files and target files with the same sequence number using Merkle trees in an embodiment of this application;

[0035] Figure 5 is a schematic diagram of a user using the data synchronization system provided in the embodiments of this application;

[0036] Figure 6 shows a flowchart of a method for verifying the consistency of synchronized data provided in an embodiment of this application;

[0037] Figure 7 is a schematic diagram of the structure of a consistency verification device for synchronized data provided in an embodiment of this application;

[0038] Figure 8 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0039] Figure 9 is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of this application;

[0040] Figure 10 is a schematic diagram of another computing device cluster architecture provided in the embodiments of this application. Detailed Implementation

[0041] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0042] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0043] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0044] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0045] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0046] Before introducing the technical solution protected by this application, several technical terms involved in the technical solution protected by this application will be explained in advance, namely:

[0047] A table is a basic data structure in a database used to store related data. A table consists of rows and columns; each row represents a data record, and each column represents a field or attribute of that record. Tables are used to organize and store structured data, making data retrieval and management more efficient. During data synchronization, the table, as the basic unit of data storage, carries the data that needs to be synchronized. The purpose of data synchronization is to replicate and maintain the consistency of table data between two or more databases. Whether it's full data synchronization or incremental data synchronization, it's necessary to accurately identify and transfer the data in the table to ensure data consistency and integrity across different databases.

[0048] A primary key is one or more fields in a database table whose values ​​uniquely identify each record in the table. Primary key values ​​must be unique and cannot be NULL. Once created, the primary key value cannot be changed. The main purpose of a primary key is to provide a unique record identifier, facilitating fast data retrieval, referencing, and maintaining data integrity. In data synchronization, primary keys are used to match corresponding records in the source and target databases, ensuring the accuracy and consistency of data synchronization.

[0049] A table without a primary key is a table in a database that lacks a primary key. A primary key is a field or combination of fields used to uniquely identify each record in a table. Because a table without a primary key lacks this unique identifier, the records in the table cannot be uniquely identified by a clear standard. Compared to tables with primary keys, tables without primary keys are more prone to data duplication or loss during data synchronization due to the lack of a unique identifier, which affects data integrity and consistency. Therefore, performing data consistency verification after data synchronization is a necessary step to ensure that the synchronized data remains consistent.

[0050] Next, the technical solution provided in this application will be introduced.

[0051] Generally, when synchronizing data from a source database to a target database, the changed data in the source database can be read, and the tables containing the changed data in the source database can be stored in the target database, thus achieving data synchronization between the two databases. If the created table is a table without a primary key, the data stored in this table may encounter problems such as duplicate storage and partial data loss due to the lack of a primary key as a unique identifier for each record. This situation illustrates that during data synchronization, the primary key cannot be relied upon to ensure the uniqueness and consistency of each row of data, thereby increasing the risk of data loss and incorrect data updates. To solve these problems, additional data consistency verification measures may be required.

[0052] When performing data consistency checks using related technologies, the first step is to sort the data in the table by the primary key using the ORDER BY clause. The uniqueness and sequentiality of the primary key assist in subsequent sharding and comparison. Next, the LIMIT clause limits the amount of data returned in each query, thus achieving data sharding. For example, LIMIT 0,1000 retrieves the first 1000 sorted records, LIMIT 1000,2000 retrieves the next 1000 records, and so on. A query is executed on each shard to obtain a sorted subset of data. Then, the data in each shard in the source database is compared with the corresponding shard in the target database to check for discrepancies. During the comparison, any inconsistencies are recorded, and appropriate data synchronization or correction measures are taken. This process is repeated until all shards of the entire table have been processed, ensuring the completeness and accuracy of the data consistency check.

[0053] However, during data consistency verification, the relevant technologies require a full table sort, which can place a significant load on the database and lead to lengthy processing times. Furthermore, for tables without primary keys, effective data sharding is not possible; comparing data on a table-by-table basis may cause out-of-memory (OOM) errors.

[0054] In view of this, embodiments of this application provide a data synchronization system. When partitioning tables in a source database and a target database requiring data consistency verification, this system can convert each piece of data in the table into a hash value. After the hash values ​​of each piece of data are transformed according to set rules, they can be allocated to multiple data files with the same identifier as the transformed data, thus distributing the data from one table to multiple data files. This data synchronization system reduces resource consumption and saves time by using hash conversion instead of data sorting. Furthermore, the partitioning method used by this data synchronization system is applicable not only to tables with primary keys but also to tables without primary keys.

[0055] Furthermore, when performing data consistency checks, this system can compare data at the granularity of data files. That is, the system compares the hash values ​​in the source database's data files with the hash values ​​in the target database's data files for the same identifier. The system allows for flexible configuration of the number of data files for each table, effectively controlling the memory usage of the data comparison process, thereby preventing excessive memory consumption and avoiding OutOfMemoryError (OOM) issues.

[0056] Figure 1 is a schematic diagram of a data synchronization system provided in an embodiment of this application. As shown in Figure 1, the data synchronization system 100 may include a source database 110, a target database 120, and a verification device 130.

[0057] Source database 110 refers to the database from which data is read for synchronization; it is the starting point of the data synchronization process, i.e., the database containing the original data. During synchronization, data changes (such as additions, modifications, and deletions) in source database 110 are captured and prepared to be sent to target database 120. Source database 110 can be any type of database system, such as relational databases (e.g., MySQL, PostgreSQL, Oracle, etc.) or non-relational databases (e.g., MongoDB, Cassandra, etc.).

[0058] Target database 120 refers to the database that receives data copied from source database 110. It is the endpoint of the data synchronization process, i.e., the database to which the data is synchronized. Target database 120 is used to store the data synchronized from the source database and can be used for purposes such as data analysis, backup, disaster recovery, load balancing, or business expansion. Similar to source database 110, target database 120 can be any type of database system and can be the same as or different from source database 110.

[0059] The verification device 130 compares the data read from the source database 110 with the data received from the target database 120 to determine whether the data stored in the two databases as tables without primary keys is the same, thereby achieving the function of data consistency verification. In this embodiment, the verification device 130 can be divided into a query module 131, a generation module 132, and a comparison module 133 according to its execution function. The specific functions of each module are as follows:

[0060] The query module 131 is used to perform a full table query on the source database 110 to obtain the target data in the source database 110, and to perform a full table query on the target database 120 to obtain the target data in the target database 120. The target data in the source database 110 refers to the data sent from the source database 110 to the target database 120, generally referring to data from one or more tables. The target data in the target database 120 refers to the data in the target database 120 that originates from the source database 110, generally referring to data from one or more tables.

[0061] After receiving a query command from the user, the query module 131 can trigger a full table search of the tables stored in the source database 110 based on the query command. The query command typically carries an identifier for the table the user needs to query, such as the table name. Based on the table identifier carried in the query command, the query module 131 can retrieve all the data stored in the corresponding table when the table is found. In this application, in a data synchronization scenario, the query module 131 can retrieve target data from tables stored in the source database 110 and from tables stored in the target database 120 by executing Structured Query Language (SQL), such as `SELECT col_1, col_2 FROM schema.table`.

[0062] The query module 131 can split the query thread executing the query task into two independent query threads: one for querying target data in the source database 110, and the other for querying target data in the target database 120. This design allows the query module 131 to perform data queries simultaneously, achieving concurrent execution, thereby improving query efficiency and reducing the total time required for data comparison. Optionally, the query module 131 can be split into 2N independent query threads, where N is a positive integer greater than 1. The query module 131 can use N query threads to simultaneously query data from different tables in the source database 110, achieving concurrent queries on the source database 110, thereby further improving query efficiency. The query module 131 can also use another N query threads to simultaneously query data from different tables in the target database 120, achieving concurrent queries on the target database 120, thereby further improving query efficiency.

[0063] After completing the query operation, the query module 131 can obtain target data from the source database 110 and the target database 120 respectively, and write the two target data to the local disk to prepare data for the data comparison process, ensuring that the data for the comparison operation is up-to-date and complete.

[0064] The generation module 132 is used to convert each piece of data in each table of the source database 110 into a hash value, and after converting the hash value of each piece of data according to the set rules, it can be assigned to multiple data files with the same identifier as the conversion result of each piece of data. It also converts each piece of data in each table of the target database 120 into a hash value, and after converting the hash value of each piece of data according to the set rules, it can be assigned to multiple data files with the same identifier as the conversion result of each piece of data.

[0065] During the data initialization phase, generation module 132 can calculate the number of data files (N, a positive integer greater than 1) needed for each table, based on the data volume of a single table and the data volume stored in a single data file. Based on this standard, after obtaining the target data from source database 110, generation module 132 can create a corresponding number of data files for the target data in source database 110, based on the number of tables in the target data. Similarly, after obtaining the target data from target database 120, generation module 132 can create a corresponding number of data files for the target data in target database 120, based on the number of tables in the target data.

[0066] Furthermore, when the generation module 132 creates data files, it can set the number of data files corresponding to tables with the same identifier in the source database 110 and the target database 120 to be the same, and the sequence numbers of the data files to be the same. By specifying this configuration, after the generation module 132 stores the data from the source database 110 and the target database 120 separately in different data files, it can compare data from tables with the same identifier in the source database 110 and the target database 120 by selecting two data files with the same sequence number.

[0067] As shown in Figure 2, during the process of storing data from the table into a data file, the generation module 132 can first retrieve row data from the table in a row-by-row manner, and concatenate each row of data into a string so that all values ​​in a row of data can be hashed into a single hash value. Then, the generation module 132 can perform a hash calculation on each string to obtain the hash value of each string. Optionally, the generation module 132 can divide the data in the table according to other schemes, such as retrieving column data from the table in a column-by-column manner.

[0068] In determining which data file each row of data should be written to, the generation module 132 can transform the hash value of each row of data according to a set rule to obtain the transformation result of each row of data, and then allocate each row of data to the corresponding data file according to the transformation result. For example, referring to Figure 2, the process of the generation module 132 executing the set rule is as follows:

[0069] The hash value obtained by the generation module 132 is a 64-bit (or other number of bits) hash value, which is 8 bytes. The generation module 132 can first split the 64-bit hash value into two 32-bit (4-byte) arrays, ensuring the hash value's bit length meets the requirements for generating integer (INT) type values. Then, the generation module 132 can convert the two arrays into two INT type values. Next, the generation module 132 can perform an exclusive OR (XOR) operation on the two INT type values ​​to obtain one INT type value, and then take the absolute value of this INT type value to obtain the number (NUMBER) for each row of data. Finally, after calculating the number for each row of data, the generation module 132 can perform a modulo operation, i.e., calculate NUMBER % N, to determine the conversion result (RESULT) for each row of data, thus reducing a relatively large number to a smaller number within the range of N.

[0070] Each RESULT indicates which data file number the data line should be stored in. After determining the data file number to which the data belongs, the generation module 132 can serialize or compress the string, corresponding hash value, and other information of each data line and write it to the data file with the corresponding number, thereby realizing the storage of each data line in the data file with the corresponding number.

[0071] When writing data, the generation module 132 can split the write operation into multiple write threads, allowing different write threads to concurrently write data to different data files. This improves the data writing speed and reduces the total time required for data comparison. If multiple write threads write data to the same data file simultaneously, the generation module 132 needs to set up a locking mechanism to ensure that multiple write threads write data to the same data file sequentially, preventing errors during data writing.

[0072] For example, as shown in Figure 3(a), the generation module 132 retrieves five rows of data from a table in the source database 110, with row numbers Row_1, Row_2, Row_3, Row_4, and Row_5. Assume that there are three data files corresponding to this table, with file numbers source file 1, source file 2, and source file 3.

[0073] After the processing shown in Figure 2, the generation module 132 can record the data with serial number Row_1 into source file 1, record the data with serial numbers Row_2, Row_4 and Row_5 into source file 2, and record the data with serial number Row_3 into source file 3.

[0074] As shown in Figure 3(b), the generation module 132 retrieves five rows of data from the corresponding table in the target database 120, with row numbers Row_1, Row_2', Row_3, Row_4, and Row_5. Assume that there are three data files corresponding to this table, with file numbers Target File 1, Target File 2, and Target File 3.

[0075] After the processing shown in Figure 2, the generation module 132 can record the data with serial numbers Row_1 and Row_2' into target file 1, record the data with serial numbers Row_4 and Row_5 into target file 2, and record the data with serial number Row_3 into target file 3.

[0076] In this embodiment, when the generation module 132 partitions a table, it converts each row of data into a hash value and writes the hash value corresponding to each row of data into a data file for subsequent data comparison using the hash value. Compared with related technologies, since data sorting requires a large amount of resources and time, this data synchronization system 100 reduces resource consumption and saves sorting time by using hash value conversion instead of data sorting. Furthermore, the data synchronization system 100's use of hash value for data comparison is applicable not only to data comparison in tables without primary keys but also to data comparison in tables with primary keys.

[0077] The comparison module 133 is used to compare the hash value in the data file of the same source database 110 with the hash value in the data file of the target database 120 to determine whether the data synchronized by the target database 120 is consistent with the data in the source database 110.

[0078] After the comparison module 133 writes all the data from tables with the same identifier in the source database 110 and the target database 120 to the corresponding data files, it can initiate data comparison. The comparison module 133 performs data comparison at the data file level. For example, the comparison module 133 can compare the data file with sequence number 1 in the source database 110 with the data file with sequence number 1 in the target database 120.

[0079] When comparing data, the comparison module 133 can read the hash values ​​from data files with the same sequence number in both the source database 110 and the target database 120 into local memory. Then, it uses strategies such as row-by-row comparison or Merkle tree comparison to compare the hash values ​​in the two data files. If there are hash values ​​with different comparison results, the comparison module 133 can determine the row data corresponding to the two hash values. The comparison module 133 can then query and correct the two rows of data to ensure that the rows belonging to the target database 120 are identical to the rows belonging to the source database 110. Optionally, the querying and correction of the two rows of data can be handled by staff.

[0080] Taking the Merkle tree comparison method as an example, when comparison module 133 compares data files with the same index from source database 110 and target database 120 using the Merkle tree comparison method, it can create two Merkle trees: one Merkle tree corresponding to the data file from source database 110, and the other Merkle tree corresponding to the data file from target database 120. The leaf nodes at the bottom level of each Merkle tree can store one or more rows of data. The hash values ​​of the row data stored in the leaf nodes with the same index in both Merkle trees are identical.

[0081] Each node in the second-to-last level of each Merkle tree stores a hash value, which is calculated by hashing the rows stored in the two leaf nodes connected to it. Similarly, each node in the third-to-last level of each Merkle tree stores a hash value, calculated by hashing the rows stored in the two leaf nodes connected to it. This process continues in the same manner.

[0082] When comparing two Merkle trees, the comparison module 133 first compares the hash values ​​of the root nodes of the two Merkle trees. If the comparison results are the same, it indicates that the data files with the same sequence number from the source database 110 and the data files from the target database 120 are completely identical.

[0083] If the comparison results are different, it indicates that the data files with the same sequence number from the source database 110 and the data files from the target database 120 are different. At this time, the comparison module 133 compares the hash values ​​of the nodes with the same sequence number at the second level of the two Merkle trees, checks for nodes with different comparison results, and then compares the hash values ​​of the nodes connected to that node at the third level, and so on, until the leaf nodes at the bottom level are compared.

[0084] After the comparison module 133 finds leaf nodes with the same sequence number from two Merkle trees, it can obtain the row data stored in the two leaf nodes and query the row data stored in the two leaf nodes to make the row data belonging to the target database 120 the same as the row data belonging to the source database 110.

[0085] As shown in Figure 4, when the comparison module 133 compares the data file with sequence number 1 corresponding to the source database 110 and the data file with sequence number 1 corresponding to the target database 120 using the Merkle tree comparison method, it creates two Merkle trees with the same structure. Since the source file 1 records data with sequence number Row_1 and the target file 1 records data with sequence numbers Row_1 and Row_2', the comparison module 133 finds that the root nodes, leaf nodes storing the row data "Row_2'", and the nodes in the middle of the two Merkle trees of the source file 1 and the target file 1 are all different. This allows the module to accurately identify that the row data "Row_2" in the source file 1 and the row data "Row_2'" in the target file 1 are different.

[0086] When comparing the data in the source database 110 (sequence number 2) with the data in the target database 120 (sequence number 2) using the Merkle tree comparison method, the comparison module 133 creates two Merkle trees with the same structure. Since source file 2 records data with sequence numbers Row_2, Row_4, and Row_5, and target file 2 records data with sequence numbers Row_4 and Row_5, the comparison module 133 finds that the root nodes, leaf nodes storing the row data "Row_2", and intermediate nodes of the two Merkle trees in source file 2 and target file 2 are all different. This allows the module to accurately identify the difference between the row data "Row_2" in source file 2 and the row data "Row_2'" in target file 2.

[0087] When comparison module 133 compares the data file with sequence number 3 in source database 110 and the data file with sequence number 3 in target database 120 using the Merkle tree comparison method, it creates two Merkle trees with the same structure. During the comparison, since the root nodes of the two Merkle trees in source file 1 and target file 1 are the same, it indicates that the data recorded in source file 3 is identical to the data recorded in target file 3.

[0088] In this embodiment, the comparison module 130 performs data comparison at the granularity of data files; that is, it compares the hash values ​​in the data files of the source database with the hash values ​​in the data files of the target database for data with the same identifier. The comparison module 130 can effectively control the memory usage of the data comparison process by flexibly setting the number of data files corresponding to each table. This approach helps prevent excessive memory usage, thereby avoiding OutOfMemoryError (OOM) issues.

[0089] It should be understood that the functional modules and devices involved in the aforementioned data synchronization system 100 can all be implemented through software or hardware, depending on the actual situation, and are not limited here. Furthermore, the functional modules and devices involved in the aforementioned data synchronization system 100 can be arranged individually or integrated, and are not limited here.

[0090] The above is a description of the data synchronization system 100 provided in this application embodiment. It is understood that the data synchronization system 100 can be configured on a cloud management platform, for example, deployed on at least one virtual machine or container instance, so that the cloud management platform can provide data synchronization services. Of course, the data synchronization system 100 can also be configured on nodes other than the cloud management platform, for example, deployed in at least one data center or on at least one server, depending on the actual situation, and is not limited here. The cloud management platform can provide pages related to public cloud services for users to remotely access public cloud services. In this embodiment, users can pre-purchase the data synchronization services provided by the data synchronization system 100 on the cloud management platform. For ease of understanding, the interaction between the user and the cloud management platform is described below.

[0091] As shown in Figure 5, the interaction between the user and the cloud management platform mainly includes: the user logs into the cloud management platform 500 through a web page on the client side, selects and purchases cloud services (i.e., data synchronization services) related to the data synchronization system 100 on the cloud management platform 500. After purchase, the user can generate the data synchronization system 100 on the cloud management platform 500 based on the functions provided by the data synchronization service. The cloud management platform 500 is primarily used to manage the infrastructure for running the data synchronization service. For example, the infrastructure for the data synchronization service may include multiple data centers located in different regions, each data center including multiple servers. The data centers can provide basic resources for the data synchronization service, such as computing resources and storage resources. Therefore, when purchasing and using the data synchronization service, the user mainly pays for the resources used. When using the data synchronization service, the user can input their requirements for the data synchronization service through the configuration interface, application program interface (API), or user interaction interface provided by the cloud management platform 500. The cloud management platform 500 can then generate a data synchronization service matching the user's (or other software / hardware, etc.) input requirements.

[0092] Alternatively, some modules in the data synchronization system 100 can be configured on the cloud side and others on the edge side, thereby achieving data synchronization services through edge-cloud collaboration. Furthermore, the data synchronization system 100 can also be entirely configured on the edge side, depending on the specific circumstances; no limitation is made here.

[0093] The above is an introduction to the data synchronization system 100 provided in the embodiments of this application. Next, based on the above content, the simulation method provided in the embodiments of this application will be introduced.

[0094] For example, Figure 6 shows a flowchart of a data synchronization consistency verification method provided in an embodiment of this application. It can be understood that this method can be executed by the verification device 130 in the data synchronization system 100 described above, and the specific implementation process is as follows:

[0095] Step S601: Obtain multiple first data files. The source database includes a first table, each first data file has an identifier, and each first data file includes hash values ​​of a portion of the data in the first table. The hash values ​​in each first data file are associated with the identifier of the corresponding data file.

[0096] Step S602: Obtain multiple second data files. The target database includes a second table, the data in which is obtained by synchronizing data from the first table. Each second data file has an identifier and includes hash values ​​of some data from the second table. The hash values ​​in each second data file are associated with the identifier of the corresponding data file. The identifiers of multiple second data files are the same as the identifiers of multiple first data files.

[0097] Specifically, after receiving a query command input by the user, the verification device 130 can trigger a full table query of the tables stored in the source database 110 based on the query command. When the corresponding table is found based on the table identifier carried in the query command, the verification device 130 can retrieve all the data stored in that table. After completing the query operation, the query module 131 can retrieve target data from both the source database 110 and the target database 120, and write the two sets of target data to the local disk to prepare data for the data comparison process, ensuring that the data for the comparison operation is up-to-date and complete.

[0098] Optionally, the verification device 130 can split the query thread executing the query task into two independent query threads: one for querying target data in the source database 110, and the other for querying target data in the target database 120. This design allows the verification device 130 to perform data queries simultaneously, achieving concurrent execution, thereby improving query efficiency and reducing the total time required for data comparison. The verification device 130 can be split into 2N independent query threads, where N is a positive integer greater than 1. The verification device 130 can use N query threads to simultaneously query data in different tables in the source database 110, achieving concurrent queries on the source database 110, thereby further improving query efficiency. The verification device 130 can also use another N query threads to simultaneously query data in different tables in the target database 120, achieving concurrent queries on the target database 120, thereby further improving query efficiency.

[0099] When the verification device 130 creates data files, it can set the number of data files corresponding to tables with the same identifier in the source database 110 and the target database 120 to be the same, and the serial numbers of the data files to be the same. By specifying this configuration, after the verification device 130 stores the data from the source database 110 and the target database 120 separately in different data files, it can compare data from tables with the same identifier in the source database 110 and the target database 120 by selecting two data files with the same serial number.

[0100] During the process of storing table data in a data file, the verification device 130 can first retrieve row data from the table row by row, and concatenate each row data into a string so that all values ​​in a row can be hashed into a single hash value. Then, the verification device 130 can perform a hash calculation on each string to obtain the hash value for each string. Optionally, the verification device 130 can divide the data in the table according to other schemes, such as retrieving column data from the table column by column.

[0101] In determining which data file each row of data should be written to, the verification device 130 can convert the hash value of each row of data according to a set rule to obtain the conversion result of each row of data, and then allocate each row of data to the corresponding data file according to the conversion result. For example, referring to Figure 2, the process of the verification device 130 executing the set rule is as follows:

[0102] The hash value obtained by the verification device 130 is a 64-bit (or other number of bits) hash value, i.e., an 8-byte hash value. The verification device 130 can first split the 64-bit hash value into two 32-bit (4-byte) arrays, ensuring the hash value's bit length meets the requirements for generating an INT type value. Then, the verification device 130 can convert the two arrays into two INT type values. Next, the verification device 130 can perform an XOR operation on the two INT type values ​​to obtain one INT type value, and then take the absolute value of this INT type value to obtain the number corresponding to each row of data. Finally, after calculating the number for each row of data, the verification device 130 can perform a modulo operation to determine the conversion result for each row of data, reducing larger numbers to smaller numbers within the range of N. Each RESULT indicates which data file sequence number the row of data should be stored in. After determining the data file number to which the data belongs, the verification device 130 can serialize or compress the string, corresponding hash value and other information of each line of data and write it into the data file with the corresponding number, thereby realizing the storage of each line of data in the data file with the corresponding number.

[0103] For example, when writing data, the verification device 130 can split the write thread performing the write operation into multiple write threads, allowing different write threads to concurrently write data to different data files, thereby improving the data writing speed and reducing the total time required for data comparison. If multiple write threads write data to the same data file simultaneously, the verification device 130 needs to set up a locking mechanism to ensure that multiple write threads write data to the same data file sequentially, preventing errors in the written data.

[0104] Step S603: When the hash value in the first data file with the same identifier is the same as the hash value in the second data file, determine that the data in the first table corresponding to the hash value in the first data file with the same identifier is the same as the data in the second table corresponding to the hash value in the second data file.

[0105] Specifically, after the verification device 130 writes all the data from tables with the same identifier in both the source database 110 and the target database 120 to their respective data files, it can initiate a data comparison. The verification device 130 performs the data comparison at the data file level. During the comparison, the verification device 130 reads the hash values ​​from the data files with the same sequence number in both the source database 110 and the target database 120 into its local memory, and then uses strategies such as row-by-row comparison and Merkle tree comparison to compare the hash values ​​in the two data files. If there are different hash values, the verification device 130 can determine the row data corresponding to the two hash values. The verification device 130 can then query and correct the two rows of data to ensure that the row data belonging to the target database 120 is identical to the row data belonging to the source database 110. Optionally, the query and correction of the two rows of data can be handled by a staff member.

[0106] In this embodiment, when partitioning tables in the source and target databases requiring data consistency verification, the method can convert each piece of data in the table into a hash value. After the hash values ​​of each piece of data are transformed according to set rules, they can be allocated to multiple data files with the same identifier as the transformation results of each piece of data, thus distributing the data from one table to multiple data files. This method does not require data sorting, reducing resource consumption and saving time.

[0107] Furthermore, this method allows for data consistency checks by comparing data files at a granular level. It allows for flexible setting of the number of data files for each table, effectively controlling memory usage during the data comparison process and preventing excessive memory consumption and OutOfMemoryError (OOM) issues.

[0108] Based on the above description, this application provides a data consistency verification device 700 for synchronizing data. This device connects a source database and a target database; wherein the source database is used to synchronize locally stored data to the target database. As shown in Figure 7, the device 700 includes:

[0109] The first processing unit 710 is used to acquire multiple first data files; the source database includes a first table, each first data file has an identifier, each first data file includes a hash value of a portion of the data in the first table, and the hash value in each first data file is associated with the identifier of the corresponding data file; the second processing unit 720 is used to acquire multiple second data files; the target database includes a second table, the data in the second table is obtained by synchronizing the data in the first table, each second data file has an identifier, each second data file includes a hash value of a portion of the data in the second table, the hash value in each second data file is associated with the identifier of the corresponding data file, and the identifiers of multiple second data files are the same as the identifiers of multiple first data files; the third processing unit 730 is used to determine that when the hash value in the first data file with the same identifier is the same as the hash value in the second data file, the portion of the data in the first table corresponding to the hash value in the first data file with the same identifier is the same as the portion of the data in the second table corresponding to the hash value in the second data file.

[0110] In one embodiment, before acquiring multiple first data files, the first processing unit 710 is further configured to split the hash values ​​of each data in the first table to obtain at least one array of each data in the first table; perform integer conversion on the at least one array of each data in the first table to obtain at least one integer type value of each data in the first table; perform an XOR operation on the at least one integer type value of each data in the first table to obtain a value of each data in the first table; perform a modulo operation on the value of each data in the first table to obtain the conversion result of each data in the first table; and assign the hash values ​​of each data in the first table to a first data file with the same identifier as the conversion result of each data.

[0111] In one embodiment, before acquiring multiple first data files, the first processing unit 710 is further configured to concatenate each row of data in the first table to obtain a string of each row of data in the first table; and perform hash calculation on the string of each row of data in the first table to obtain the hash value corresponding to each data in the first table.

[0112] In one embodiment, before acquiring multiple first data files, the first processing unit 710 is further configured to write the hash values ​​corresponding to each piece of data in the first table to the multiple data files respectively through at least one write thread.

[0113] In one embodiment, the third processing unit 730 is further configured to create a first Merkle tree and a second Merkle tree; the leaf nodes of the first Merkle tree store data from a first data file, and the nodes of the first Merkle tree other than the leaf nodes store hash values ​​obtained by hashing the data stored in the two next-level nodes connected to the first node; the leaf nodes of the second Merkle tree store data from a second data file, and the nodes of the second Merkle tree other than the leaf nodes store hash values ​​obtained by hashing the data stored in the two next-level nodes connected to the second node; the identifier of the first data file stored in the first Merkle tree is the same as the identifier of the second data file stored in the second Merkle tree; when the hash value stored in the root node of the first Merkle tree is the same as the hash value stored in the root node of the second Merkle tree, it is determined that the data in the first data file stored in the first Merkle tree is the same as the data in the second data file stored in the second Merkle tree.

[0114] In one embodiment, the third processing unit 730 is further configured to, when the hash value stored in the root node of the first Merkle tree is different from the hash value stored in the root node of the second Merkle tree, compare the hash values ​​stored in the nodes of the next level of the first Merkle tree with the same identifier with the hash values ​​stored in the nodes of the next level of the second Merkle tree, determine the target leaf node in the first Merkle tree and the second Merkle tree, and the data stored in the target leaf node in the first Merkle tree is different from the data stored in the target leaf node in the second Merkle tree; and modify the row data stored in the target leaf node in the second Merkle tree to the row data stored in the target leaf node in the first Merkle tree.

[0115] The first processing module 710, the second processing module 720, and the third processing module 730 can all be implemented in software or in hardware. For example, the implementation of the first processing module 710 will be described below. Similarly, the implementation of the second processing module 720 and the third processing module 730 can refer to the implementation of the first processing module 710.

[0116] As an example of a software functional unit, the first processing module 710 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the first processing module 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0117] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0118] As an example of a hardware functional unit, the first processing module 710 may include at least one computing device, such as a server. Alternatively, the first processing module 710 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0119] The multiple computing devices included in the first processing module 710 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the first processing module 710 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the first processing module 710 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0120] It should be noted that, in other embodiments, the first processing module 710 can be used to execute any step in the method shown in FIG. 6, the second processing module 720 can be used to execute any step in the method shown in FIG. 6, and the third processing module 730 can be used to execute any step in the method shown in FIG. 6. The steps implemented by the first processing module 710, the second processing module 720, and the third processing module 730 can be specified as needed. The first processing module 710, the second processing module 720, and the third processing module 730 respectively implement different steps in the method shown in FIG. 6 to realize all the functions of the device 700.

[0121] Figure 8 is a schematic diagram of a computing device provided in an embodiment of this application. As shown in Figure 8, the computing device 800 includes a bus 810, a processor 820, a memory 830, and a communication interface 840. The processor 820, the memory 830, and the communication interface 840 communicate with each other via the bus 810. The computing device 800 can be a server, computer, laptop, server rack, etc. It should be understood that this application does not limit the number of processors and memories in the computing device 800.

[0122] Bus 810 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 8, but this does not imply that there is only one bus or one type of bus. Bus 810 can include pathways for transmitting information between various components of computing device 800 (e.g., processor 820, memory 830, communication interface 840).

[0123] The processor 820 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0124] The memory 830 may include volatile memory, such as random access memory (RAM). The memory 830 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0125] The memory 830 stores executable program code, which the processor 820 executes to implement the functions of the aforementioned multiple modules, such as the first processing module 710, the second processing module 720, and the third processing module 730, thereby realizing the method shown in FIG6. That is, the memory 830 stores instructions for executing the method shown in FIG6.

[0126] Alternatively, the memory 830 stores executable code, which the processor 820 executes to implement the functions of the aforementioned modules, thereby achieving the method shown in FIG6. That is, the memory 830 stores instructions for executing the method shown in FIG6.

[0127] The communication interface 840 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 800 and other devices or communication networks.

[0128] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0129] As shown in Figure 9, the computing device cluster includes at least one computing device 800. The memory 830 of one or more computing devices 800 in the computing device cluster may store the same instructions for executing the method shown in Figure 6.

[0130] In some possible implementations, the memory 830 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing the method shown in FIG. 6. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for executing the method shown in FIG. 6.

[0131] It should be noted that the memories 830 in different computing devices 800 within the computing device cluster can store different instructions, which are used to execute parts of the functions of the first processing module 710, the second processing module 720, and the third processing module 730 described above. That is, the instructions stored in the memories 830 of different computing devices 800 can implement the functions of one or more of the first processing module 710, the second processing module 720, and the third processing module 730 described above.

[0132] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 illustrates one possible implementation. As shown in Figure 10, two computing devices, computing device 800A and computing device 800B, are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 830 in computing device 800A stores instructions for executing some of the functions of the first processing module 710. Simultaneously, the memory 830 in computing device 800B stores instructions for executing other parts of the functions of the second processing module 720 and the third processing module 730.

[0133] The connection method between the computing device clusters shown in Figure 10 can be considered in light of the fact that the method provided in this application as shown in Figure 6 requires a large amount of data storage. Therefore, it is considered that the functions implemented by another part of the modules in the first processing module 710, the second processing module 720 and the third processing module 730 are handed over to the computing device 800B for execution.

[0134] It should be understood that the functions of computing device 800A shown in Figure 10 can also be performed by multiple computing devices 800. Similarly, the functions of computing device 800B can also be performed by multiple computing devices 800.

[0135] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster described in Figures 8 and 9. The difference is that the memory 830 of one or more computing devices 800 in this computing device cluster can store the same instructions for executing the method shown in Figure 6.

[0136] In some possible implementations, the memory 830 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing the method shown in FIG. 6. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for executing the method shown in FIG. 6.

[0137] It should be noted that the memory 830 in different computing devices 800 within the computing device cluster can store different instructions to execute some functions of the computing device 800. That is, the instructions stored in the memory 830 of different computing devices 800 can implement the functions of one or more of the first processing module 710, the second processing module 720, and the third processing module 730 mentioned above.

[0138] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the method shown in FIG6.

[0139] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the method shown in FIG6.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for verifying the consistency of synchronized data, characterized in that, The source database includes a first table, and the target database includes a second table. The data in the second table is obtained by synchronizing the data in the first table. The method includes: Obtain multiple first data files; each first data file has an identifier, and each first data file includes the hash value of a portion of the data in the first table, wherein the hash value in each first data file is associated with the identifier of the corresponding data file; Obtain multiple second data files; each second data file has an identifier, each second data file includes a hash value of a portion of the data in the second table, the hash value in each second data file is associated with the identifier of the corresponding data file, and the identifiers of the multiple second data files are the same as the identifiers of the multiple first data files; When the hash value in the first data file with the same identifier is the same as the hash value in the second data file, it is determined that the data in the first table corresponding to the hash value in the first data file with the same identifier is the same as the data in the second table corresponding to the hash value in the second data file.

2. The method according to claim 1, characterized in that, Before acquiring the multiple first data files, the method further includes: The hash values ​​of each data in the first table are split to obtain at least one array of each data in the first table; Convert at least one array of data in the first table to an integer to obtain at least one integer value for each data in the first table; Perform an XOR operation on at least one integer value of each data in the first table to obtain a value of each data in the first table; Perform a modulo operation on one of the values ​​in the first table to obtain the conversion result of each data in the first table; The hash values ​​of each data in the first table are assigned to a first data file with the same identifier as the transformation result of each data.

3. The method according to claim 1 or 2, characterized in that, Before acquiring the multiple first data files, the method further includes: Concatenate the data from each row in the first table to obtain the string of each row in the first table; Perform a hash calculation on the string of each row of data in the first table to obtain the hash value corresponding to each data in the first table.

4. The method according to any one of claims 1-3, characterized in that, Before acquiring the multiple first data files, the method further includes: The hash values ​​corresponding to each piece of data in the first table are written to the plurality of data files by at least one write thread.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Create a first Merkle tree and a second Merkle tree; the leaf nodes of the first Merkle tree store data from a first data file, and the nodes of the first Merkle tree other than the leaf nodes store the hash value obtained by hashing the data stored in the two nodes connected to the first node at the next level; the leaf nodes of the second Merkle tree store data from a second data file, and the nodes of the second Merkle tree other than the leaf nodes store the hash value obtained by hashing the data stored in the two nodes connected to the first node at the next level; the identifier of the first data file stored in the first Merkle tree is the same as the identifier of the second data file stored in the second Merkle tree. When the hash value stored in the root node of the first Merkle tree is the same as the hash value stored in the root node of the second Merkle tree, it is determined that the data of the first data file stored in the first Merkle tree is the same as the data of the second data file stored in the second Merkle tree.

6. The method according to claim 5, characterized in that, The method further includes: When the hash value stored in the root node of the first Merkle tree is different from the hash value stored in the root node of the second Merkle tree, the hash values ​​stored in the nodes of the next level of the first Merkle tree with the same identifier are compared with those stored in the nodes of the next level of the second Merkle tree to determine the target leaf node in the first Merkle tree and the second Merkle tree. The data stored in the target leaf node in the first Merkle tree is different from the data stored in the target leaf node in the second Merkle tree. Modify the row data stored in the target leaf node of the second Merkle tree to the row data stored in the target leaf node of the first Merkle tree.

7. A device for verifying the consistency of synchronized data, characterized in that, include: The first processing unit is used to acquire multiple first data files; The source database includes a first table, each first data file has an identifier, each first data file includes a hash value of a portion of the data in the first table, and the hash value in each first data file is associated with the identifier of the corresponding data file; The second processing unit is used to acquire multiple second data files; The target database includes a second table, the data in the second table is obtained by synchronizing the data in the first table, each second data file has an identifier, each second data file includes the hash value of a portion of the data in the second table, the hash value in each second data file is associated with the identifier of the corresponding data file, and the identifiers of the multiple second data files are the same as the identifiers of the multiple first data files; The third processing unit is configured to determine, when the hash value in the first data file with the same identifier is the same as the hash value in the second data file, that the partial data in the first table corresponding to the hash value in the first data file with the same identifier is the same as the partial data in the second table corresponding to the hash value in the second data file.

8. The apparatus according to claim 7, characterized in that, The first processing unit, before acquiring the plurality of first data files, is further configured to... The hash values ​​of each data in the first table are split to obtain at least one array of each data in the first table; Convert at least one array of data in the first table to an integer to obtain at least one integer value for each data in the first table; Perform an XOR operation on at least one integer value of each data in the first table to obtain a value of each data in the first table; Perform a modulo operation on one of the values ​​in the first table to obtain the conversion result of each data in the first table; The hash values ​​of each data in the first table are assigned to a first data file with the same identifier as the transformation result of each data.

9. The apparatus according to claim 7 or 9, characterized in that, The first processing unit, before acquiring the plurality of first data files, is further configured to... Concatenate the data from each row in the first table to obtain the string of each row in the first table; Perform a hash calculation on the string of each row of data in the first table to obtain the hash value corresponding to each data in the first table.

10. The apparatus according to any one of claims 7-9, characterized in that, The first processing unit, before acquiring the plurality of first data files, is further configured to... The hash values ​​corresponding to each piece of data in the first table are written to the plurality of data files by at least one write thread.

11. The apparatus according to any one of claims 7-11, characterized in that, The third processing unit is also used for Create a first Merkle tree and a second Merkle tree; the leaf nodes of the first Merkle tree store data from a first data file, and the nodes of the first Merkle tree other than the leaf nodes store the hash value obtained by hashing the data stored in the two nodes connected to the first node at the next level; the leaf nodes of the second Merkle tree store data from a second data file, and the nodes of the second Merkle tree other than the leaf nodes store the hash value obtained by hashing the data stored in the two nodes connected to the first node at the next level; the identifier of the first data file stored in the first Merkle tree is the same as the identifier of the second data file stored in the second Merkle tree. When the hash value stored in the root node of the first Merkle tree is the same as the hash value stored in the root node of the second Merkle tree, it is determined that the data of the first data file stored in the first Merkle tree is the same as the data of the second data file stored in the second Merkle tree.

12. The apparatus according to claim 11, characterized in that, The third processing unit is also used for When the hash value stored in the root node of the first Merkle tree is different from the hash value stored in the root node of the second Merkle tree, the hash values ​​stored in the nodes of the next level of the first Merkle tree with the same identifier are compared with those stored in the nodes of the next level of the second Merkle tree to determine the target leaf node in the first Merkle tree and the second Merkle tree. The data stored in the target leaf node in the first Merkle tree is different from the data stored in the target leaf node in the second Merkle tree. Modify the row data stored in the target leaf node of the second Merkle tree to the row data stored in the target leaf node of the first Merkle tree.

13. A computing device cluster, characterized in that, include: At least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-6.

15. A computer program product containing instructions, characterized in that, The computer program product stores instructions that, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1-6.

Images