Entity resolution parallel processing method and apparatus, device, and medium

Abstract

The present application is applicable to the technical field of big data mining, and in particular, relates to an entity resolution parallel processing method and apparatus, a device, and a medium. According to the method, entity tuples in a data set to be resolved are divided into disjoint partitions; for any partition, entity tuples in the partition are evenly divided into N intervals, M consecutive intervals are formed into a window, and each window is processed in series; for any window, a thread block is assigned to each interval in the window, the thread block is configured for respectively matching target tuples within a corresponding interval with the entity tuples in the partition other than the target tuple to obtain a matching result, and the thread blocks in the window are processed in parallel; when a thread block corresponding to any interval in the window completes execution, the executed thread block is used to process an interval in a window immediately next to the window until each interval is processed by the corresponding thread block so as to obtain complete matching results. The present invention effectively partitions entity data, optimizes resource utilization and parallelism, and improves entity resolution efficiency.

Description

Parallel processing methods, apparatus, equipment and media for solid digestion

[0001] This application is based on and claims priority to Chinese Invention Application No. 202411569289.9, filed on November 5, 2024, entitled “Physical Dissolution Parallel Processing Method, Apparatus, Device and Medium”. Technical Field

[0002] This application relates to the field of big data mining technology, and in particular to a parallel processing method, apparatus, equipment and medium for entity decomposition. Background Technology

[0003] Entity resolution refers to identifying all data in a given dataset that points to the same real-world entity. Entity resolution methods can be mainly divided into two categories: rule-based methods and deep learning model-based methods. In comparison, rule-based entity resolution methods have the advantages of high efficiency and easy interpretation. This method filters out mismatches in the dataset through a series of rules, thereby identifying potential matching entities. Existing large-scale rule-based entity resolution systems rely on multiple central processing units (CPUs) in a shared-nothing architecture to drive the machines. They use large-scale computing clusters to integrate the resources of multiple computers to complete the entity resolution task. The specific steps are as follows: (1) In a shared-nothing architecture, the data is partitioned and distributed across multiple processing units. Each processing unit uses local memory to clean the data independently; (2) Data that originally belonged to the same entity may be assigned to different processing units. In order to avoid missing these matches, different processing units need to transfer local data to remote processing units through a computer network; (3) After all processing units have completed the entity resolution task, the entity resolution results on each machine are collected.

[0004] However, this method has certain limitations. For example, (1) these systems are usually expensive to maintain and build, making large-scale graph computing a privilege of a few companies that are capable of maintaining large-scale computer clusters; (2) these distributed computing engines are usually based on the assumption that "the more machines used, the shorter the time it takes for the entire cluster system to complete the computing task." However, this assumption is not always true. More computing nodes mean greater communication costs between potential nodes, which means that for some scenarios, no matter how the computing cluster is horizontally scaled, the system performance cannot be significantly improved; (3) in order to avoid missing a match, different processing units need to transmit local data to remote processing units through computer networks, which brings additional data transmission overhead.

[0005] Given the above issues, new hardware acceleration technologies and parallel processing strategies can be explored to improve entity resolution performance. Graphics Processing Units (GPUs), as processing units based on a shared memory architecture, possess efficient data sharing, inter-unit collaboration, and dynamic workload scheduling capabilities, providing a parallel processing solution for accelerating rule-based entity resolution. Therefore, how to effectively partition entity data to optimize resource utilization and improve entity resolution efficiency through parallelism has become an urgent problem to be solved. Summary of the Invention

[0006] In view of this, embodiments of this application provide a parallel processing method, apparatus, device, and medium for entity resolution to solve the problem of how to effectively segment entity data to optimize resource utilization and improve entity resolution efficiency through parallel processing.

[0007] In a first aspect, embodiments of this application provide a parallel processing method for entity resolution, the parallel processing method for entity resolution comprising:

[0008] Obtain the dataset to be resolved, and divide the entity tuples in the dataset into at least one disjoint partition, wherein the dataset to be resolved includes at least two entity tuples;

[0009] For any partition, all entity tuples within the partition are divided into N intervals. M consecutive intervals are formed into a window. All windows are processed sequentially. N is greater than M, and both N and M are positive integers greater than zero.

[0010] For any given window, a thread block is allocated to each interval within the window. The thread block is used to match the target tuple within the corresponding interval with the entity tuples within the partition excluding the target tuple to obtain the matching result. The thread blocks within the window are processed in parallel.

[0011] If the thread block corresponding to any interval within the window is completed, the completed thread block is used to process the interval in the next window of the window until all intervals are processed by the corresponding thread blocks to obtain all matching results. The complete matching results are used to guide the dataset to be resolved to remove duplicate entity tuples.

[0012] Secondly, embodiments of this application provide a parallel processing apparatus for entity resolution, the parallel processing apparatus for entity resolution comprising:

[0013] The entity partitioning module is used to obtain the dataset to be resolved and divide the entity tuples in the dataset to be resolved into at least one disjoint partition, wherein the dataset to be resolved includes at least two entity tuples.

[0014] The interval window module is used to divide all entity tuples in any partition into N intervals, form a window with M consecutive intervals, and process all windows sequentially. Here, N is greater than M, and N and M are both positive integers greater than zero.

[0015] The thread allocation module is used to allocate a thread block to each interval within any window. The thread block is used to match the target tuple in the corresponding interval with the entity tuples in the partition other than the target tuple to obtain the matching result. The thread blocks within the window are processed in parallel.

[0016] The thread loop execution module is used to process the interval in the next window of the window if the thread block corresponding to any interval in the window has been executed, until all intervals have been processed by the corresponding thread blocks and all matching results are obtained. The all matching results are used to guide the dataset to be resolved to remove duplicate entity tuples.

[0017] Thirdly, embodiments of this application provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the entity resolution parallel processing method as described in the first aspect.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the entity resolution parallel processing method as described in the first aspect.

[0019] The beneficial effects of this application embodiment compared with the prior art are as follows: This application divides the entity tuples in the dataset to be resolved into at least one non-overlapping partition. For any partition, all tuples in the partition are divided into N intervals. M consecutive intervals are formed into a window. All windows are processed sequentially. For any window, a thread block is assigned to each partition in the window. If the thread block corresponding to any interval in the window is completed, the completed thread block is used to process the interval in the next window of the window, until all intervals are processed by the corresponding thread blocks and all matching results are obtained.

[0020] By employing an asynchronous pipeline architecture, the tuples in the dataset to be resolved are divided into at least one disjoint partition. Each partition is further subdivided into multiple intervals, forming windowed processing. The intervals within each window are processed in parallel by independent thread blocks, which optimizes resource utilization, ensures balanced workload of each thread block, and improves parallel efficiency, thereby accelerating the execution of parallel entity resolution tasks. At the same time, serial processing between windows also ensures the order and consistency of data, allows for the continuous transmission of processing results, and ensures the accuracy of processing. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 is a schematic diagram of an application environment for a parallel processing method for entity resolution provided in Embodiment 1 of this application;

[0023] Figure 2 is a flowchart illustrating a parallel processing method for entity resolution provided in Embodiment 2 of this application;

[0024] Figure 3 is a schematic diagram of the partitioning intervals and window divisions provided in Embodiment 2 of this application;

[0025] Figure 4 is a flowchart illustrating a parallel processing method for entity resolution provided in Embodiment 3 of this application;

[0026] Figure 5 is a schematic diagram of the execution time of a thread block provided in Embodiment 3 of this application;

[0027] Figure 6 is a schematic diagram of the execution time of a thread block cross-interval task theft provided in Embodiment 3 of this application;

[0028] Figure 7 is a flowchart illustrating a parallel processing method for entity resolution provided in Embodiment 4 of this application;

[0029] Figure 8 is a schematic diagram of the execution time of task theft within a thread block interval provided in Embodiment 4 of this application;

[0030] Figure 9 is a flowchart illustrating a parallel processing method for entity resolution provided in Embodiment 5 of this application;

[0031] Figure 10 is a flowchart illustrating a parallel processing method for entity resolution provided in Embodiment Six of this application;

[0032] Figure 11 is a schematic diagram of a physical digestion parallel processing device provided in Embodiment 7 of this application;

[0033] Figure 12 is a schematic diagram of the structure of a computer device provided in Embodiment 8 of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] To illustrate the technical solution of this application, specific embodiments are described below.

[0036] The entity resolution parallel processing method provided in Embodiment 1 of this application can be applied in the application environment shown in Figure 1. The server and client communicate, with the server providing entity resolution parallel processing services and the client triggering entity resolution parallel processing tasks to the server. The client includes, but is not limited to, devices such as handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, and personal digital assistants (PDAs). The server-side computer device can be implemented using a dedicated server or a server cluster consisting of multiple servers.

[0037] Referring to Figure 2, which is a flowchart illustrating a parallel entity resolution processing method according to Embodiment 2 of this application, the above-described parallel entity resolution processing method is applied to the server in Figure 1. The server connects to the client to obtain the dataset to be resolved sent by the client. As shown in Figure 2, the parallel entity resolution processing method may include the following steps:

[0038] Step S201: Obtain the dataset to be resolved, and divide the entity tuples in the dataset into at least one disjoint partition.

[0039] In this embodiment, the dataset to be resolved can refer to the dataset to be resolved into entities. The dataset to be executed includes at least two tuples that record data information. Each tuple consists of one or more attributes and corresponding attribute values. Attributes can refer to variables or fields that describe the characteristics of the tuples. Attribute values ​​can refer to the specific numerical values ​​of the corresponding attributes. Partitions can refer to disjoint data partitions obtained by dividing the entity tuples in the dataset to be resolved.

[0040] For example, given a pattern R = (eid, A1, ..., A... n ), where Ai As an attribute, eid is the entity identifier (ID). Each tuple in R represents an entity. The dataset to be resolved is a set of tuples with an R schema.

[0041] Specifically, for any tuple in the dataset to be resolved, a hash key can be calculated based on the attributes of the tuple using a hash function, and tuples with the same hash key can be grouped into the same partition.

[0042] Step S202: For any partition, divide all entity tuples in the partition into N intervals, form a window with M consecutive intervals, and process all windows sequentially.

[0043] Step S203: For any window, allocate a thread block for each interval within the window.

[0044] Step S204: If the thread block corresponding to any interval within the window has finished executing, the executed thread block is used to process the interval in the next window of the window until all intervals have been processed by their corresponding thread blocks and all matching results are obtained.

[0045] In this embodiment, an interval can refer to a data interval obtained by dividing a partition, and a window can refer to a parallel sliding window (PSW). Here, N is greater than M, and both N and M are positive integers greater than zero. The thread block is used to match the target tuple in the corresponding interval with the entity tuples in the partition excluding the target tuple to obtain the matching result. The target tuple can refer to any tuple in the corresponding interval, and the matching result can refer to the result of entity resolution between the target tuple and the entity tuples in the partition excluding the target tuple. That is, if the target tuple matches the tuple in the partition excluding the target tuple, it is determined that the two tuples represent the same real entity. If they do not match, it is determined that the two tuples represent different real entities. All matching results are used to guide the dataset to be resolved to remove duplicate entity tuples, that is, to determine the pairs of tuples in the dataset to be resolved that represent the same real entity, and the pairs of tuples in the dataset to be resolved that represent different real entities.

[0046] Specifically, for any group, all entity tuples within the partition are divided into N intervals. A window of preset length is used, sliding across the N intervals with a preset step size. The interval included when the sliding stops is determined as the interval of the corresponding window. The window is used to process the N intervals sequentially. For any window, a thread block is allocated to each interval within the window, and the thread blocks within the window are processed in parallel. When all threads in any thread block within the window have finished executing, the thread block writes the processing result back to the host memory, and then continues to process the intervals in the next window, until the window slides to the end of the partition, and all intervals have been processed by the corresponding thread blocks to obtain all matching results.

[0047] For example, as shown in Figure 3, it is a schematic diagram of the partitioning interval and window division provided in Embodiment 2 of this application.

[0048] All entity tuples within partition P are divided into 9 equal intervals: interval 1, interval 2, interval 3, interval 4, interval 5, interval 6, interval 7, interval 8, and interval 9. The window size is 3, meaning that 3 consecutive intervals form one window. Thus, the 9 intervals are divided into 3 windows: Sliding window 1, Sliding window 2, and Sliding window 3. Sliding window 1 includes intervals 1, 2, and 3; Sliding window 2 includes intervals 4, 5, and 6; and Sliding window 3 includes intervals 7, 8, and 9. Sliding window 1, Sliding window 2, and Sliding window 3 are processed sequentially.

[0049] Each window's intervals are assigned to three thread blocks (TBs): TB1, TB2, and TB3. For example, in Sliding window 1, interval 1 is assigned to TB1, and each thread in TB1 is responsible for matching the tuples in interval 1 with all other tuples in partition P to obtain a matching result. Interval 2 is assigned to TB2, and each thread in TB2 is responsible for matching the tuples in interval 2 with all other tuples in partition P to obtain a matching result. Interval 3 is assigned to TB3, and each thread in TB3 is responsible for matching the tuples in interval 3 with all other tuples in partition P to obtain a matching result. TB1, TB2, and TB3 process interval 1, interval 2, and interval 3 in parallel.

[0050] If all threads within TB1 in Sliding window 1 have finished executing, TB1 writes the processing result back to host memory. Then, TB1 continues processing interval 4 in Sliding window 2. After all threads within TB1 in Sliding window 2 have finished executing, TB1 writes the processing result back to host memory. Then, TB1 continues processing interval 7 in Sliding window 3. Correspondingly, TB2 processes interval 2, interval 5, and interval 8 in sequence, and TB3 processes interval 3, interval 6, and interval 9 in sequence, until all intervals have been processed by their corresponding thread blocks and all matching results are obtained.

[0051] In this embodiment, by adopting an asynchronous pipeline architecture, the tuples in the dataset to be resolved are divided into at least one disjoint partition. Each partition is further subdivided into multiple intervals to form windowed processing. The intervals within each window are processed in parallel by independent thread blocks, which optimizes resource utilization, ensures the workload balance of each thread block, and improves parallel efficiency, thereby accelerating the execution of parallel entity resolution tasks. At the same time, the serial processing between windows also ensures the order and consistency of data, allows the continuous transmission of processing results, and ensures the accuracy of processing.

[0052] Referring to Figure 4, which is a flowchart illustrating a parallel processing method for entity resolution provided in Embodiment 3 of this application, as shown in Figure 4, after forming a window from M consecutive intervals in step S202 above, the following steps may also be included:

[0053] Step S401: Construct a bitmap.

[0054] Step S402: For any thread block, if the thread block is in an idle state, select an interval in the bitmap where the processing state is unprocessed.

[0055] Step S403: Use a thread block to process the unprocessed intervals to obtain the corresponding matching results.

[0056] In this embodiment, a bitmap is maintained in global memory. Each bit in the bitmap represents the processing status of an interval processed by the corresponding thread block. An idle thread block is one that has processed all intervals allocated to all windows and has completed the processing. For example, as shown in Figure 3, if TB1 processes all intervals 1, 4, and 7 allocated to all windows and has completed the processing, then TB1 is an idle thread block.

[0057] In the above embodiment two, for any window, a thread block is allocated to each interval within the window. For a partition, the number of intervals processed by each thread block is approximately equal. For example, in the partition shown in Figure 3, the three thread blocks are each allocated to three partitions: TB1 processes interval 1, interval 4, and interval 7; TB2 processes interval 2, interval 5, and interval 8; and TB3 processes interval 3, interval 6, and interval 9. However, since the GPU adopts a single instruction multi-threaded execution mode, when some threads have long execution times, other threads may be in an idle state. For example, Figure 5 shows a schematic diagram of the execution time of a thread block provided in embodiment three of this application. For the partitions shown in Figure 3, TB1 needs 10 time units to complete interval 1, interval 4 and interval 7, TB2 needs 4 time units to complete interval 2, interval 5 and interval 8, and TB3 needs 3 time units to complete interval 3, interval 6 and interval 9. The total time taken by TB1, TB2 and TB3 is inconsistent, resulting in TB2 having 6 time units in an idle state and TB3 having 7 time units in an idle state.

[0058] To further balance the workload, a bitmap can be maintained in global memory. For any thread block, when the thread block is executing, it first processes the intervals allocated to it by all windows. After processing the intervals allocated to all windows, if the thread block is in an idle state, it can check the bitmap, steal an interval from the bitmap that is not being processed by another thread block, and update the bitmap to avoid duplicate processing. The stolen unprocessed interval is then processed to obtain the corresponding matching result. This process can be called the cross-interval task stealing mechanism.

[0059] For example, Figure 6 shows a schematic diagram of the execution time of cross-interval task stealing for a thread block according to Embodiment 3 of this application. For the partition shown in Figure 3, TB3 completes all window-allocated intervals after 3 time units, and then steals interval 4 from TB1 by checking the bitmap. TB2 completes all window-allocated intervals after 4 time units, and then steals interval 7 from TB1 by checking the bitmap for processing. Through cross-interval task stealing, the total execution time is reduced from 10 time units as shown in Figure 5 to 7 time units.

[0060] In this embodiment, after distributing tasks roughly equally to threads through a window, a cross-interval task stealing mechanism is used during execution to adaptively adjust the tasks among different threads. This avoids situations where some thread blocks are idle after completing their tasks while other thread blocks remain busy, further achieving load balancing, improving the overall utilization of hardware resources, and reducing the waiting and idle time of thread blocks, thus reducing the overall execution time and improving the efficiency of parallel processing.

[0061] Referring to Figure 7, which is a flowchart illustrating a parallel processing method for entity resolution provided in Embodiment 4 of this application. As shown in Figure 7, before selecting an unprocessed interval from the bitmap in step S402 above, the following steps may also be included:

[0062] Step S701: Detect whether there is an area in the bitmap that is in an unprocessed state.

[0063] Step S702: If an unprocessed interval is detected in the bitmap, then select an unprocessed interval from the bitmap.

[0064] Specifically, for any thread block, if the thread block is in an idle state, it checks whether there is an unprocessed interval in the bitmap. If an unprocessed interval is detected in the bitmap, an unprocessed interval is selected from the bitmap, and the thread block is used to process the unprocessed interval to obtain the corresponding matching result.

[0065] Step S703: If no interval in the bitmap is detected to be in an unprocessed state, then the interval in the processing state is determined to be in execution.

[0066] Step S704: Notify the thread block corresponding to the executing interval to divide the remaining execution tasks corresponding to the executing interval into a preset number of subtasks.

[0067] Step S705: Leave one of the subtasks in the thread block corresponding to the execution interval for processing to obtain the corresponding matching result.

[0068] Step S706: Distribute each remaining subtask to an idle thread block for processing to obtain the corresponding matching result.

[0069] If no unprocessed interval is detected in the bitmap, the interval is determined to be in the process of execution. The number of idle thread blocks is obtained and sent as a notification to the thread block corresponding to the executing interval. The number of idle threads is incremented by 1 to obtain a preset number of subtasks. The remaining execution tasks corresponding to the executing interval are divided into the preset number of subtasks. One of the subtasks is left to be processed in the thread block corresponding to the executing interval to obtain the corresponding matching result. Each of the remaining subtasks is distributed to an idle thread block for processing to obtain the corresponding matching result. This process can be called the interval task stealing mechanism.

[0070] For example, Figure 8 shows a schematic diagram of the execution time of task stealing within a thread block interval according to Embodiment 4 of this application. For the partition shown in Figure 3, TB3 completes all window-allocated intervals after 3 time units and then performs cross-interval task stealing, stealing interval 4 from TB1 for processing. TB2 completes all window-allocated intervals after 4 time units and then performs cross-interval task stealing, stealing interval 7 from TB1 for processing. After TB3 completes processing the stolen interval 4, it checks that there are no intervals in the bitmap with an unprocessed state, thus determining that interval 7 is in the process of execution. At this time, the number of idle thread blocks is 1, i.e., TB3. The preset number of parts is 2. The remaining tasks corresponding to the executing interval 7 are divided into 2 subtasks. One subtask is processed by TB2, and the other subtask is processed by TB3. Through cross-interval task stealing and intra-interval task stealing, the total execution time is reduced from 10 time units as shown in Figure 5 to 6 time units.

[0071] In this embodiment, after distributing tasks roughly equally to threads through a window, during execution, a cross-interval task stealing mechanism and an intra-interval task stealing mechanism are used to adaptively and dynamically adjust the tasks among different threads. This avoids situations where some thread blocks are idle after completing their tasks, while other thread blocks are still busy. This further achieves load balancing, improves the overall utilization of hardware resources, and reduces the waiting time and idle time of thread blocks, thereby reducing the overall execution time and improving the efficiency of parallel processing.

[0072] Referring to Figure 9, which is a flowchart illustrating a parallel processing method for entity resolution according to Embodiment 5 of this application, as shown in Figure 9, dividing the entity tuples in the dataset to be resolved into at least one disjoint partition in step S201 may include the following steps:

[0073] Step S901: Obtain the execution plan of the dataset to be resolved, determine the S equal sign predicates in the execution plan that are not used repeatedly by at least two rules, and obtain the S hash functions based on the S equal sign predicates.

[0074] Step S902: For any entity tuple in the dataset to be resolved, call any one of the S hash functions to calculate the corresponding partition key.

[0075] Step S903: Traverse all entity tuples to obtain the partition keys of all entity tuples. Based on the partition keys, divide the entity tuples in the dataset to be resolved to obtain at least one disjoint partition.

[0076] In this embodiment, the dataset to be resolved can be defined as a set of tuples with pattern R (refer to step S201). Then, when performing entity resolution on the dataset based on rules, the rules in pattern R can be defined as follows: Here, X is the set of predicates for two tuples t and s, l denotes t.eid = s.eid, and X is the rule. The prerequisite, l, is the result, when tuples t and tuples s satisfy the rule The prerequisite is that tuple t and tuple s represent the same entity. (Rule) The predicate in the expression can be defined as p::=tA=c|tA=sB|tA≈sB, where A and B are attributes, c is a constant, tA=sB and tA=c are used to compare whether the A attribute value of tuple t is equal to the B attribute value of tuple s, and tA≈sB is used to compare the similarity between the A attribute value of tuple t and the B attribute value of tuple s. The similarity can be measured by various similarity measures, such as edit distance or Jaccard similarity.

[0077] The execution plan specifies the order of evaluation of rules and predicates in the rule set for entity resolution of the dataset to be resolved. The equality predicate can refer to the predicate that performs equality comparison, such as tA = sB and tA = c mentioned above. The partition key can refer to the hash value calculated by the hash function, which is used to determine which partition the tuple should be assigned to.

[0078] Specifically, an execution plan can be generated by the database query optimizer based on the rule set discovered offline. The evaluation order of rules and predicates in the execution rule set is then determined. After obtaining the execution plan, S non-reused equals predicates are identified. Based on the S equals predicates, S hash functions are obtained. For any tuple in the dataset to be resolved, an identifier (ID) can be assigned to the tuple to indicate the hash function used. Based on the ID of the tuple, the corresponding hash function is called from the S hash functions to calculate the corresponding partition key. All tuples in the dataset to be resolved are traversed to obtain the partition keys of all tuples. Based on the partition keys, the entity tuples in the dataset to be resolved are partitioned to obtain at least one disjoint partition. Only tuples that share the same hash function will be partitioned into the same partition.

[0079] Alternatively, if data partitioning is accomplished using a hash function of a similarity predicate in the execution plan (e.g., tA≈sB as mentioned above), there may be cases where tuples t1 and t2 represent the same real entity, but t1 and t2 are located in different partitions. In this case, the kernel of partition pi with local data can choose to pull partition pj from another kernel. If pi and pj are located on the same GPU, the pull operation is performed directly to pull t1 and t2 to the same partition without data transfer. If pi and pj are located on different GPUs, the corresponding function can be called to pull the corresponding tuples from the corresponding GPU to the same partition.

[0080] In this embodiment, based on S equal sign predicates in the execution plan that are not reused by at least two rules, S hash functions are used to partition the tuples in the dataset to be resolved, resulting in at least one disjoint partition. The provided partitioning strategy, based on the execution plan generated by the database optimizer, allows the S hash functions to be selective and may be shared by multiple rules. Therefore, efficient partitioning can be achieved with a small number of hash functions. Furthermore, each tuple is assigned an ID indicating the hash function used; only tuples sharing the same hash function are compared, thereby reducing redundant computation caused by multiple hash functions.

[0081] Referring to Figure 10, which is a flowchart illustrating a parallel processing method for entity resolution provided in Embodiment Six of this application, as shown in Figure 10, after dividing the entity tuples in the dataset to be resolved into at least one disjoint partition in step S201 above, the following steps may also be included:

[0082] Step S1001: Randomly hash and allocate a GPU to each partition. For any partition, if the GPU corresponding to the partition is available, then divide all entity tuples in the partition into N intervals.

[0083] In step S1002, if the GPU corresponding to the partition is unavailable, the system routes to the GPU corresponding to other partitions via a preset direction and executes the process of dividing all entity tuples within the partition into N intervals.

[0084] Specifically, when extending the entity resolution task to multiple GPUs, the partitions obtained by dividing the dataset to be resolved and the multiple GPUs can be randomly hashed to positions on the unit circle. For any partition, if the GPU corresponding to the partition is available, then all entity tuples in the partition are divided into N intervals (see steps S202 to S204 for details). If the partition is assigned to an unavailable GPU (the GPU has no free cores or no available PCIe (PCI Express) bandwidth), then it can be rerouted to the nearest available GPU in a clockwise direction, and all entity tuples in the partition are divided into N intervals (see steps S202 to S204 for details).

[0085] In this embodiment, when extending the entity resolution task to multiple GPUs, a GPU is randomly hashed and allocated to each partition. For any partition, if the GPU corresponding to the partition is available, all entity tuples within the partition are divided into N intervals. If the GPU corresponding to the partition is unavailable, the task is routed to the GPU corresponding to another partition via a preset direction, and all entity tuples within that partition are divided into N intervals. This achieves load balancing across multiple GPUs and reduces resource contention.

[0086] Corresponding to the entity resolution parallel processing method in the above embodiments, Figure 11 shows a structural block diagram of an entity resolution parallel processing apparatus provided in Embodiment 7 of this application. The above-mentioned entity resolution parallel processing apparatus is applied to the server in Figure 1. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0087] Referring to Figure 11, the entity digestion parallel processing device includes:

[0088] The entity partitioning module 1101 is used to obtain the dataset to be resolved and divide the entity tuples in the dataset to be resolved into at least one disjoint partition, wherein the dataset to be resolved includes at least two entity tuples.

[0089] The interval window module 1102 is used to divide all entity tuples in any partition into N intervals, form a window with M consecutive intervals, and process all windows sequentially, where N is greater than M and N and M are both positive integers greater than zero.

[0090] The thread allocation module 1103 is used to allocate a thread block to each interval within any window. The thread block is used to match the target tuple in the corresponding interval with the entity tuples in the partition other than the target tuple to obtain the matching result. The thread blocks within the window are processed in parallel.

[0091] The thread loop execution module 1104 is used to process the interval in the next window of the window if the thread block corresponding to any interval in the window has been executed, until all intervals have been processed by the corresponding thread blocks and all matching results are obtained. The all matching results are used to guide the dataset to be resolved to remove duplicate entity tuples.

[0092] Optionally, the aforementioned interval window module 1102 includes:

[0093] A sliding window unit is used to slide a window of a preset length over N intervals in steps of the preset length, wherein the preset length is the length of M consecutive intervals.

[0094] The interval determination unit is used to determine the interval included when each sliding stops, which is the interval of the corresponding window.

[0095] Optionally, the entity-dissolving parallel processing device further includes:

[0096] The bitmap construction module is used to construct a bitmap, wherein each bit in the bitmap represents the processing status of a range being processed by the corresponding thread block.

[0097] The cross-interval task stealing module is used to select an unprocessed interval from the bitmap for any thread block if the thread block is in an idle state. The idle thread block has processed all the intervals allocated to the window and completed the processing.

[0098] The cross-interval task processing module is used to process the unprocessed intervals using the thread block to obtain the corresponding matching results.

[0099] Optionally, the entity-dissolving parallel processing device further includes:

[0100] The detection module is used to detect whether there are any regions in the bitmap that are in an unprocessed state;

[0101] The first detection and determination module is used to select an unprocessed interval from the bitmap if an interval with an unprocessed state is detected in the bitmap.

[0102] The second detection and determination module is used to determine the interval whose processing status is "in execution" if no interval with an unprocessed status is detected in the bitmap.

[0103] The first task division module is used to notify the thread block corresponding to the execution interval to divide the remaining execution tasks corresponding to the execution interval into a preset number of subtasks;

[0104] The task processing module is used to leave one of the subtasks in the thread block corresponding to the execution interval for processing, and obtain the corresponding matching result;

[0105] The task stealing module within the interval is used to distribute each remaining subtask to an idle thread block for processing, and obtain the corresponding matching result.

[0106] Optionally, the entity-dissolving parallel processing device further includes:

[0107] The idle acquisition module is used to obtain the number of idle threads in all idle states;

[0108] The notification to the thread block corresponding to the execution interval divides the remaining execution tasks corresponding to the execution interval into a preset number of subtasks, including:

[0109] An idle sending unit is used to send the number of idle slots as a notification to the thread block corresponding to the executing interval.

[0110] The second task division unit is used to add 1 to the number of idle tasks to obtain a preset number of subtasks, and to divide the remaining tasks corresponding to the execution interval into the preset number of subtasks.

[0111] Optionally, the aforementioned entity partitioning module 1101 includes:

[0112] The predicate determination unit is used to obtain the execution plan of the dataset to be resolved, determine S equal sign predicates in the execution plan that are not used repeatedly by at least two rules, and obtain S hash functions based on the S equal sign predicates;

[0113] The hash calculation unit is used to call any one of the S hash functions to calculate the corresponding partition key for any entity tuple in the dataset to be resolved.

[0114] The hash partitioning unit is used to traverse all entity tuples to obtain the partition key of all entity tuples, and to divide the entity tuples in the dataset to be resolved according to the partition key to obtain at least one disjoint partition.

[0115] Optionally, the entity-dissolving parallel processing device further includes:

[0116] The first GPU scheduling module is used to randomly hash and allocate a GPU to each partition. For any partition, if the GPU corresponding to the partition is available, then the module executes the step of dividing all entity tuples in the partition into N intervals.

[0117] The second GPU scheduling module is used to route to the GPU corresponding to other partitions through a preset direction if the GPU corresponding to the partition is unavailable, and to execute the step of dividing all entity tuples in the partition into N intervals.

[0118] In one embodiment, a computer device, which may be a server, is provided, and its internal structure is shown in Figure 12. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, a readable storage medium, and the database. The internal memory provides an environment for the operation of the operating system and the readable storage medium. The database stores the dataset to be resolved. The network interface communicates with external terminals via a network connection. When executed by the processor, the readable storage medium implements a parallel processing method for entity resolution.

[0119] In one embodiment, a computer device is provided, including a memory, a processor, and a readable storage medium stored on the memory and operable on the processor. When the processor executes the readable storage medium, it implements the steps of the entity resolution parallel processing method in the above embodiments, such as steps S201-S204 shown in FIG2, or the steps shown in FIG3 to FIG10. To avoid repetition, these steps will not be described again here. Alternatively, the processor executes the readable storage medium, implementing the functions of each module / unit in this embodiment of the entity resolution parallel processing device, such as the functions of the entity partitioning module 1101, the interval window module 1102, the thread allocation module 1103, and the thread loop execution module 1104 shown in FIG11. To avoid repetition, these functions will not be described again here.

[0120] In one embodiment, one or more readable storage media storing computer-readable instructions are provided. When executed by one or more processors, these computer-readable instructions cause the processors to perform the steps of the entity resolution parallel processing method described in the above embodiments, such as steps S201-S204 shown in FIG2, or the steps shown in FIG3 to FIG10. To avoid repetition, these steps will not be described again here. Alternatively, when the processor executes the readable storage medium, it implements the functions of each module / unit in this embodiment of the entity resolution parallel processing apparatus, such as the functions of the entity partitioning module 1101, interval window module 1102, thread allocation module 1103, and thread loop execution module 1104 shown in FIG11. To avoid repetition, these functions will not be described again here. The readable storage medium in this embodiment includes non-volatile readable storage media and volatile readable storage media.

[0121] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by instructing related hardware through a readable storage medium. The readable storage medium can be stored in a non-volatile computer-readable storage medium, which, when executed, can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0122] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0123] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A parallel processing method for entity resolution, characterized in that, The parallel processing method for entity resolution includes: Obtain the dataset to be resolved, and divide the entity tuples in the dataset into at least one disjoint partition, wherein the dataset to be resolved includes at least two entity tuples; For any partition, all entity tuples within the partition are divided into N intervals. M consecutive intervals are formed into a window. All windows are processed sequentially. N is greater than M, and both N and M are positive integers greater than zero. For any given window, a thread block is allocated to each interval within the window. The thread block is used to match the target tuple within the corresponding interval with the entity tuples within the partition excluding the target tuple to obtain the matching result. The thread blocks within the window are processed in parallel. If the thread block corresponding to any interval within the window is completed, the completed thread block is used to process the interval in the next window of the window until all intervals are processed by the corresponding thread blocks to obtain all matching results. The complete matching results are used to guide the dataset to be resolved to remove duplicate entity tuples.

2. The parallel processing method for entity resolution according to claim 1, characterized in that, The step of forming a window from M consecutive intervals includes: Using a window of a preset length, slide across the N intervals with a step size of the preset length, where the preset length is the length of M consecutive intervals; The interval included when each slide stops is determined to be the interval of the corresponding window.

3. The parallel processing method for entity resolution according to claim 1, characterized in that, After forming a window from M consecutive intervals, the method further includes: Construct a bitmap, where each bit in the bitmap represents the processing status of a range being processed by the corresponding thread block; For any thread block, if the thread block is in an idle state, then select an unprocessed interval from the bitmap, wherein the idle thread block has processed all the intervals allocated to the window and the processing is complete. The thread block is used to process the unprocessed interval to obtain the corresponding matching result.

4. The parallel processing method for entity resolution according to claim 3, characterized in that, Before selecting an unprocessed interval from the bitmap, the method further includes: Detect whether there are any regions in the bitmap that are in an unprocessed state; If an unprocessed interval is detected in the bitmap, then the step of selecting an unprocessed interval from the bitmap is executed. If no interval in the bitmap is detected to be in an unprocessed state, then the interval is determined to be in an executing state. The thread block corresponding to the execution interval is notified to divide the remaining execution tasks corresponding to the execution interval into a preset number of subtasks; One of the subtasks is left to be processed in the thread block corresponding to the execution interval to obtain the corresponding matching result; Each remaining subtask is distributed to an idle thread block for processing to obtain the corresponding matching result.

5. The parallel processing method for entity resolution according to claim 4, characterized in that, Before notifying the thread block corresponding to the interval in execution, the following is also included: Get the number of idle threads in all idle states; The notification to the thread block corresponding to the execution interval divides the remaining execution tasks corresponding to the execution interval into a preset number of subtasks, including: The number of idle slots is sent as a notification to the thread block corresponding to the execution interval. The number of idle tasks is incremented by 1 to obtain a preset number of tasks, and the remaining tasks corresponding to the execution interval are divided into the preset number of subtasks.

6. The parallel processing method for entity resolution according to claim 1, characterized in that, The step of dividing the entity tuples in the dataset to be resolved into at least one disjoint partition includes: Obtain the execution plan of the dataset to be resolved, determine S equal sign predicates in the execution plan that are not used repeatedly by at least two rules, and obtain S hash functions based on the S equal sign predicates; For any entity tuple in the dataset to be resolved, any one of the S hash functions is called to perform calculation to obtain the corresponding partition key; Traverse all entity tuples to obtain the partition keys of all entity tuples. Based on the partition keys, divide the entity tuples in the dataset to be resolved to obtain at least one disjoint partition.

7. The parallel processing method for entity resolution according to any one of claims 1 to 6, characterized in that, After dividing the entity tuples in the dataset to be resolved into at least one disjoint partition, the method further includes: For each partition, a GPU is randomly hashed and allocated. For any partition, if the GPU corresponding to the partition is available, then the process of dividing all entity tuples in the partition into N intervals is executed. If the GPU corresponding to the partition is unavailable, then the system routes to the GPU corresponding to other partitions via a preset direction and executes the step of dividing all entity tuples in the partition into N intervals.

8. A parallel processing apparatus for entity digestion, characterized in that, The entity resolution parallel processing device includes: The entity partitioning module is used to obtain the dataset to be resolved and divide the entity tuples in the dataset to be resolved into at least one disjoint partition, wherein the dataset to be resolved includes at least two entity tuples. The interval window module is used to divide all entity tuples in any partition into N intervals, form a window with M consecutive intervals, and process all windows sequentially. Here, N is greater than M, and N and M are both positive integers greater than zero. The thread allocation module is used to allocate a thread block to each interval within any window. The thread block is used to match the target tuple in the corresponding interval with the entity tuples in the partition other than the target tuple to obtain the matching result. The thread blocks within the window are processed in parallel. The thread loop execution module is used to process the interval in the next window of the window if the thread block corresponding to any interval in the window has been executed, until all intervals have been processed by the corresponding thread blocks and all matching results are obtained. The all matching results are used to guide the dataset to be resolved to remove duplicate entity tuples.

9. A computer device comprising a memory, a processor, and a readable storage medium stored in the memory and operable on the processor, wherein, When the processor executes the readable storage medium, it performs the following steps: Obtain the dataset to be resolved, and divide the entity tuples in the dataset into at least one disjoint partition, wherein the dataset to be resolved includes at least two entity tuples; For any partition, all entity tuples within the partition are divided into N intervals. M consecutive intervals are formed into a window. All windows are processed sequentially. N is greater than M, and both N and M are positive integers greater than zero. For any given window, a thread block is allocated to each interval within the window. The thread block is used to match the target tuple within the corresponding interval with the entity tuples within the partition excluding the target tuple to obtain the matching result. The thread blocks within the window are processed in parallel. If the thread block corresponding to any interval within the window is completed, the completed thread block is used to process the interval in the next window of the window until all intervals are processed by the corresponding thread blocks to obtain all matching results. The complete matching results are used to guide the dataset to be resolved to remove duplicate entity tuples.

10. The computer device according to claim 9, wherein, The step of forming a window from M consecutive intervals includes: Using a window of a preset length, slide across the N intervals with a step size of the preset length, where the preset length is the length of M consecutive intervals; The interval included when each slide stops is determined to be the interval of the corresponding window.

11. The computer device according to claim 9, wherein, After forming a window from M consecutive intervals, the method further includes: Construct a bitmap, where each bit in the bitmap represents the processing status of a range being processed by the corresponding thread block; For any thread block, if the thread block is in an idle state, then select an unprocessed interval from the bitmap, wherein the idle thread block has processed all the intervals allocated to the window and the processing is complete. The thread block is used to process the unprocessed interval to obtain the corresponding matching result.

12. The computer device according to claim 11, wherein, Before selecting an unprocessed interval from the bitmap, the method further includes: Detect whether there are any regions in the bitmap that are in an unprocessed state; If an unprocessed interval is detected in the bitmap, then the step of selecting an unprocessed interval from the bitmap is executed. If no interval in the bitmap is detected to be in an unprocessed state, then the interval is determined to be in an executing state. The thread block corresponding to the execution interval is notified to divide the remaining execution tasks corresponding to the execution interval into a preset number of subtasks; One of the subtasks is left to be processed in the thread block corresponding to the execution interval to obtain the corresponding matching result; Each remaining subtask is distributed to an idle thread block for processing to obtain the corresponding matching result.

13. The computer device according to claim 12, wherein, Before notifying the thread block corresponding to the interval in execution, the following is also included: Get the number of idle threads in all idle states; The notification to the thread block corresponding to the execution interval divides the remaining execution tasks corresponding to the execution interval into a preset number of subtasks, including: The number of idle slots is sent as a notification to the thread block corresponding to the execution interval. The number of idle tasks is incremented by 1 to obtain a preset number of tasks, and the remaining tasks corresponding to the execution interval are divided into the preset number of subtasks.

14. The computer device according to claim 9, wherein, The step of dividing the entity tuples in the dataset to be resolved into at least one disjoint partition includes: Obtain the execution plan of the dataset to be resolved, determine S equal sign predicates in the execution plan that are not used repeatedly by at least two rules, and obtain S hash functions based on the S equal sign predicates; For any entity tuple in the dataset to be resolved, any one of the S hash functions is called to perform calculation to obtain the corresponding partition key; Traverse all entity tuples to obtain the partition keys of all entity tuples. Based on the partition keys, divide the entity tuples in the dataset to be resolved to obtain at least one disjoint partition.

15. The computer device according to claims 9 to 14, wherein, After dividing the entity tuples in the dataset to be resolved into at least one disjoint partition, the method further includes: For each partition, a GPU is randomly hashed and allocated. For any partition, if the GPU corresponding to the partition is available, then the process of dividing all entity tuples in the partition into N intervals is executed. If the GPU corresponding to the partition is unavailable, then the system routes to the GPU corresponding to other partitions via a preset direction and executes the step of dividing all entity tuples in the partition into N intervals.

16. One or more readable storage media storing computer-readable instructions, wherein, When the computer-readable instructions are executed by one or more processors, the one or more processors cause the processors to perform the following steps: Obtain the dataset to be resolved, and divide the entity tuples in the dataset into at least one disjoint partition, wherein the dataset to be resolved includes at least two entity tuples; For any partition, all entity tuples within the partition are divided into N intervals. M consecutive intervals are formed into a window. All windows are processed sequentially. N is greater than M, and both N and M are positive integers greater than zero. For any given window, a thread block is allocated to each interval within the window. The thread block is used to match the target tuple within the corresponding interval with the entity tuples within the partition excluding the target tuple to obtain the matching result. The thread blocks within the window are processed in parallel. If the thread block corresponding to any interval within the window is completed, the completed thread block is used to process the interval in the next window of the window until all intervals are processed by the corresponding thread blocks to obtain all matching results. The complete matching results are used to guide the dataset to be resolved to remove duplicate entity tuples.

17. The readable storage medium according to claim 16, wherein, The step of forming a window from M consecutive intervals includes: Using a window of a preset length, slide across the N intervals with a step size of the preset length, where the preset length is the length of M consecutive intervals; The interval included when each slide stops is determined to be the interval of the corresponding window.

18. The readable storage medium according to claim 16, wherein, After forming a window from M consecutive intervals, the method further includes: Construct a bitmap, where each bit in the bitmap represents the processing status of a range being processed by the corresponding thread block; For any thread block, if the thread block is in an idle state, then select an unprocessed interval from the bitmap, wherein the idle thread block has processed all the intervals allocated to the window and the processing is complete. The thread block is used to process the unprocessed interval to obtain the corresponding matching result.

19. The readable storage medium according to claim 18, wherein, Before selecting an unprocessed interval from the bitmap, the method further includes: Detect whether there are any regions in the bitmap that are in an unprocessed state; If an unprocessed interval is detected in the bitmap, then the step of selecting an unprocessed interval from the bitmap is executed. If no interval in the bitmap is detected to be in an unprocessed state, then the interval is determined to be in an executing state. The thread block corresponding to the execution interval is notified to divide the remaining execution tasks corresponding to the execution interval into a preset number of subtasks; One of the subtasks is left to be processed in the thread block corresponding to the execution interval to obtain the corresponding matching result; Each remaining subtask is distributed to an idle thread block for processing to obtain the corresponding matching result.

20. The readable storage medium according to claim 19, wherein, Before notifying the thread block corresponding to the interval in execution, the following is also included: Get the number of idle threads in all idle states; The notification to the thread block corresponding to the execution interval divides the remaining execution tasks corresponding to the execution interval into a preset number of subtasks, including: The number of idle slots is sent as a notification to the thread block corresponding to the execution interval. The number of idle tasks is incremented by 1 to obtain a preset number of tasks, and the remaining tasks corresponding to the execution interval are divided into the preset number of subtasks.

Images