12 results about "Hash join" patented technology

A join decision manager (JDM) generates a data processing pipeline. The data processing pipeline includes at least one join operation associated with build-side row data and probe-side row data. The JDM determines the maximum cardinality associated with the probe-side row data. The JDM determines size of the build-side row data at a decision node of the at least one join operation. The JDM configures execution of the at least one join operation as one of a broadcast join or a hash-hash join based on the size of the build-side row data and the maximum cardinality.

Provided in the embodiments of the present disclosure are a hash join processing method, system and device, and a storage medium and a program product. The method comprises: scanning a plurality of rows of data in a first data table within a target scanning range corresponding to a target worker thread, and on the basis of a first hash join mode, writing the plurality of rows of data into a first hash table in a memory; if it is determined that an available memory capacity corresponding to a hash join task is lower than a set threshold, sending the scanned plurality of rows of data to a data partitioning management thread, and switching to use a second hash join mode; and receiving first target row data distributed by the data partitioning management thread, and on the basis of the second hash join mode, writing the first target row data into a second hash table corresponding to the target worker thread in the memory, so as to perform persistence processing of the data in the second hash table. The present disclosure can flexibly switch between different hash join modes on the basis of the memory occupancy of a data table for hash table creation.

In an embodiment, a computer system performs a hash join. During a build phase, the computer system constructs a hash table in memory based on rows of a first table. The constructing may result in build batches of rows, including one build batch stored in memory and multiple build batches stored in a storage. The computer system determines whether any of the multiple build batches is skewed according to a data skew condition. In response to determining that there is at least one build batch that is skewed, the computer system loads one or more of the multiple build batches into memory such that there are at least two build batches stored in memory. During a probe phase, the computer system identifies, based on the at least two build batches stored in memory, rows of a second table to join with the rows of the first table.

Provided herein are systems, methods, and computer-storage media for managing data skew in hash join operations. A skew manager partitions build-side row data into multiple sets corresponding to hash-join-build (HJB) instances based on hash values. The skew manager detects skew in a build-side row set associated with a first HJB instance by analyzing the number of rows. Upon detecting skew, the skew manager redirects data rows to at least a second HJB instance. The method involves configuring skew caches, generating histograms, and detecting frequent hash values to identify skew. It also includes communicating skew notifications, broadcasting probe-side row data, and adjusting partitioning of probe-side data. The disclosed techniques further include buffering build-side row sets in streams and performing join operations based on these streams, enhancing efficiency in distributed computing environments.

The application provides a Bloom filter configuration method for a hash connection, comprising: sampling initial data, analyzing the initial data, dynamically analyzing a variety of change trends, fitting a change trend model, constructing a Bloom filter, evaluating a Bloom filter size, and adjusting a Bloom filter configuration. The application combines probability statistics and mathematical modeling to dynamically analyze part of data sets and estimate code value inclination, uses a change slope and a coefficient of variation to judge a stable trend of data, uses Hausdorff Distance to select an optimal model to estimate a Bloom filter, finally proposes different configuration Bloom filter strategies, efficiently configures a Bloom filter size, fully utilizes memory resources, and improves overall query efficiency of the hash connection.

This invention relates to the field of database query technology, providing a method and system for parallel hash join queries based on the OpenGauss database. The method includes: determining whether a shared hash table can store all inner table data; creating initialization conditions for the construction of the shared hash table and the joining of inner and outer tuples; when the shared hash table can store all inner table data, multi-threaded parallel scanning of the inner tables is performed, with a barrier structure controlling the construction of the shared hash table; after the shared hash table construction is completed, parallel scanning of the outer table is performed, and tuples from the inner and outer tables are joined; when the shared hash table cannot store all inner table data, the shared hash table is constructed in batches; after the current batch of shared hash table construction is completed, parallel scanning of the outer table is performed; after joining the current batch of tuples from the inner and outer tables, the next batch of shared hash tables is constructed, and the next batch of tuple joins from the inner and outer tables are executed. This method and system can improve CPU utilization and increase the efficiency of hash join queries.

A system includes at least one hardware processor and memory storing instructions. The processor generates a query plan for a received query. The query plan includes multiple hash-join-build and hash-join-probe operations. A primary decision node is configured in the query plan. The primary decision node receives build-side data information from the hash-join-build operations. For each hash-join-build operation, a memory amount for performing a broadcast is determined. A subset of hash-join-build operations is selected for broadcast join distribution by comparing the memory amount to a broadcast memory threshold. The system selects a broadcast join distribution for the subset and a hash-hash join distribution for the remaining hash-join-build operations. The query plan is executed using the broadcast join distribution for the selected subset and the hash-hash join distribution for the remaining operations. This approach optimizes memory usage and join distribution during query execution.

The invention is suitable for the technical field of databases, and relates to a data query method and device, equipment and a medium. The method comprises the following steps: converting a received structured query statement into a physical execution plan, and if the physical execution plan comprises a Hash join operator, allocating a corresponding memory quota to the Hash join operator; if the first memory demand of the Hash table constructed by the Hash join operator is greater than the memory quota, executing the Hash join operator by adopting a memory elimination strategy; the memory elimination strategy comprises the steps that an execution quota is divided from memory quotas, the execution quota comprises a first sub-quota used for storing a hash table structure corresponding to a hash table and connection key data and a second sub-quota used for storing a data record of a small table related to the hash table, and the first sub-quota is larger than the second sub-quota; and respectively performing longest unused elimination management on the first sub-quota and the second sub-quota in the execution process of the Hash join operator. According to the invention, the execution efficiency of the Hash connection can be improved under the condition that the memory is limited.

The application discloses a big data table hash connection hardware acceleration method and device, equipment and medium, and relates to the database heterogeneous acceleration field. The method is applied to an acceleration board card and comprises the following steps: receiving a connection operation type instruction; applying a first memory space of a first to-be-connected data table and a second memory space of a second to-be-connected data table respectively by using a connection table size instruction, so as to store each first shard of the first to-be-connected data table into the first memory space and store each second shard of the second to-be-connected data table into the second memory space; and performing hash connection calculation on each first shard in the first memory space and each second shard in the second memory space according to the connection operation type instruction and a preset parallel processing row number, so as to obtain corresponding connection table row data. Through the above scheme, the hash connection hardware acceleration effect can be effectively improved.

The application discloses a method for performing hash connection of a distributed database, and relates to the technical field of distributed databases, which comprises the following steps: setting a relative inclination rate, calculating the product of the relative inclination rate and table data volume to obtain an inclination threshold; obtaining a heat map of table statistical information, screening elements in the table which exceed the inclination threshold to obtain an inclination value; expanding the table statistical information by using the inclination value; executing an SQL statement to obtain two input data tables, generating a hash connection physical plan by using the new table statistical information, performing hash distribution, average distribution or mirror distribution on tuples in the input data tables according to the plan, and after each node receives the tuples of the input data tables, establishing a hash table by using input data table data with small data volume, performing detection by using input data table data with large data volume, and finally performing a set union on the hash connection results of each node, so that the set union is the final hash connection result. The application can realize balanced distribution of tasks and shorten the total SQL execution time.

A data page index-based Hash Join query method, apparatus and system, and a medium. The method comprises: determining a build table and a probe table for a join query, as well as a join key field (S101); building a corresponding Hash table on the basis of data of the join key field in the build table, and determining whether a pre-created data page index of the join key field is present in the probe table, the data page index being an index structure using at least one data page as a unit (S102); if a pre-created data page index of the join key field is present in the probe table, associating the data page index of the join key field with the Hash table, and performing data page filtering for the probe table on the basis of an association result to generate a list of data pages to be accessed (S103); and reading corresponding data in the probe table on the basis of the list of data pages to be accessed, and performing a Hash Join with the Hash table to generate a corresponding data query result (S104).

There are provided a method, device, and storage medium for processing a data table. The method includes: performing an equivalent join on a left table and a right table in two data tables that are to be joined by a hash join, and acquiring first associated data of an association between the left table and the right table; filtering the first associated data according to a predetermined non-equivalent filtering condition, and determining second associated data satisfying the predetermined non-equivalent filtering condition in the first associated data; identifying, in a predetermined data structure according to a target join type of the two data tables; and processing the left table and / or the right table according to the target join type of the two data tables and the predetermined data structure to generate a target join table.