Data processing method and apparatus, and server, system, medium and product
By allocating data table routing information to database servers and utilizing indexes and numerical ranges, the problem of frequent data transfer between multiple database servers was solved, improving system performance and stability.
Patent Information
- Application Number
- PCT/CN2025/141138
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-09
- Filing Date
- 2025-12-09
- Publication Date
- 2026-11-12
AI Technical Summary
When multiple database servers access the same data in shared storage, the data may be frequently transferred to another database server, causing data access conflicts and performance bottlenecks.
By assigning specific data table routing information to each database server and utilizing the indexes and value ranges of the target data table, business requests are ensured to be distributed to the corresponding database server, avoiding random distribution and reducing frequent data transfers between servers.
This effectively avoids situations where multiple database servers access the same data simultaneously, reduces data access conflicts, and improves system response speed and throughput.
Smart Images

Figure CN2025141138_12112026_PF_FP_ABST
Abstract
Description
Data processing method, device, server, system, medium and product
[0001] The present application claims priority from the Chinese patent application No. 202510600993.4 filed on May 09, 2025, and entitled "Data processing method, device, server, system, medium and product", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD
[0002] The present application relates to the technical field of database, and in particular to a data processing method, device, server, system, medium and product. BACKGROUND
[0003] In a distributed database based on shared storage, multiple database servers can access data in the shared storage. However, when multiple database servers need to access the same data in the shared storage, the data may be frequently transferred from one database server to another database server, resulting in data access conflicts. SUMMARY
[0004] The present application provides a data processing method, device, server, system, medium and product, which can reduce the situation that data is frequently transferred from one database server to another database server, thereby avoiding data access conflicts.
[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions.
[0006] In a first aspect, a data processing method is provided. The method can be executed by a database server, or by a component of the database server, such as a processor, a chip, or a chip system of the database server, or by a logic module or software that can realize all or part of the functions of the database server, such as a database instance in the database server.
[0007] The method includes receiving, by a database server, a target query statement from a business server. The target query statement carries information of a target data table. The database server sends routing information of the target data table to the business server based on the target query statement.
[0008] The routing information is used to determine a database server that processes at least one data table requested by a business request. The target data table is one of the at least one data table.
[0009] In the above technical solution, when the database server receives the target query statement from the business server, it sends routing information of the target data table to the business server based on the information of the target data table carried in the query statement. This allows the business server to determine a database server to handle the business request based on the routing information of the target data table. In this way, a business request can be handled by a specific database server; that is, a business request will be distributed to its corresponding database server, rather than being randomly distributed to different database servers. This avoids the situation where multiple database servers access the same data, and also prevents data from being frequently transferred from one database server to another, thereby reducing data access conflicts.
[0010] In some possible implementations, the routing information includes multiple numerical ranges of the index of the target data table and information about the database server corresponding to each numerical range, with one numerical range corresponding to one database server.
[0011] In this context, the index of the target data table is a data structure used to optimize data retrieval. It can be understood as similar to a table of contents in a book, used to establish fast query paths for data in the target data table, thereby improving query efficiency. For example, the index of the target data table can be a B-tree index. Multiple numerical ranges are obtained by dividing the numerical distribution range of the index of the target data table.
[0012] In the above implementation, when determining the database server to process a business request, business requests for data belonging to the same numerical range will be distributed to the database server corresponding to that range, rather than being randomly distributed to different database servers. This avoids situations where multiple database servers access the same data, and prevents data from frequently being transferred from one database server to another, thereby reducing data access conflicts.
[0013] In some possible implementations, the process of determining the routing information of the target data table includes: obtaining the numerical distribution range of the index of the target data table; dividing the numerical distribution range into multiple numerical intervals according to the number of multiple database servers included in the database system; and generating the routing information of the target data table based on the multiple numerical intervals and the information of the multiple database servers.
[0014] In this context, the multiple database servers can refer to all the data servers included in the database system. Furthermore, in some possible implementations, the multiple database servers can also refer to the active database servers among all the data servers included in the database system. Here, an active database server refers to a database server that is currently operating normally and capable of providing services. It is understood that the number of multiple database servers is the same as the number of multiple numerical ranges, to ensure that one numerical range corresponds to one database server.
[0015] The above embodiments provide a method for determining routing information for a target data table. Specifically, the numerical distribution range of the target data table's index is divided according to the number of database servers, resulting in an equal number of numerical intervals, thus ensuring that each numerical interval corresponds to one database server. Furthermore, the multiple numerical intervals and the corresponding database server information are stored to generate routing information, facilitating the normal execution of subsequent data processing flows. It is understood that the above method can be used to determine the routing information for each data table included in the database system.
[0016] In some possible implementations, the index is a B-tree index. Accordingly, the process of obtaining the numerical distribution range of the index of the target data table can include the following two implementation methods.
[0017] Implementation Method 1: Based on the key values stored in the root node of the B-tree index of the target data table, determine the numerical distribution range of the index. The root node is the top-level node of the B-tree index. It's understood that there is only one root node in a B-tree index. The key values stored in the root node are used to divide the overall numerical range of the B-tree index (i.e., the numerical range of the subtrees connected to the root node) into multiple sub-ranges. In other words, the numerical distribution range of the index can include these multiple sub-ranges.
[0018] Method 2: Determine the numerical distribution range of the target data table's B-tree index based on the high key values stored in multiple intermediate nodes. An intermediate node can be a non-leaf node at the next level below the root node (e.g., a second-level non-leaf node) or the next-next-level non-leaf node (e.g., a third-level non-leaf node). It's understood that there are multiple intermediate nodes in the B-tree index. The key values stored in the intermediate nodes are used to divide the numerical range of the subtrees connected to that intermediate node into multiple sub-ranges. The high key value refers to the largest key value among at least one key value stored in the intermediate node, that is, the largest key value in the subtree managed by the intermediate node. It's understood that the high key values stored in multiple intermediate nodes of the B-tree index also divide the numerical range of the subtrees connected to the root node into multiple sub-ranges, thus dividing the overall numerical range of the B-tree index into multiple sub-ranges. Therefore, the numerical distribution range of the B-tree index can also be determined based on the high key values stored in multiple intermediate nodes.
[0019] In the above implementation, by analyzing the key values stored in the root node of the B-tree index of the target data table, or the high key values stored in multiple intermediate nodes of the B-tree index of the target data table, the numerical distribution range of the index of the target data table can be accurately and efficiently determined.
[0020] In some possible implementations, the index of the target data table is the index corresponding to the target index column of the target data table. The target index column is at least one of the most frequently accessed index columns.
[0021] Accordingly, the method further includes: selecting a target index column from at least one index column of the target data table based on the business access records of the target data table. Then, the index corresponding to the target index column is determined as the index of the target data table.
[0022] In the above implementation, by selecting the most frequently accessed index column in the data table, we can obtain the index columns that are frequently queried in business requests, i.e., we can obtain the index columns favored by the business. Furthermore, determining the index of the target data table based on the business-favored index columns increases the amount of information referenced in index determination, enabling us to obtain the most frequently accessed index, thereby significantly improving query performance based on business requests and optimizing the overall performance of the database system.
[0023] In some possible implementations, the target index column is the index column corresponding to the first edge with the largest weight value among at least one first edge connecting the node corresponding to the target data table in the index analysis graph.
[0024] Accordingly, the method also includes generating an index analysis graph based on at least one index column of all data tables included in the database system and the business access records of each data table.
[0025] The index analysis graph includes multiple nodes, at least one first edge connected to each node, and the weight of each first edge. Nodes are used to indicate data tables, and at least one first edge connected to each node is used to indicate at least one index column of the data table corresponding to the node. The weight of the first edge is used to indicate the number of times the index column corresponding to the first edge has been accessed.
[0026] Furthermore, after generating the index analysis graph, the index column corresponding to the first edge with the largest weight value can be selected from at least one first edge connecting the node corresponding to the target data table in the index analysis graph, and used as the target index column.
[0027] In some possible implementations, the index analysis graph is also used to characterize the relationships between all the data tables included in the database system. The index analysis graph also includes a second edge between nodes, which is used to characterize the relationship between the data tables corresponding to the two nodes connected by the second edge.
[0028] In the above implementation, by constructing an index analysis graph, using nodes to represent data tables, using the first edge to represent index columns, and adding edge weights to characterize the number of times the corresponding index column is accessed, the index column with the most accesses in the target data table can be quickly determined by analyzing the edge weights of each first edge connected to the corresponding node of the target data table, that is, the target index column is obtained.
[0029] In some possible implementations, the index can be a single-column index or a composite index. A single-column index is created based on a single indexed column. A composite index is created based on multiple indexed columns.
[0030] Accordingly, if the index is a single-column index, the index of the target data table is an index created based on the target index column. Alternatively, if the index is a multi-column index, the index of the target data table is an index created with the target index column as the first index column. This provides a method for determining whether an index is a single-column or composite index, allowing for quick and efficient determination of the corresponding index after obtaining the target index column.
[0031] In some possible implementations, after generating routing information for the target data table based on multiple numerical ranges and information from multiple database servers, the method further includes:
[0032] The routing information for the target data table is stored in the database. The database stores the routing information for the data tables within the database system and supports access from multiple database servers.
[0033] In the above implementation, the routing information of the target data table is stored in the database so that the database server can subsequently obtain the routing information of the target data table from the database. Furthermore, the database supports access from multiple database servers, meaning it can be accessed by any one of them, thus improving the flexibility of the database server.
[0034] In some possible implementations, before sending the routing information of the target data table to the business server, the method further includes: obtaining the routing information of the target data table from the database based on the information of the target data table carried in the target query statement, so as to send the routing information of the target data table to the business server.
[0035] In some possible implementations, the method further includes sending index reference information to a business server. The index reference information indicates the position of a target parameter in a target query statement, the target parameter indicates data to be processed related to an index of a target data table, and the data to be processed is used to determine a database server for processing at least one data table requested by the business request.
[0036] In the above implementation, by sending index reference information to the business server, the business server can know the position of the target parameter in the target query statement, and then determine a database server to process the business request based on the data to be processed indicated by the target parameter.
[0037] Secondly, a data processing method is provided. This method can be executed by a business server, or by a component of the business server, such as the processor, chip, or chip system of the business server, or by a logic module or software that can implement all or part of the functions of the business server, such as the database driver in the business server.
[0038] The method includes: a business server receiving a business request. The business request requests processing of at least one data table, specifically requesting processing of data in at least one data table. The business request includes at least one query statement. The business server sends the target query statement to any one of the multiple database servers included in the database system. The target query statement is one of the at least one query statement, and it carries information about a target data table, which is one of the at least one data table.
[0039] Furthermore, the business server receives routing information for the target data table from the database server. Based on this routing information, the business server determines which database server will handle the business request.
[0040] In the above technical solution, the business server receives a business request and sends the target query statement from the request to any one of the multiple database servers included in the database system. When the database server receives the target query statement, it sends routing information for the target data table to the business server based on the information of the target data table carried in the query statement. Then, after receiving the routing information for the target data table from the database server, the business server can determine which database server will handle the business request based on this information. This ensures that a business request can be processed by a specific database server; that is, a business request will be distributed to its corresponding database server, rather than being randomly distributed to different database servers. This avoids the situation where multiple database servers access the same data, and prevents data from being frequently transferred from one database server to another, thereby reducing data access conflicts.
[0041] In some possible implementations, the routing information includes multiple numerical ranges of the index of the target data table and information about the database server corresponding to each numerical range, with one numerical range corresponding to one database server.
[0042] In the above implementation, when determining the database server to process a business request, business requests for data belonging to the same numerical range will be distributed to the database server corresponding to that range, rather than being randomly distributed to different database servers. This avoids situations where multiple database servers access the same data, and prevents data from frequently being transferred from one database server to another, thereby reducing data access conflicts.
[0043] In some possible implementations, the business server determines a database server to handle the business request based on the routing information of the target data table. This includes: the business server parsing the target query statement to obtain the data to be processed from the target data table requested by the business request; and the business server, based on the data to be processed, determining from the routing information a database server corresponding to the numerical range to which the data to be processed belongs as the database server to handle the business request.
[0044] In the above implementation, by parsing the target query statement of the business request, the data to be processed in the target data table requested by the business request can be obtained. Then, based on the value range to which the data to be processed belongs, a database server corresponding to the value range to which the data belongs can be determined, that is, the database server to process the business request is determined.
[0045] In some possible implementations, before determining a database server to process a business request based on routing information of a target data table, the method further includes: receiving index reference information sent by the database server, the index reference information indicating the position of a target parameter in a target query statement, the target parameter indicating data to be processed related to an index of the target data table.
[0046] Accordingly, the target query statement is parsed, including parsing the target query statement based on index reference information.
[0047] In the above implementation, by sending index reference information to the business server, the business server can know the position of the target parameter in the target query statement, and then determine a database server to process the business request based on the data to be processed indicated by the target parameter.
[0048] In some possible implementations, after receiving the routing information of the data table from the database server, the method further includes: the business server caching the routing information of the target data table. In this way, the business server can determine the database server to execute the business request based on the cached routing information.
[0049] Thirdly, a data processing apparatus is provided for implementing any of the methods provided in the first aspect. This data processing apparatus includes modules, units, or means corresponding to the methods described above. The actions performed by these modules, units, or means can be implemented in hardware, software, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the functions described above.
[0050] In one possible implementation, the device may include a receiving module and a transmitting module; wherein:
[0051] The receiving module is used to receive the target query statement from the business server. The target query statement carries information about the target data table.
[0052] The sending module is used to send routing information of the target data table to the business server based on the target query statement;
[0053] The routing information is used to determine a database server that requests the processing of at least one data table for the business request, and the target data table is one of the at least one data table.
[0054] Fourthly, a data processing apparatus is provided for implementing any of the methods provided in the second aspect above. The data processing apparatus includes modules, units, or means corresponding to the methods described above. The actions performed by these modules, units, or means can be implemented in hardware, software, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the functions described above.
[0055] In one possible implementation, the device may include a receiving module, a transmitting module, and a determining module; wherein:
[0056] The receiving module is used to receive business requests. The business requests are for processing at least one data table and include at least one query statement.
[0057] The sending module is used to send a target query statement to any one of the multiple database servers included in the database system. The target query statement is one of at least one query statement and carries information about the target data table. The target data table is one of at least one data table.
[0058] The receiving module is also used to receive routing information of the target data table sent by the database server;
[0059] The determination module is used to determine a database server to handle business requests based on the routing information of the target data table.
[0060] Fifthly, a database server is provided, comprising: a memory and a processor, the memory and the processor being connected; the memory being used to store computer-executed instructions; and the processor being used to invoke the computer-executed instructions, thereby implementing the method of the first aspect above or any implementation thereof.
[0061] The database server in the fifth aspect can be: a database server in any implementation of the first aspect, or an apparatus containing the database server, or an apparatus contained in the database server, such as a chip.
[0062] In a sixth aspect, a business server is provided, comprising: a memory and a processor, the memory and the processor being connected; the memory being used to store computer-executed instructions; and the processor being used to invoke the computer-executed instructions, thereby implementing the method of the second aspect above or any implementation thereof.
[0063] The service server in the sixth aspect can be: a service server in any implementation of the second aspect, or a device containing the service server, or a device contained in the service server, such as a chip.
[0064] In a seventh aspect, a data processing system is provided, including a database server as described in the fifth aspect above and a business server as described in the sixth aspect above.
[0065] Eighthly, a chip is provided, comprising: a processor and an interface circuit; the interface circuit for receiving computer execution instructions and transmitting them to the processor; and the processor for executing the computer execution instructions to perform the methods of the first aspect, the second aspect, or any implementation thereof described above.
[0066] When the chip is deployed on a database server, the processor is used to execute computer execution instructions to perform the method of the first aspect or any implementation thereof described above. When the chip is deployed on a business server, the processor is used to execute computer execution instructions to perform the method of the second aspect or any implementation thereof described above.
[0067] Ninthly, a computer-readable storage medium is provided, comprising computer-executable instructions that, when executed on a database server, cause the database server to perform the method described in the first aspect or any implementation thereof, and when executed on a business server, cause the business server to perform the method described in the second aspect or any implementation thereof.
[0068] In a tenth aspect, a computer program product is provided, comprising computer execution instructions that, when the computer execution instructions are run on a database server, cause the database server to execute the method described in the first aspect or any implementation thereof, and when the computer execution instructions are run on a business server, cause the business server to execute the method described in the second aspect or any implementation thereof.
[0069] The technical effects of any of the implementation methods in aspects three through ten can be found in the technical effects of the corresponding implementation methods in aspects one or two, and will not be repeated here.
[0070] All possible implementations of any of the above aspects can be combined, provided that the solutions do not contradict each other. Attached Figure Description
[0071] Figure 1 is a schematic diagram of the architecture of a database system provided by related technologies;
[0072] Figure 2 is a flowchart illustrating a data processing method provided by related technologies;
[0073] Figure 3 is a schematic diagram of the hardware architecture of a data processing system provided in an embodiment of this application;
[0074] Figure 4 is a schematic diagram of the software architecture of a data processing system provided in an embodiment of this application;
[0075] Figure 5 is a schematic diagram of the software architecture of another data processing system provided in an embodiment of this application;
[0076] Figure 6 is a schematic diagram of the hardware structure of a server provided in an embodiment of this application;
[0077] Figure 7 is a flowchart illustrating a data processing method provided in an embodiment of this application;
[0078] Figure 8 is a schematic diagram of a B-tree index provided in an embodiment of this application;
[0079] Figure 9 is a flowchart illustrating a method for determining routing information of a target data table according to an embodiment of this application.
[0080] Figure 10 is a schematic diagram of an index analysis diagram provided in an embodiment of this application;
[0081] Figure 11 is a schematic diagram of another index analysis diagram provided in an embodiment of this application;
[0082] Figure 12 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;
[0083] Figure 13 is a schematic diagram of another data processing device provided in an embodiment of this application. Detailed Implementation
[0084] In the description of this application, unless otherwise stated, "multiple" means two or more. At least one of the following or similar expressions refer to any combination of these terms, including any combination of single or plural terms. For example, at least one of a, b, and / or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0085] Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.
[0086] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.
[0087] It is understood that the term "embodiment" used throughout the specification means that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, throughout the specification, various embodiments do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0088] It is understood that some optional features in the embodiments of this application can be implemented independently in certain scenarios without relying on other features, such as the current solution on which they are based, to solve the corresponding technical problems and achieve the corresponding effects. Alternatively, they can be combined with other features as needed in certain scenarios. Correspondingly, the apparatus given in the embodiments of this application can also implement these features or functions, which will not be elaborated here.
[0089] In this application, unless otherwise specified, the same or similar parts between the various embodiments can be referred to each other. In the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments are consistent and can be mutually referenced. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. The following embodiments of this application do not constitute a limitation on the scope of protection of this application.
[0090] The following provides an exemplary description of the application scenarios of the embodiments of this application.
[0091] A distributed database based on shared storage is a distributed database architecture. In a distributed database based on shared storage, all nodes (such as multiple database servers) can share data in the same storage pool. For example, any one of the multiple database servers can access and manipulate the same data stored on shared storage, thereby achieving high availability, high performance, and load balancing.
[0092] In the field of distributed databases, due to the differences in the evolution cycles and demand change cycles of computing and storage hardware, the overall architecture of distributed databases is evolving from "integrated storage and computing" to "separate storage and computing" in order to achieve flexible hardware deployment and configuration and reduce the overhead of repeated data migration and copying. This means adopting an architecture design that separates data storage and computation, thereby decoupling the computing and storage sides. This separate storage and computing architecture brings advantages such as high performance, high flexibility, scalability, high availability, and high utilization to distributed databases. The following section details these advantages.
[0093] (1) High performance. In a compute-storage separation architecture, the compute layer is responsible for handling tasks such as querying, analysis and computation, while the storage layer focuses on data storage and management.
[0094] On the one hand, this separation of storage and computation allows the computation layer and storage layer to be optimized independently. For example, the computation layer is no longer responsible for persisting any state, allowing it to allocate more resources to querying, analysis, and computation, thereby achieving higher performance. Data persistence refers to the process of transferring data from temporary storage (such as memory) to permanent storage (such as hard drives, distributed storage systems, etc.).
[0095] On the other hand, this storage-compute separation characteristic allows for global sharing of data, avoiding unnecessary overhead and thus improving overall database performance. For example, the storage-compute separation architecture eliminates the need for semi-synchronous replication, thus avoiding the performance degradation that occurs when adding slave nodes. Semi-synchronous replication is a common mechanism in database master-slave replication, requiring the master node to wait for at least one slave node to confirm receipt and synchronize the transaction log before committing a transaction. Adding slave nodes increases network communication complexity, potentially leading to increased network latency, which prolongs the time the master node waits for slave node confirmation, thereby impacting performance.
[0096] (2) High flexibility. Due to the separation of the computing layer and the storage layer, business applications can flexibly adjust the configuration of computing and storage resources according to actual needs to achieve the best performance and cost-effectiveness.
[0097] (3) Scalability. As the amount of data continues to grow, the system's processing capacity can be independently expanded by adding storage nodes and computing nodes in the storage-compute separation architecture, thereby meeting the ever-increasing data processing needs.
[0098] (4) High Availability. External storage arrays inherently possess excellent reliability design features, such as redundant arrays of independent disks (RAID), silent data verification, and two-site three-center architecture, ensuring reliability and availability. Compared to database servers, the reliability level of external storage arrays is at least two orders of magnitude higher. Therefore, in a storage-compute separation architecture, the use of long-term shared storage allows for rapid business recovery without data replication, even if the database server fails, due to the globally shared data, thus achieving overall high availability of the distributed database.
[0099] (5) High utilization rate. In the architecture of storage and computing separation, resource utilization can be improved by building computing resource pools and storage resource pools separately.
[0100] In the aforementioned storage-compute separation architecture, the compute layer refers to the part of the distributed database responsible for handling tasks such as querying, analysis, and computation. In a distributed database, the compute layer typically consists of multiple compute nodes, such as multiple database servers. For example, a database server can be used to parse business requests, query the data tables requested by the business request, and process the data requested by the business request.
[0101] The storage layer refers to the part of a distributed database responsible for data storage and management. In a distributed database, the storage layer typically consists of multiple storage nodes, such as multiple storage servers. It is understood that a database composed of multiple storage servers can form the aforementioned shared storage. For example, storage servers can store data in the form of data tables. A data table is a logical structure used to store structured data, consisting of rows and columns, where each row represents a data record and each column represents a data attribute.
[0102] For example, as shown in Table 1, taking a user table as an example, the user table can include multiple rows, where each row represents a user's data record, such as user identification (ID), name, gender, age, etc. Each column represents a data attribute, such as a user ID column, name column, gender column, age column, etc.
[0103] Table 1
[0104] However, in distributed databases that include multiple database servers, the same business request may be randomly distributed to different database servers. In this case, multiple database servers need to access the same data in the shared storage, which may result in data being frequently transferred from one database server to another, leading to data access conflicts.
[0105] This involves the frequent transfer of data from one database server to another, which can also be understood as the frequent transfer of data pages from one database server to another. A data page is the basic unit for storing data in a distributed database; it is typically a fixed-size storage block used to divide and store data from a data table. The embodiments of this application will subsequently use data as the object of transfer as an example to illustrate the scheme.
[0106] For example, when a database server receives a business request, if the data required by the request is not on the local database server, the database server will query other database servers to see if they have the data. If they do, the database server will load the data page from the other database servers to obtain the data, at which point the data will be transferred from the other database servers to the local database server. If the other database servers also do not have the data, the database server will directly load the data page from shared storage to obtain the data.
[0107] Therefore, if the same business request is randomly distributed to different database servers, data will be frequently transferred from one database server to another according to the needs of the business request. For example, for a business request that requires multiple accesses to shared storage, each access request may be distributed to a different database server. For a particular access request, the data required by that request may not be on the current database server; for example, it may be on a different database server that processed the request in a previous access. In this case, the data page needs to be loaded from that other database server to obtain the data. It should be understood that the time when the business request arrives at the database server varies with each access. Therefore, the database server that receives the request earlier will retrieve the data from the shared storage first, while the database server that receives the request later needs to retrieve the data from the database server that received the request earlier. This leads to the phenomenon of data being frequently transferred from one database server to another.
[0108] Frequent data transfer between database servers increases communication overhead, impacting response latency for business requests and overall system throughput. In the worst-case scenario, the same business request might be distributed to different database servers, causing "hot data" to be passed around like a ping-pong ball, becoming a performance bottleneck. Hot data refers to data frequently accessed within a given period. If business requests related to this hot data are distributed to different database servers, this data may be frequently transferred between them, impacting system performance. The "ping-pong" effect refers to the frequent back-and-forth transmission of data between database servers, similar to passing a ping-pong ball between two players. For example, suppose a business request involves updating data. When one database server updates certain data, the updated data needs to be synchronized to other database servers to ensure consistency. Therefore, if the same business request is randomly distributed to different database servers, multiple database servers will frequently update the same data, and each update requires data transfer to other database servers for synchronization, resulting in the ping-pong phenomenon.
[0109] Therefore, how to reduce the frequent transfer of data between multiple database servers is a technical problem that urgently needs to be solved.
[0110] For example, Figure 1 is a schematic diagram of the architecture of a database system provided by related technologies. Referring to Figure 1, the database system includes: a database, two or more database instances based on shared storage, and a global enqueue service, a global resource directory, and a global cache service.
[0111] In this context, a database instance refers to a running instance deployed on a database server, used to manage the storage, access, and operations of the database. A single database server can run one or more database instances. Referring to Figure 1, and taking database instance 1 and database instance 2 as examples, database instance 1 and database instance 2 can be deployed on the same database server or on two different database servers. This embodiment will subsequently use deployment on two different database servers as an example to illustrate the architecture of the database system.
[0112] Referring to Figure 1, the database can be shared storage, allowing database instance 1 and database instance 2 to share data within the database. In some embodiments, the database can store data in the form of data tables.
[0113] Referring to Figure 1, the database instances are interconnected. Each database instance includes a learning management system (LMS) that partitions data based on hashes, data blocks, queues, locks (LCK), a lock monitor (LMON), and a lock monitor daemon (LMD) process.
[0114] A storage management system based on hash partitioning is used to configure hash partitioning algorithms for each data table in the database, dividing the data in each table into multiple partitions according to the hash partitioning algorithm. The hash partitioning algorithm is a technique that uses a hash function to evenly distribute data across multiple partitions. Partitioning is a technique for dividing the data in a large table into multiple smaller, more manageable parts. Understandably, the multiple partitions (such as multiple data partitions) can be stored, managed, and queried independently, thereby improving the performance and scalability of the database system.
[0115] A data block is the smallest unit of data storage in a database instance, such as storing information about a data table. Queues are used to store messages or requests to be processed, such as business requests. By setting up queues, it is possible to ensure that the database system can smoothly handle high-concurrency requests and avoid overload caused by sudden traffic surges. Locks are used to control access to data resources in shared storage, preventing multiple processes or threads from simultaneously writing to the same data, ensuring data consistency and integrity. Lock monitors the lock status in the database system, ensuring the stability and performance of the database system. The lock management daemon manages the lock status in the database system. The global queue service manages queues. The global resource management service is responsible for managing the resource allocation and access control of the database system to ensure the rational use of data resources and the stability of the database system. The global cache service manages data blocks from multiple database instances.
[0116] When a business request is triggered, the processing flow of the database system as shown in Figure 1 can be as follows: First, the business request enters the queue of the database instance through the upper layer. The upper layer refers to the layer above the database server, such as the business server or middleware of the business application, such as a web server or application programming interface gateway (API gateway). Then, after retrieving the business request from the queue, the database instance sends an authorization request to the global resource management service to request authorization for data access. Next, the global resource management service decides whether to authorize the business request to continue execution based on factors such as the current database system load, resource usage, and the priority of the business request. If the business request is not authorized, it may be rejected or placed in a waiting queue, awaiting subsequent resource release. If the business request is authorized, the database system will continue to process it. For example, after obtaining authorization, the database instance can access the global cache service based on the business request to continue processing it. For instance, the database instance can send the business request to the global cache service. This business request can be a read request or a write request. Specifically, if the business request is a read request, the global caching service checks if the requested data exists in the cache. If it exists, the cached data is returned directly; otherwise, the data is retrieved from the database and the cache is updated. If the business request is a write request, the global caching service updates the cached data to ensure data consistency. Finally, after processing the business request, the global caching service returns the result to the upper layer, and ultimately to the user.
[0117] To reduce the frequent transfer of data between multiple database servers, the data processing method shown in Figure 2 can be adopted in the database system shown in Figure 1. Figure 2 is a flowchart illustrating a data processing method provided by related technologies. Referring to Figure 2, the flow of the data processing method may include S201 to S203.
[0118] S201. In a database system, the data in each data table in the database is divided into multiple partitions according to the hash partitioning algorithm.
[0119] In this system, a data table can correspond to multiple partitions, and a partition can correspond to a portion of the data in the data table. The database system maintains a mapping relationship between partitions and hash values, as well as a mapping relationship between partitions and resource management owners. For example, one partition corresponds to one hash value, and one partition corresponds to one resource management owner. The resource management owner is the database server used to manage the portion of data corresponding to the partition.
[0120] S202. When a business request is triggered, the database server to be used to execute the business request is determined according to the hash partitioning algorithm.
[0121] In some embodiments, a hash value can be calculated based on the page information of the data requested by the business request according to a hash partitioning algorithm, and the resource control owner of the partition corresponding to the hash value can be determined as the database server used to execute the business request.
[0122] The page information of the data requested by the business request can be the page offset or page number of the data page.
[0123] Understandably, in database systems, there is a one-to-one mapping between hash values and partitions. Therefore, by calculating a hash value, the partition corresponding to that hash value can be determined. Furthermore, in database systems, there is also a one-to-one mapping between partitions and resource control owners, thus allowing the determination of the resource control owner of a partition.
[0124] S203. Execute business requests using the determined database server.
[0125] In some embodiments, a business request may be sent to a determined database server to query the data requested by the business request and to process the data requested by the business request.
[0126] In the technical solution shown in Figure 2 above, by dividing the data in each data table of the database into multiple partitions, and since each partition corresponds to a resource management owner (i.e., a database server), it means that data belonging to the same partition is managed by the database server corresponding to that partition. Therefore, for business requests involving data belonging to the same partition, the request will be distributed to the database server corresponding to that partition, rather than being randomly distributed to different database servers. This avoids the phenomenon of multiple database servers accessing the same data in the database, and while ensuring cache consistency, it reduces the frequent transfer of data between multiple database servers, thereby reducing business conflicts in some scenarios.
[0127] However, the partitioning methods described above rely on a hash partitioning algorithm pre-configured by the database system. The number of partitions created by this algorithm is fixed and difficult to adjust dynamically, resulting in poor flexibility. Furthermore, hash partitioning randomly distributes data across different partitions, meaning the storage order of the data is independent of the original data order. Since data within the same table is continuous, hash partitioning can cause some consecutive data to be scattered across different partitions. Therefore, when sequential access to large amounts of data in the same table is required, it may be necessary to read data across multiple partitions, increasing the complexity and overhead of read / write operations and limiting performance when performing sequential access to large amounts of data in the same table. In addition, the partitioning methods described above do not consider business access patterns, such as frequently accessed hot data, leading to hot data being randomly distributed across multiple partitions. This may affect the processing efficiency of business requests, resulting in poor overall database system performance.
[0128] In view of this, embodiments of this application provide a data processing method in which a business server receives a business request and sends a target query statement from the business request to any one of a plurality of database servers included in the database system. When a database server receives the target query statement from the business server, it sends routing information of the target data table to the business server based on the information of the target data table carried in the target query statement. Furthermore, after receiving the routing information of the target data table sent by the database server, the business server can determine a database server to process the business request based on the routing information of the target data table. In this way, a business request can be processed by a single database server; that is, a business request will be distributed to its corresponding database server, rather than being randomly distributed to different database servers. This avoids the situation where multiple database servers access the same data, and also prevents data from being frequently transferred from one database server to another, thereby reducing data access conflicts.
[0129] The routing information for the target data table includes multiple numerical ranges of the table's indexes and the corresponding database server information for each range, with one numerical range corresponding to one database server. This ensures that when determining the database server to handle a business request, requests for data belonging to the same numerical range will be distributed to the database server corresponding to that range, rather than being randomly distributed to different database servers. This avoids situations where multiple database servers access the same data, preventing data from frequently being transferred from one database server to another, thus reducing data access conflicts.
[0130] Compared with the technical solutions provided by related technologies, the embodiments of this application reference multiple numerical ranges of the target data table's index, with each numerical range corresponding to one database server. Thus, on the one hand, the number of database servers is considered when partitioning, allowing the number of partitions to be dynamically adjusted according to the number of database servers, providing good flexibility. On the other hand, referencing the target data table's index ensures that the resulting multiple numerical ranges still conform to the original data order. Therefore, when sequential access to a large amount of data in the same data table is required, it is unnecessary to read data across multiple partitions, reducing the complexity and overhead of read / write operations and improving performance when sequentially accessing large amounts of data in the same data table.
[0131] The data processing method provided in this application can be applied to distributed database scenarios based on shared storage. Of course, in other embodiments, the data processing method provided in this application can also be applied to other scenarios based on distributed computing frameworks. It is understood that, when the data source of the distributed computing framework has an index, the data processing method provided in this application can be used to partition the data according to the index to obtain multiple numerical intervals, then generate routing information including multiple numerical intervals and the corresponding database servers for each numerical interval. Furthermore, business requests for the data table can be distributed based on the routing information of the data table, which can reduce the frequency of data transfer between different database servers, reduce waiting time and network overhead, and improve the overall performance of the database system. Here, the data source is a basic component in the distributed computing framework used to provide data for computation, such as a distributed database.
[0132] To facilitate understanding of the embodiments of this application, the following points will be explained before introducing the embodiments of this application.
[0133] 1. In the embodiments of this application, "instruction" can include direct instruction and indirect instruction, as well as explicit instruction and implicit instruction. The information indicated by a certain piece of information is called the information to be instructed. In specific implementation, there are many ways to indicate the information to be instructed, such as, but not limited to, directly indicating the information to be instructed, such as the information to be instructed itself or its index. It can also indirectly indicate the information to be instructed by indicating other information, where there is a relationship between the other information and the information to be instructed. It can also indicate only a part of the information to be instructed, while the other parts of the information to be indicated are known or pre-agreed.
[0134] 2. In the embodiments of this application, the descriptions such as "in the case of", "if", and "if" all refer to the fact that the device (e.g., the database server or the business server) will take corresponding actions under certain objective circumstances. They are not time limits, nor do they require the device (e.g., the database server or the business server) to have a judgment action when implementing it, nor do they mean that there are other limitations.
[0135] Furthermore, the system architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0136] Furthermore, the actions, terms, etc., involved in the various embodiments of this application can be referenced interchangeably without limitation. The message names or parameter names in the messages exchanged between the various devices in the embodiments of this application are merely examples, and other names may be used in specific implementations without limitation.
[0137] The system architecture of the embodiments of this application will be described below as an example.
[0138] In some embodiments, the data processing method provided in this application can be applied to the data processing system shown in FIG3. Exemplarily, FIG3 is a schematic diagram of the hardware architecture of a data processing system provided in an embodiment of this application. Referring to FIG3, the data processing system includes: a business client 301, a business server 302, a database server 303, and a database 304.
[0139] The business client 301 runs a business application. This business application refers to a software application designed and developed to meet specific business needs. In some embodiments, the business application may be an e-commerce application, a social media application, a mobile office application, etc., but this application embodiment does not limit this.
[0140] Understandably, the business client 301 is deployed on a terminal device to provide a rich user interface. In some embodiments, the terminal device can be at least one of the following: smartphone, smartwatch, printer, tablet computer, PDA, desktop computer, laptop, in-vehicle terminal, wearable electronic device, virtual reality terminal, augmented reality terminal, wireless terminal, and laptop computer. This application does not limit the terminal device.
[0141] In this embodiment of the application, the business client 301 is used to: receive a trigger operation from a user, and in response to the user's trigger operation, send a business request to the business server 302. The business request is used to request processing of at least one data table, specifically, to request processing of data in at least one data table.
[0142] The business request can be a Structured Query Language (SQL) request. SQL is a declarative programming language used to manage relational database systems, allowing users to perform various database operations in a concise way, such as querying, inserting, updating, and deleting data, as well as creating, modifying, and deleting database objects (such as tables, indexes, and views). For example, a business request can be a request to read, write, modify, or update data in a data table; this embodiment of the application does not limit this.
[0143] The business server 302 is a server used to run various business logics, primarily responsible for handling a series of business operations generated by business applications. For example, the business server 302 can also be referred to as a business server. In some embodiments, the business client 301 communicates with the business server 302. Referring to Figure 3, there can be multiple business servers 302.
[0144] Database server 303 is a server responsible for handling tasks such as querying, analysis, and calculation, primarily providing data management functions such as data querying and updating for business applications. In some embodiments, business server 302 is communicatively connected to database server 303. Referring to Figure 3, there can be multiple database servers 303.
[0145] Database 304 is used for data storage and management. In this embodiment, the database can be shared storage, allowing multiple database servers to share data within the database. In some embodiments, database 304 may consist of multiple storage servers.
[0146] Understandably, database server 303 and database 304 can constitute a database system, such as a database management system (DBMS).
[0147] The aforementioned business server 302 and database server 303 can be independent physical servers, or server clusters or distributed file systems composed of multiple physical servers, or at least one of cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data or artificial intelligence platforms. This application embodiment does not limit this.
[0148] In this embodiment, the service server 302 is configured to: receive a service request; send the target query statement in the service request to any one of the multiple database servers 303 included in the database system to request the routing information of the target data table; receive the routing information of the target data table sent by the database server 303; and determine a database server to process the service request based on the routing information of the target data table.
[0149] In this embodiment of the application, the database server 303 is used to: receive a target query statement from the business server 302; and send routing information of the target data table to the business server 302 based on the target query statement.
[0150] In the system architecture shown in Figure 3, when database server 303 receives the target query statement from business server 302, it obtains the routing information of the target data table based on the query statement and sends this information to business server 302. This allows business server 302 to determine the database server 303 to handle the business request based on the routing information. This ensures that one business request can be handled by one database server 303; that is, a business request will be distributed to its corresponding database server 303, rather than being randomly distributed to different database servers 303. This avoids the situation where multiple database servers 303 access the same data, and also prevents data from frequently being transferred from one database server 303 to another, thereby reducing data access conflicts.
[0151] Figure 3 above describes the data processing system from a hardware architecture perspective. The following describes the data processing system from a software architecture perspective. Figure 4 is a schematic diagram of the software architecture of a data processing system provided in an embodiment of this application. Referring to Figure 4, the data processing system includes: a business application 401, a database driver 402, a database instance 403, and shared storage management 404.
[0152] Among them, business application 401 is a software application deployed on business client 301.
[0153] Database driver 402 is a software component deployed on business server 302, used to connect business application 401 and database instance 403, acting as a bridge between business application 401 and database instance 403. In some embodiments, database driver 402 is responsible for tasks such as handling connections to the database instance, executing business requests, and obtaining query results. One or more database drivers 402 may be deployed on a single business server 302.
[0154] In some possible implementations, the database driver 402 is used to convert business requests from the business application 401 into a format that the database instance 403 can understand, and send the converted business requests to the database instance 403 for parsing, execution, and return of processing results. The database driver 402 is also used to receive the processing results returned by the database instance 403, convert the processing results returned by the database instance 403 into a format that the business application 401 can understand, and return them to the business application 401.
[0155] In some embodiments, the database driver 402 may be an Open Database Connectivity (ODBC) driver, a Java Database Connectivity (JDBC) driver, or other types of drivers, and this application embodiment does not limit this. The ODBC driver provides a general database connection API, allowing business applications 401 to access different database instances through a general API, and can be widely used for database access in various programming languages. The JDBC driver provides a Java-based database connection API, allowing Java-based business applications 401 to access database instances through the Java-based database connection API, and can be used for database access in Java.
[0156] Database instance 403 is a running instance deployed on database server 303, responsible for handling tasks such as querying, analysis, and calculation. In some embodiments, one or more database instances 403 may be deployed on a single database server 303; this application embodiment does not limit this. Referring to Figure 4, the number of database instances 403 can be multiple, such as database instance 1, database instance 2, database instance 3, etc. Database instance 1, database instance 2, and database instance 3 can be deployed on the same database server or on three different database servers. This application embodiment will subsequently describe the data processing system using deployment on three different database servers as an example.
[0157] In this embodiment of the application, the business application 401 can send a business request to the database instance 403 through the database driver 402.
[0158] Shared storage management 404 is a software application deployed on database 304. Referring to Figure 4, shared storage management 404 includes log files (such as redo log files) and data files. The log files may include business access records, which are data records of business request accesses obtained based on historical business request statistics. The data files may include multiple data tables and related information for each table, such as index information.
[0159] Understandably, multiple database instances can access shared storage management (404) even with a 403 error.
[0160] In database instance 403 and shared storage management 404, data pages can be used as the basic unit for storing data. In some embodiments, database instance 403 can maintain cached data pages. For example, taking database instance 1 shown in Figure 4 as an example, it can maintain data pages such as P1, P11, P21, and P31; taking database instance 2 shown in Figure 4 as an example, it can maintain data pages such as P2, P12, P22, and P32; and taking database instance 3 shown in Figure 4 as an example, it can maintain data pages such as P3, P13, P23, and P33. When business application 401 sends a business request to database instance 403 through database driver 402, database instance 403 can preferentially load data pages from the cache to read, write, modify, or update data.
[0161] The database driver 402 and database instance 403 shown in Figure 4 will be described in detail below based on Figure 5. Figure 5 is a schematic diagram of the software architecture of another data processing system provided in an embodiment of this application.
[0162] Referring to Figure 5, the database driver 402 includes a client connection management module 501, a request sending and result reading module 502, and a database instance routing management module 503.
[0163] The client connection management module 501 is responsible for establishing a connection between the business application 401 and the database instance 403. The request sending and result reading module 502 is responsible for receiving business requests (such as SQL requests) from the business application 401 and sending them to one of the multiple database instances 403. The request sending and result reading module 502 is also responsible for waiting for the processing result from the database instance 403 and returning the result to the business application 401. The database instance routing management module 503 is responsible for determining the database instance to execute the business request. In this embodiment, the database instance routing management module 503 receives routing information of the data table sent by the database instance 403 and determines the database instance to execute the business request based on the routing information of the data table.
[0164] Referring to Figure 5, each of the multiple database instances 403 includes a server connection management module 504, an SQL engine 505, system tables 506, inter-node communication 507, and a storage engine 508. The SQL engine 505 includes an SQL parsing / optimization / execution module 5051 and a routing management module 5052. The storage engine 508 includes an index 5081 and a page buffer 5082.
[0165] The system comprises the following components: Server Connection Management 504, responsible for establishing a connection with the business application 401 via the database driver 402; SQL Parsing / Optimization / Execution Module 5051, responsible for parsing, optimizing, and executing SQL requests; Routing Management Module 5052, responsible for generating routing information for data tables and returning this information to the database driver 402 upon receiving the target query statement in a business request; Inter-node Communication 507, responsible for communicating with other database instances; Index 5081, a data structure used to improve database query efficiency; and Page Buffer 5082, used to determine the current database server where the data page resides based on the access needs of the business request and to transfer the data page from other database servers to the current database server.
[0166] In this embodiment, a new database instance routing management module 503 is added to the database driver 402, and a new routing management module 5052 is added to the SQL engine 505 of the database instance 403. The routing management module 5052 includes a routing algorithm layer. This algorithm accesses data tables, index information, and business access records in the database to periodically calculate and update the routing information for each data table. When a target query statement is received in a business request, the routing information of the corresponding target data table is fed back to the database driver 402 for caching. This allows the database driver 402, such as the database instance routing management module 503, to select the corresponding database server to execute the business request based on the routing information.
[0167] Specifically, database driver 402 may contain first program code that runs on the hardware of the business server to implement the processes described above: sending business requests, receiving routing information, and determining the database server to execute the business request based on the routing information. Second program code may be deployed in the SQL engine 505 of database instance 403. This second program code runs on the hardware of the database server to implement the processes described above: receiving target query statements, generating routing information, and sending routing information.
[0168] In one example of this application, a schematic diagram of the server's hardware structure is shown in Figure 6. Figure 6 is a schematic diagram of the hardware structure of a server provided in an embodiment of this application. Referring to Figure 6, the server shown in Figure 6 may include: a processor 601, a memory 602, a communication module 603, and a bus 604. The processor 601, the memory 602, and the communication module 603 can be connected via the bus 604.
[0169] The processor 601 is the control center of the server and can be a general-purpose central processing unit such as a CPU, or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor. In this embodiment, taking a business server as an example, the processor 601 in the business server can execute the process of determining the database server to which the business request to be executed is located based on routing information. Taking a database server as an example, the processor 601 in the database server can execute the process of determining the routing information for each data table.
[0170] As an example, processor 601 may include one or more CPUs, such as CPU0 and CPU1 shown in Figure 6.
[0171] The memory 602 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.
[0172] In one possible implementation, the memory 602 may exist independently of the processor 601. The memory 602 can be connected to the processor 601 via a bus 604 and is used to store data, instructions, or program code. When the processor 601 calls and executes the instructions or program code stored in the memory 602, it can implement the data processing method provided in the embodiments of this application.
[0173] In another possible implementation, the memory 602 can also be integrated with the processor 601.
[0174] The communication module 603 is used for connecting the server to other devices via a communication network, which can be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication module 603 may include a receiving unit for receiving data and a transmitting unit for transmitting data.
[0175] Bus 604 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 6, but this does not indicate that there is only one bus or one type of bus.
[0176] It should be noted that the structure shown in Figure 6 does not constitute a limitation on the server. In addition to the components shown in Figure 6, the server may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0177] For ease of understanding, the data processing method provided in the embodiments of this application will be described by way of example below with reference to the above system architecture and the accompanying drawings.
[0178] It is understood that in the embodiments of this application, the database server or business server may execute some or all of the steps in the embodiments of this application. These steps or operations are merely examples, and the embodiments of this application may also execute other operations or variations of various operations.
[0179] Figure 7 is a flowchart illustrating a data processing method provided in an embodiment of this application. In some possible implementations, this data processing method can be completed by the database server and the business server in the above system architecture. Referring to Figure 7, taking the interaction flow between the database server and the business server as an example, the method includes the following steps S701 to S706.
[0180] S701, The business server receives business requests.
[0181] The business request is used to request processing of at least one data table, specifically, to request processing of data in at least one data table. For example, the business request may be a request to perform operations such as reading, writing, modifying, or updating data in the at least one data table; this embodiment does not limit this. In some possible implementations, the at least one data table may be a user table, employee table, department table, order table, or other types of data table; this embodiment does not limit this.
[0182] In some embodiments, a service request may be triggered by a service client. For example, the service client receives a trigger operation from a user and, in response to the user's trigger operation, sends a service request to the service server. The service server receives the service request from the service client and executes the process in S702.
[0183] S702. The business server sends the target query statement to any one of the multiple database servers included in the database system.
[0184] The business request may include at least one query statement (such as an SQL statement). The target query statement is one of the at least one query statement. For example, if the business request includes multiple query statements, the target query statement may be the first query statement among the multiple query statements. The first query statement refers to the first query statement listed in the order of the multiple query statements. Of course, in other possible implementations, the target query statement may also be any other query statement among the multiple query statements.
[0185] In this embodiment, the target query statement carries information about the target data table. It is understood that the target data table refers to the data table requested to be accessed by the target query statement. The target data table is one of the at least one data table mentioned above. In some embodiments, the information of the target data table may be, for example, the name of the target data table, the number of the target data table, or the object identifier (OID) of the target data table; this embodiment does not limit this.
[0186] Specifically, a target query statement may include one or more predicate conditions. These predicate conditions are conditional expressions used in the target query statement to filter data, specifying rules for filtering data in the target data table to retrieve data that meets specific conditions.
[0187] Understandably, in complex business scenarios, different predicate conditions may correspond to different target data tables. This means that the target query statement can carry information from one or more target data tables.
[0188] For example, suppose the target query statement includes predicate condition 1: SELECT * FROM employees WHERE department_id = ×, which means querying data in the employees table (employee table) where department_id = × (department is ×). It can be seen that the target data table corresponding to predicate condition 1 is the employee table.
[0189] For example, suppose the target query statement includes predicate condition 2: SELECT * FROM orders WHERE status = ×, which means querying data in the orders table (orders table) where status = × (status is ×). It can be seen that the target data table corresponding to predicate condition 2 is the orders table.
[0190] For example, suppose the target query statement includes predicate condition 3: SELECT * FROM users WHERE age=×, which means querying data in the users table (user table) where age=× (age is ×). It can be seen that the target data table corresponding to predicate condition 3 is the users table.
[0191] In S702 above, the service server sends the target query statement from the service request to any one of the multiple database servers to request routing information for the target data table specified in the target query statement from that database server. In this embodiment, the routing information is used to determine a database server that processes at least one data table requested by the service request, so as to successfully complete the execution of the service request. Specifically, by sending the target query statement (such as the first query statement) from among the multiple query statements included in the service request, it is ensured that a service request corresponds to obtaining routing information for the target data table specified in one query statement, laying the foundation for subsequently ensuring that a service request is executed by a single database server.
[0192] It is worth noting that S701 to S702 described above can be executed by the database driver in the business server.
[0193] S703, The database server receives the target query statement from the business server.
[0194] It is worth noting that the above S703 can be executed by a database instance in a database server.
[0195] S704. The database server sends routing information for the target data table to the business server based on the target query statement.
[0196] In this embodiment, the routing information of the target data table includes multiple numerical ranges of the target data table's index and information about the database server corresponding to each numerical range. In some possible implementations, the routing information can be in the form of a routing table. Of course, in other possible implementations, the routing information can also be in other forms, and this embodiment does not limit this.
[0197] An index for the target data table is a data structure used to optimize data retrieval. Think of it like a table of contents in a book; it establishes fast query paths for data in the target table, improving search efficiency.
[0198] An index for a target data table is created based on the values of an index column in the target data table. An index column is a column (or a set of columns) in the target data table used to create the index. For example, an index column can be a primary key column, a foreign key column, or other columns. The primary key column is a column (or a set of columns) in the target data table, and the values in the primary key column uniquely identify each row of data in the target data table. For example, in the user table of an electronic order system, the primary key column of the user table could be the user ID column. Similarly, in the order table of an electronic order system, the primary key column of the order table could be the order ID column. A foreign key column is a column (or a set of columns) in the target data table, and the values in the foreign key column are used to reference the values of primary key columns in other data tables, representing the relationships between the data tables. For example, in the user table and order table of an electronic order system, the order table could include a user ID column, which is also a foreign key column in the order table, used to reference the primary key column, i.e., the user ID column, in some embodiments. In some cases, the primary key column can be created automatically, while the foreign key column can be created manually. It should be understood that the selection of index columns can be set according to actual business needs.
[0199] For example, the index of the target data table can be a B-tree index. B-tree indexes are a commonly used index type in database systems, suitable for scenarios handling large amounts of data, and can support fast data retrieval, sequential access, insertion, and deletion operations. A B-tree index is a multi-level index structure, specifically stored hierarchically according to the values of the indexed columns, typically divided into root nodes, intermediate nodes, and leaf nodes. Each node contains a certain number of key-value pairs, where the key is the value of the indexed column, and the value is a pointer to a data row. Furthermore, a B-tree index is a self-balancing tree data structure, capable of automatically adjusting its structure after insertion, deletion, and other operations to maintain tree balance. It is worth noting that the index can also be other types of indexes, and this application embodiment does not limit this. The following embodiments of this application will use a B-tree index as an example to illustrate the scheme.
[0200] For example, taking the user table shown in Table 1 above as the target data table, assuming that the age column (22, 25, 23, 30, 27) is used as the index column, a B-tree index of order 3 is created. Here, order 3 means that each node of the B-tree index can have a maximum of 3 child nodes, and each node can store a maximum of 2 key values. The resulting B-tree index can be shown in Figure 8. Figure 8 is a schematic diagram of the structure of a B-tree index provided in an embodiment of this application. Referring to Figure 8, the root node of the B-tree index includes key values 23 and 27, indicating that key values less than 23 are stored in the left subtree (e.g., key value 22), key values greater than 23 and less than 27 are stored in the middle subtree (e.g., key value 25), and key values greater than 27 are stored in the right subtree (e.g., key value 30). Figure 8 illustrates the structure of a B-tree index using a two-level B-tree index as an example. In other embodiments, the B-tree index may also have a three-level or higher structure. Here, the key value is the value of the index column in the target data table.
[0201] Multiple numerical intervals are obtained by dividing the numerical distribution range of the index of the target data table. For example, taking the B-tree index shown in Figure 8 as an example, assuming the numerical distribution range of this B-tree index is divided into 3 numerical intervals, it can be divided into intervals with values less than 23, intervals with values greater than 23 and less than 27, and intervals with values greater than 27. Assuming the numerical distribution range of this B-tree index is divided into 5 numerical intervals, it can be divided into intervals with values less than 20, intervals with values greater than 20 and less than 23, intervals with values greater than 23 and less than 27, intervals with values greater than 27 and less than 30, and intervals with values greater than 30.
[0202] In this embodiment, a numerical range corresponds to a database server. The information of the database server may be the name, number, or ID of the database server; this embodiment does not limit this information.
[0203] Furthermore, routing information may also include version information, such as a version number, to indicate the version of the routing information.
[0204] For example, taking the target data table as the user table shown in Table 1 above, and the index of the target data table as the B-tree index shown in Figure 8 above, assuming that the numerical distribution range of the B-tree index is divided into 5 numerical intervals, the routing information of the target data table can be provided as a routing table as shown in Table 2.
[0205] Table 2
[0206] In some embodiments, the database server obtains the routing information of the target data table based on the information of the target data table carried in the target query statement, and then sends the routing information of the target data table to the business server.
[0207] In some possible implementations, the database can be used to store routing information for all data tables included in the database system, such as storing information about multiple data tables (e.g., OIDs) and the routing information corresponding to each data table. Accordingly, the database server can obtain the routing information of a target data table from the database based on the information of the target data table. Then, the database server sends the obtained routing information of the target data table to the business server.
[0208] In this embodiment of the application, the database supports access from multiple database servers (that is, a shared storage is formed). Therefore, any one of the multiple database servers can access the database to obtain the routing information of any data table (such as the target data table).
[0209] As described above, a target query statement can carry information about one or more target data tables. For example, if the target query statement includes a single predicate condition, that predicate condition may correspond to one target data table. Conversely, if the target query statement includes multiple predicate conditions that correspond to different target data tables, these multiple predicate conditions may correspond to multiple target data tables. The following sections explain the process of obtaining the routing information for the target data tables in both cases: when the target query statement carries information about one target data table and when it carries information about multiple target data tables.
[0210] Scenario 1: When the target query statement includes a predicate condition, the database server can determine whether the target data table exists in the database based on the information of the target data table corresponding to this predicate condition. If the target data table exists in the database, the routing information corresponding to the target data table is obtained, that is, the routing information for the target data table is obtained.
[0211] For example, assuming the target data table information corresponding to this predicate condition is the OID of the target data table, we can determine whether the OID exists in the database based on the OID of the target data table. If the OID exists in the database, we can obtain the routing information corresponding to the OID, that is, obtain the routing information of the target data table.
[0212] In scenario one above, if the target query statement includes a predicate condition, the process of retrieving the routing information for the target data table can be executed directly. This ensures that one business request corresponds to the retrieval of routing information for one target data table, thus laying the foundation for ensuring that subsequent business requests correspond to one database server.
[0213] Scenario 2: When the target query statement includes multiple predicate conditions that correspond to different target data tables, the database server can select a target predicate condition from the multiple predicate conditions in the target query statement. Based on the information of the target data table corresponding to the target predicate condition, it can determine whether the target data table exists in the database. If the target data table exists in the database, the routing information corresponding to the target data table is obtained, i.e., the routing information for the target data table is obtained.
[0214] The target predicate condition is one of the multiple predicate conditions, such as the first predicate condition. The first predicate condition refers to the first predicate condition listed in the order of the multiple predicate conditions. In other possible implementations, the target predicate condition can also be any other predicate condition among the multiple predicate conditions.
[0215] In scenario two above, when the target query statement includes multiple predicate conditions and these predicate conditions correspond to different target data tables, the information of the target data table corresponding to one of the predicate conditions can be selected to perform the process of obtaining the routing information of that target data table. This ensures that one business request corresponds to obtaining the routing information of one target data table, thereby laying the foundation for ensuring that a subsequent business request corresponds to one database server.
[0216] The above embodiments illustrate the scheme using the example of routing information for the target data table existing in the database. In other embodiments, when routing information for the target data table does not exist in the database, the database server can randomly determine a database server from multiple database servers based on information such as the target data table's OID, and then send the determined database server's information to the business server, so as to instruct the business server to use the determined database server as the database server for subsequent business requests.
[0217] It is worth noting that the above-mentioned S704 can be executed by a database instance in a database server. Specifically, the process of obtaining routing information for the target data table based on its information can be executed by the routing management module within the database instance.
[0218] Furthermore, in some embodiments, the database server also sends index reference information to the business server. This index reference information indicates the position of a target parameter in the target query statement. This target parameter indicates data to be processed related to the index of the target data table, such as a parameter indicating the value of an indexed column. Exemplarily, the index reference information can indicate the position of the target parameter by adding a parameter subscript to the target query statement. In this embodiment, the data to be processed is used to determine a database server that processes at least one data table requested by the business request.
[0219] For example, taking the target data table as the user table shown in Table 1 above, and the index of the target data table as the B-tree index shown in Figure 8 above, the target parameter can be the age parameter in predicate condition 3 of the target query statement. Accordingly, index reference information can be used to indicate the location of the age parameter.
[0220] In this way, by sending index reference information to the business server, the business server can then determine the position of the target parameter in the target query statement based on the index reference information, and then determine the database server to process the business request based on the data indicated by the target parameter.
[0221] S705, The business server receives routing information for the target data table sent by the database server.
[0222] In some embodiments, after receiving the routing information of the target data table sent by the database server, the business server also caches the routing information of the target data table so that the business server can determine the database server to process the business request based on the cached routing information.
[0223] S706. The business server determines a database server to handle the business request based on the routing information of the target data table.
[0224] In some embodiments, the business server may cache routing information for different versions of the target data table. When it is necessary to determine the database server to execute a business request, the routing information for the latest version of the target data table can be determined from the cached routing information for different versions. Then, based on the routing information for the latest version of the target data table, the database server to execute the business request can be determined.
[0225] In some embodiments, the business server parses the target query statement to obtain the data to be processed from the target data table requested by the business request. Based on the data to be processed, the business server determines, from the routing information, a database server corresponding to the numerical range to which the data to be processed belongs as the database server to process the business request.
[0226] As described above, in some embodiments, the business server also receives index reference information sent by the database server. Based on this index reference information, it parses the target query statement to obtain the data to be processed related to the index of the target data table, as indicated by the target parameter. Then, based on this data to be processed, it determines from the routing information a database server corresponding to the numerical range to which the data to be processed belongs, as the database server to process the business request. Thus, the position of the target parameter in the target query statement can be determined through the index reference information, and a database server to process the business request can be determined based on the data to be processed indicated by the target parameter.
[0227] For example, taking the routing information of the target data table as shown in Table 2 above, assuming the target parameter indicates data "age=25", then the value range of "age=25" is greater than 23 and less than 27. Accordingly, the database server information corresponding to "age=25" is "Server 3". This means that Server 3 is determined to be the database server that executes the business request.
[0228] In the above embodiments, obtaining the routing information of the target data table based on the target query statement of the business request can ensure that the database server determined for the same business request is the same, which can also ensure that the same business request is executed only on a single database server, greatly reducing data access conflicts.
[0229] It is worth noting that S705 to S706 mentioned above can be executed by the database driver in the business server.
[0230] Furthermore, after determining the database server to execute the business request based on the above S706, the business server can send a business request to the determined database server to trigger the database server to execute the business request.
[0231] The technical solution provided in this application embodiment involves a business server receiving a business request and sending the target query statement from the business request to any one of the multiple database servers included in the database system. When the database server receives the target query statement from the business server, it sends routing information of the target data table to the business server based on the information of the target data table carried in the query statement. Furthermore, after receiving the routing information of the target data table sent by the database server, the business server can determine a database server to handle the business request based on the routing information. This ensures that a business request can be handled by a single database server; that is, a business request will be distributed to its corresponding database server instead of being randomly distributed to different database servers. This avoids the situation where multiple database servers access the same data, and prevents data from being frequently transferred from one database server to another, thereby reducing data access conflicts. The routing information of the target data table includes multiple numerical ranges of the index of the target data table and information about the database server corresponding to each numerical range, with one numerical range corresponding to one database server. In this way, when determining the database server to handle a business request, requests for data belonging to the same numerical range will be distributed to the database server corresponding to that range, rather than being randomly distributed to different database servers. This avoids situations where multiple database servers access the same data, and prevents data from being frequently transferred from one database server to another, thereby reducing data access conflicts.
[0232] Regarding the routing information of the target data table in Figure 7 above, the process of determining the routing information of the target data table is described below. Figure 9 is a flowchart illustrating a method for determining the routing information of a target data table according to an embodiment of this application. Referring to Figure 9, the method includes the following steps S901 to S905.
[0233] S901, The database server obtains the index of the target data table.
[0234] The target data table may include one or more indexes.
[0235] For example, in some embodiments, if the target data table includes an index, the database server can directly obtain the index and continue to the next step.
[0236] For example, in other embodiments, when the target data table includes multiple indexes, the index of the target data table is the index corresponding to the target index column of the target data table. Accordingly, the database server can select the target index column from at least one index column of the target data table based on the business access records of the target data table. Furthermore, the database server determines the index corresponding to the target index column as the index of the target data table.
[0237] Among them, business access records refer to data records of business request access obtained from the statistics of historical business requests, such as information on the data tables requested by historical business requests and the values of predicate conditions hit.
[0238] In this embodiment, the target index column is the most frequently accessed index column among at least one index column. It is understood that accessing an index column means that the predicate condition of the business request matches a value in that index column. Thus, by selecting the most frequently accessed index column in the target data table, we can obtain the index columns frequently queried in business requests, i.e., we obtain the index columns favored by the business. Furthermore, determining the indexes of the target data table based on these business-favored index columns increases the amount of information referenced in index determination, significantly improving query performance based on business requests and optimizing the overall database performance.
[0239] In some embodiments, the target index column is the index column corresponding to the first edge with the largest weight value among at least one first edge connecting the node corresponding to the target data table in the index analysis graph.
[0240] The process of generating the index analysis graph can be as follows: Based on at least one index column of all data tables included in the database system and the business access records of each data table, an index analysis graph is generated.
[0241] The index analysis graph includes multiple nodes, at least one first edge connecting each node, and weights for each first edge. Nodes indicate data tables. The at least one first edge connecting a node indicates at least one indexed column of the data table corresponding to that node. The weight of the first edge indicates the number of times the indexed column corresponding to the first edge has been accessed.
[0242] When adding edge weights to the first edge, the predicate conditions that hit the index column in the business access records can be counted. For each predicate condition that hits the index column, the weight of the first edge corresponding to that index column is increased by one.
[0243] For example, Figure 10 is a schematic diagram of an index analysis graph provided in an embodiment of this application. Referring to Figure 10, taking an index analysis graph including a user table and a department table as an example, two nodes are used to represent the user table and the department table. The node of the user table is connected by two edges, representing index column 1 and index column 2 respectively. The weight of index column 1 is 100, indicating that index column 1 has been accessed 100 times, and the weight of index column 2 is 68, indicating that index column 2 has been accessed 68 times. The node of the department table is connected by two edges, representing index column 3 and index column 4 respectively. The weight of index column 3 is 79, indicating that index column 3 has been accessed 79 times, and the weight of index column 4 is 98, indicating that index column 4 has been accessed 98 times.
[0244] After generating the index analysis graph, you can select the first edge with the largest weight value from at least one first edge connecting the node corresponding to the target data table in the index analysis graph, and use the index column corresponding to the first edge with the largest weight value as the target index column.
[0245] For example, taking the index analysis diagram shown in Figure 10 as an example, for the user table, index column 1 has the highest weight, so index column 1 is selected as the target index column. For the department table, index column 2 has the highest weight, so index column 2 is selected as the target index column.
[0246] Furthermore, in some embodiments, the index analysis graph is also used to represent the relationships between all the data tables included in the database system. Specifically, the index analysis graph also includes a second edge between nodes, which represents the relationship between the data tables corresponding to the two nodes connected by the second edge.
[0247] In some possible implementations, the database server also obtains the relationships between multiple data tables included in the database system, so as to generate the index analysis graph based on the relationships between the multiple data tables, combined with at least one index column of each data table and the business access records of each data table.
[0248] The process by which the database server obtains the relationships between multiple data tables can be as follows: based on the foreign key columns of multiple data tables, the relationships between the multiple data tables are obtained. A foreign key column is a column in the current data table that references the primary key column of another data table. For example, Figure 11 is a schematic diagram of another index analysis graph provided in an embodiment of this application. Referring to Figure 11, taking an index analysis graph including a user table and a department table as an example, two nodes are used to represent the user table and the department table. A second edge connects the node of the user table and the node of the department table, indicating a relationship between the user table and the department table. It should be understood that the second edge connecting the node of the user table and the node of the department table can be a foreign key column of the user table that is also a primary key column of the department table, or a primary key column of the user table that is also a foreign key column of the department table. Optionally, the second edge can be a directed edge, such as a second edge pointing from the user table to the department table, which can represent a foreign key column of the user table that is also a primary key column of the department table, or a second edge pointing from the department table to the user table, which can represent a primary key column of the user table that is also a foreign key column of the department table.
[0249] In the above embodiments, by constructing an index analysis graph, using nodes to represent data tables, using first edges to represent index columns, and adding edge weights to characterize the number of times the corresponding index column is accessed, the index column with the most accesses in the target data table can be quickly determined by analyzing the edge weights of each first edge connected to the corresponding node of the target data table, that is, the target index column is obtained.
[0250] It is understandable that indexes are created based on one or more indexed columns in a data table. For example, a single-column index is created based on one indexed column in a data table. For instance, taking the user table shown in Table 1 as an example, a single-column index can be created based on the user ID column in the user table. It is understandable that when executing a SELECT query, if the predicate condition involves the user ID column, this single-column index can be used to quickly locate the data that meets the condition, thereby improving query performance. Similarly, a composite index is created based on multiple indexed columns in a data table. For instance, taking the user table as an example, a composite index can be created based on the user ID column and the age column in the user table. It is understandable that when executing a SELECT query, if the predicate condition involves the user ID column and the age column, this composite index can be used to quickly locate the data that meets the condition, thereby improving query performance.
[0251] When the index is a single-column index, the index of the target data table is an index created based on the target index column. Alternatively, when the index is a multi-column index, the index of the target data table is an index created with the target index column as the first index column. This provides a method for determining whether an index is single-column or composite, allowing for quick and efficient determination of the corresponding index after obtaining the target index column.
[0252] The information referenced in S901 for obtaining the index of the data table, such as business access records and the primary key columns, foreign key columns, index columns, and index information (such as the index name and type), can all be obtained from the database. In some embodiments, information within the current database can be collected using tools or commands provided by the database. Business access records can be obtained from the database's log file. The primary key columns, foreign key columns, index columns, and index information of the data table can be obtained from the database's data file. For example, the database's data file may store the index creation information of the data table. This index creation information may include the column information of the data table, the name of the index, and the type of the index. Optionally, the primary key columns, foreign key columns, and index columns of the data table can be determined through the column information of the data table in the index creation information (such as whether the column is unique, the column name, and the column type).
[0253] S902, The database server obtains the numerical distribution range of the indexes of the target data table.
[0254] In some embodiments, after the database server obtains the index of the target data table, it determines the numerical distribution range of the index based on the index of the target data table.
[0255] When the index is a B-tree index, either implementation method one or implementation method two can be used to determine the numerical distribution range of the index of the target data table.
[0256] Implementation Method 1: Determine the range of numerical distribution of the index of the target data table based on the key values stored in the root node of the B-tree index of the target data table.
[0257] The key values stored in the root node are used to divide the overall numerical range of the B-tree index (i.e., the numerical range of the subtrees connected to the root node) into multiple sub-ranges. In some embodiments, the numerical distribution range of the index may include multiple sub-ranges. Accordingly, by traversing the key values stored in the root node of the B-tree index, these multiple sub-ranges can be obtained, that is, the numerical distribution range of the B-tree index can be obtained.
[0258] For example, as shown in Figure 8, based on the key values 23 and 27 included in the root node of the B-tree index, it can be determined that the numerical distribution range of the B-tree index includes a sub-range less than 23, a sub-range greater than 23 and less than 27, and a sub-range greater than 27.
[0259] Method 2: Determine the range of numerical distribution of the index of the target data table based on the high key values stored in multiple intermediate nodes of the B-tree index of the target data table.
[0260] Among them, the intermediate node can be the next level non-leaf node of the root node (such as the second level non-leaf node) or the next level non-leaf node (such as the third level non-leaf node).
[0261] The key values stored in the intermediate node are used to divide the numerical range of the subtree connected to the intermediate node into multiple subranges. For example, assuming that the intermediate node includes key values k1, k2, and k3, it means that the subtree connected to the intermediate node is divided into subranges less than k1, greater than k1 and less than k2, and greater than k2.
[0262] A high key value refers to the largest key value among at least one key value stored in an intermediate node, that is, the largest key value in the subtree managed by the intermediate node, such as the key value k3 mentioned above. It can be understood that the high key values stored in multiple intermediate nodes of a B-tree index also divide the numerical range of the subtree connected by the root node into multiple sub-ranges, that is, divide the overall numerical range of the B-tree index into multiple sub-ranges. Therefore, based on the high key values stored in multiple intermediate nodes, the numerical distribution range of the B-tree index can also be determined.
[0263] In some possible implementations, multiple intermediate nodes of the B-tree index (next level non-leaf nodes) are accessed through the root node of the B-tree index. The high key values stored in each intermediate node are obtained sequentially from left to right. Then, multiple sub-ranges are obtained based on the high key values stored in the multiple intermediate nodes, which is to say, the numerical distribution range of the B-tree index is obtained.
[0264] Regarding the two implementation methods for determining the numerical distribution range of the target data table's index based on the B-tree index of the target data table, in some possible implementations, the database server can first obtain the level of the B-tree index of the target data table. If the level of the B-tree index is less than or equal to 2, the first implementation method is used to determine the numerical distribution range of the target data table's index. If the level of the B-tree index is greater than 2, the second implementation method is used to determine the numerical distribution range of the target data table's index. In other possible implementations, the database server can first obtain the number of non-leaf nodes (i.e., the number of root nodes and intermediate nodes) of the target data table's B-tree index. If the number of non-leaf nodes of the B-tree index is less than or equal to a preset number, the first implementation method is used to determine the numerical distribution range of the target data table's index. If the number of non-leaf nodes of the B-tree index is greater than the preset number, the second implementation method is used to determine the numerical distribution range of the target data table's index. The preset number can be a pre-set threshold value, such as a value between (50, 200), which is not limited in this embodiment.
[0265] In the above embodiments, after determining the index based on the target index column, the numerical distribution range of the index can be quickly and accurately obtained through the index key values, improving real-time performance and accuracy. This achieves the goal of obtaining the index's numerical distribution range faster and more accurately without relying on the execution of the database optimizer's ANALYZE command. The database optimizer's ANALYZE command is used to collect and update statistical information of the data table for statistical analysis. It is understood that the numerical distribution range of the index obtained in this embodiment can also be used for statistical analysis by the database optimizer.
[0266] S903. The database server divides the range of numerical values into multiple ranges according to the number of database servers included in the database system.
[0267] In this embodiment, the multiple database servers refer to the active database servers among all database servers included in the database system. An active database server is one that is currently operating normally and capable of providing services.
[0268] In some embodiments, the database server obtains a list of active database servers and assigns a numerical range from the numerical distribution range to each database server in the list.
[0269] S904. The database server generates routing information for the target data table based on multiple numerical ranges and information from multiple database servers.
[0270] S905, The database server stores the routing information of the target data table into the database.
[0271] The database is used to store routing information for all data tables in the database system and supports access from multiple database servers.
[0272] In some embodiments, after generating the routing information for the target data table, the database server also updates the version information of the routing information, such as incrementing the version number of the routing information by one bit and persisting it to the database.
[0273] The above steps S901 to S905 provide a method for determining routing information for a target data table. Specifically, the numerical distribution range of the target data table's indexes is divided according to the number of database servers, resulting in an equal number of numerical intervals, thus ensuring that each numerical interval corresponds to one database server. Furthermore, the multiple numerical intervals and the corresponding database server information are stored to generate routing information, facilitating the normal execution of subsequent data processing flows.
[0274] It is worth noting that the database server can generate routing information for all data tables included in the database system and store it in the database before executing the process shown in Figure 7 above.
[0275] The embodiment shown in Figure 9 above uses a database server as the execution entity to illustrate the scheme. In some embodiments, S901 to S905 can be executed by the routing management module of the SQL engine in the database instance of the database server. It is understood that for each database instance in each database server, the routing management module in that database instance can periodically generate routing information for each data table and update the routing information of each data table in the database in a timely manner during the operation of the database instance.
[0276] The technical solution provided in this application, in a distributed database based on shared storage, logically partitions each data table into multiple value ranges according to the numerical distribution range of the indexes of each data table, and assigns a database server to each value range. This guides business clients to select the appropriate database server for access, thereby aggregating similar business requests to the same database server. This effectively reduces data access conflicts, such as those involving hot data, and avoids ping-pong data transfer between database servers, thus improving the throughput of the database system. Furthermore, it combines the data distribution characteristics of the index columns in the data tables with the historical business access patterns of business requests to control data processing and prevent business requests from being frequently sent to different database servers.
[0277] It should be noted that the above description is for the purpose of more clearly explaining the data processing method described in the embodiments of this application, and should not be construed as a limitation on the specific implementation of this application.
[0278] The above mainly describes the solutions provided by the embodiments of this application from the perspective of processing flow. Correspondingly, the embodiments of this application also provide a data processing apparatus for implementing the various methods described above. This data processing apparatus can be one of the apparatuses in the above method embodiments, or include the aforementioned apparatus, or be a usable component. It is understood that, in order to achieve the above functions, the data processing apparatus includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0279] This application embodiment can divide the data processing device into functional modules according to the above method embodiment. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be understood that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.
[0280] For example, Figure 12 is a schematic diagram of a data processing device provided in an embodiment of this application. Referring to Figure 12, the data processing device includes a receiving module 1201 and a transmitting module 1202. Wherein:
[0281] The receiving module 1201 is used to execute S703 shown in Figure 7 above;
[0282] The sending module 1202 is used to execute S704 shown in Figure 7 above.
[0283] In some possible implementations, the device further includes a routing management module for performing S901 to S905 as shown in FIG9 above.
[0284] For example, Figure 13 is a schematic diagram of another data processing apparatus provided in an embodiment of this application. Referring to Figure 13, the data processing apparatus includes a receiving module 1301, a sending module 1302, and a determining module 1303. Wherein:
[0285] The receiving module 1301 is used to execute S701 and S705 shown in FIG7 above;
[0286] The sending module 1302 is used to execute S702 shown in Figure 7 above;
[0287] The determination module 1303 is used to execute S706 shown in Figure 7 above.
[0288] For a detailed description of the above-mentioned optional methods, please refer to the foregoing method embodiments, which will not be repeated here. Furthermore, the explanation of any of the data processing devices provided above and the description of their beneficial effects can be found in the corresponding method embodiments described above, and will not be repeated here.
[0289] As an example, referring to FIG6, some or all of the functions implemented in the receiving module 1201 and transmitting module 1202 in the data processing device shown in FIG12 or the receiving module 1301, transmitting module 1302 and determining module 1303 in the data processing device shown in FIG13 can be implemented by the processor 601 in FIG6 executing the computer execution instructions in the memory 602 in FIG6.
[0290] In this embodiment, the data processing device is presented as an integrated unit divided into functional modules. Here, "module" can refer to an application-specific integrated circuit (ASIC), a circuit, a processor and memory executing one or more software or firmware programs, integrated logic circuits, and / or other devices that can provide the aforementioned functions. In a simplified embodiment, those skilled in the art will recognize that the data processing device can take the form of the server shown in Figure 6.
[0291] For example, the processor 601 in the server shown in Figure 6 can call computer execution instructions stored in the memory 602 to cause the server to execute the data processing method in the above method embodiment.
[0292] Since the data processing apparatus provided in this application embodiment can execute the above data processing method, the technical effects it can obtain can be referred to the above method embodiment, and will not be repeated here.
[0293] It should be understood that one or more of the above modules or units can be implemented by software, hardware, or a combination of both. When any of the above modules or units are implemented by software, the software exists as computer program instructions and is stored in memory. The processor can be used to execute the program instructions and implement the above method flow. The processor can be built into a system-on-chip (SoC) or ASIC, or it can be a separate semiconductor chip. In addition to the core that executes software instructions for computation or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), or logic circuits that implement dedicated logic operations.
[0294] When the above modules or units are implemented in hardware, the hardware can be any one or any combination of a microprocessor, digital signal processing (DSP) chip, microcontroller unit (MCU), artificial intelligence processor, ASIC, SoC, FPGA, PLD, application-specific digital circuit, hardware accelerator, or non-integrated discrete device, which can run the necessary software or perform the above method flow independently of software.
[0295] Optionally, embodiments of this application also provide a database server (e.g., the database server may be a chip or a chip system), the database server including a processor for implementing the methods executed by the database server in any of the above method embodiments. In one possible design, the database server further includes a memory. The memory is used to store necessary program instructions and data, and the processor can call the program code stored in the memory to instruct the database server to execute the methods in any of the above method embodiments. Of course, the memory may not be present in the database server. When the database server is a chip system, it may be composed of chips or may include chips and other discrete devices; embodiments of this application do not specifically limit this.
[0296] Optionally, embodiments of this application also provide a service server (e.g., the service server may be a chip or a chip system), which includes a processor for implementing the methods executed by the service server in any of the above method embodiments. In one possible design, the service server further includes a memory. The memory is used to store necessary program instructions and data, and the processor can call the program code stored in the memory to instruct the service server to execute the methods in any of the above method embodiments. Of course, the memory may not be present in the service server. When the service server is a chip system, it may be composed of chips or may include chips and other discrete devices; embodiments of this application do not specifically limit this.
[0297] This application also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed on a database server, the database server executes the method performed by any of the data processing devices provided above. When the computer-executable instructions are executed on a business server, the business server executes the method performed by any of the data processing devices provided above.
[0298] For explanations of the relevant content and descriptions of the beneficial effects in any of the computer-readable storage media provided above, please refer to the corresponding embodiments described above, which will not be repeated here.
[0299] This application also provides a chip. The chip integrates a control circuit for implementing the functions of the aforementioned data processing device and one or more ports. Optionally, the functions supported by the chip can be referred to above, and will not be repeated here. Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. The aforementioned storage medium can be a read-only memory, random access memory, etc. The aforementioned processing unit or processor can be a central processing unit, a general-purpose processor, an ASIC, a DSP, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.
[0300] This application also provides a computer program product containing computer-executable instructions. When these computer-executable instructions are executed on a database server, they cause the database server to perform any of the methods described in the above embodiments. The computer program product includes one or more computer-executable instructions. When the computer-executable instructions are loaded and executed on the database server, all or part of the flow or function according to the embodiments of this application is generated. The database server may be a network device or other programmable device. When the computer-executable instructions are executed on a business server, they cause the business server to perform any of the methods described in the above embodiments. The computer program product includes one or more computer-executable instructions. When the computer-executable instructions are loaded and executed on the business server, all or part of the flow or function according to the embodiments of this application is generated. The business server may be a network device or other programmable device.
[0301] Computer-executable instructions can be stored in or transmitted from one computer-readable storage medium to another. For example, computer-executable instructions can be transmitted from one website, computer, network device, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. A computer-readable storage medium can be any available medium accessible to a network device or a data storage device that includes one or more network devices, data centers, etc., that can be integrated with media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).
[0302] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of this application, such as but not limited to the memory, computer-readable storage medium and communication chip, are all non-transitory.
[0303] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product.
[0304] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, the disclosure, and the appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.
[0305] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.
Claims
1. A data processing method, characterized in that, Applied to a database server, the method includes: Receive a target query statement from the business server, the target query statement carrying information about the target data table; Based on the target query statement, the routing information of the target data table is sent to the business server; The routing information is used to determine a database server for processing at least one data table requested by the service request, wherein the target data table is one of the at least one data table.
2. The method according to claim 1, characterized in that, The routing information includes multiple numerical ranges of the index of the target data table and information about the database server corresponding to each numerical range, with one numerical range corresponding to one database server.
3. The method according to claim 1 or 2, characterized in that, The process of determining the routing information of the target data table includes: Obtain the numerical distribution range of the indexes of the target data table; The numerical distribution range is divided into multiple numerical intervals according to the number of database servers included in the database system; Based on the multiple numerical ranges and the information from the multiple database servers, routing information for the target data table is generated.
4. The method according to claim 3, characterized in that, The index is a B-tree index, and the process of obtaining the numerical distribution range of the index of the target data table includes: Based on the key values stored in the root node of the B-tree index of the target data table, determine the numerical distribution range of the index of the target data table; or, Based on the high key values stored in multiple intermediate nodes of the B-tree index of the target data table, the numerical distribution range of the index of the target data table is determined, wherein the high key value refers to the largest key value among at least one key value stored in the intermediate node.
5. The method according to claim 3 or 4, characterized in that, The index of the target data table is the index corresponding to the target index column of the target data table; The method further includes: Based on the business access records of the target data table, a target index column is selected from at least one index column of the target data table, wherein the target index column is the index column that has been accessed the most times among the at least one index column.
6. The method according to claim 5, characterized in that, The target index column is the index column corresponding to the first edge with the largest weight value among at least one first edge connecting the node corresponding to the target data table in the index analysis graph; The method further includes: The index analysis graph is generated based on at least one index column of all data tables included in the database system and the business access records of each data table; The index analysis graph includes multiple nodes, at least one first edge connected to each node, and weights for each first edge. The nodes are used to indicate data tables, and the at least one first edge connected to each node is used to indicate at least one index column of the data table corresponding to the node. The weight of the first edge is used to indicate the number of times the index column corresponding to the first edge has been accessed.
7. The method according to claim 5 or 6, characterized in that, If the index is a single-column index, the index of the target data table is an index created based on the target index column; or, In the case of a multi-column index, the index of the target data table is an index created with the target index column as the first index column.
8. The method according to any one of claims 3-7, characterized in that, The plurality of database servers are the active database servers among all the database servers included in the database system.
9. The method according to any one of claims 3-8, characterized in that, After generating the routing information for the target data table based on the multiple numerical ranges and the information from the multiple database servers, the method further includes: The routing information of the target data table is stored in a database, which is used to store the routing information of the data tables included in the database system and supports access by the multiple database servers.
10. The method according to claim 9, characterized in that, Before sending the routing information of the target data table to the business server, the method further includes: From the database, based on the information of the target data table carried in the target query statement, the routing information of the target data table is obtained.
11. The method according to any one of claims 1-10, characterized in that, The method further includes: The index reference information is sent to the business server. The index reference information is used to indicate the position of the target parameter in the target query statement. The target parameter is used to indicate the data to be processed related to the index of the target data table. The data to be processed is used to determine the database server that processes at least one data table requested by the business request.
12. A data processing method, characterized in that, Applied to a business server, the method includes: Receive a business request, the business request being used to request processing of at least one data table, the business request including at least one query statement; Send a target query statement to any one of the multiple database servers included in the database system. The target query statement is one of the at least one query statement. The target query statement carries information about a target data table, which is one of the at least one data tables. Receive routing information for the target data table sent by the database server; Based on the routing information of the target data table, a database server is determined to handle the business request.
13. The method according to claim 12, characterized in that, The routing information includes multiple numerical ranges of the index of the target data table and information about the database server corresponding to each numerical range, with one numerical range corresponding to one database server.
14. The method according to claim 13, characterized in that, The step of determining a database server to handle the business request based on the routing information of the target data table includes: The target query statement is parsed to obtain the data to be processed from the target data table requested by the business request. Based on the data to be processed, a database server corresponding to the numerical range to which the data to be processed belongs is determined from the routing information as the database server to process the business request.
15. The method according to claim 14, characterized in that, Before determining a database server to handle the business request based on the routing information of the target data table, the method further includes: The system receives index reference information sent by the database server. The index reference information is used to indicate the position of the target parameter in the target query statement. The target parameter is used to indicate the data to be processed related to the index of the target data table. The parsing of the target query statement includes: The target query statement is parsed based on the index reference information.
16. The method according to any one of claims 12-15, characterized in that, After receiving the routing information of the target data table sent by the database server, the method further includes: Cache the routing information of the target data table.
17. A data processing apparatus, characterized in that, The device, applied to a database server, includes: The receiving module is used to receive a target query statement from the business server, wherein the target query statement carries information about the target data table; The sending module is used to send the routing information of the target data table to the business server based on the target query statement; The routing information is used to determine a database server for processing at least one data table requested by the service request, wherein the target data table is one of the at least one data table.
18. A data processing apparatus, characterized in that, The device, applied to a business server, includes: A receiving module is used to receive a business request, the business request being used to request processing of at least one data table, the business request including at least one query statement; The sending module is used to send a target query statement to any one of the multiple database servers included in the database system. The target query statement is one of the at least one query statement. The target query statement carries information about a target data table, which is one of the at least one data tables. The receiving module is also used to receive routing information of the target data table sent by the database server; The determination module is used to determine a database server to process the business request based on the routing information of the target data table.
19. A database server, characterized in that, The method includes a memory and a processor, the memory and the processor being connected; the memory is used to store computer-executable instructions; the processor is used to invoke the computer-executable instructions to perform the method as described in any one of claims 1-11.
20. A business server, characterized in that, The method includes a memory and a processor, the memory and the processor being connected; the memory is used to store computer-executable instructions; the processor is used to invoke the computer-executable instructions to perform the method as described in any one of claims 12-16.
21. A data processing system, characterized in that, It includes the database server as described in claim 19 and the business server as described in claim 20.
22. A computer-readable storage medium, characterized in that, The method includes computer-executed instructions that, when executed on a database server, cause the database server to perform the method as described in any one of claims 1-11, and when executed on a business server, cause the business server to perform the method as described in any one of claims 12-16.
23. A computer program product, characterized in that, The method includes computer-executed instructions that, when executed on a database server, cause the database server to perform the method as described in any one of claims 1-11, and when executed on a business server, cause the business server to perform the method as described in any one of claims 12-16.
