Memory management method and apparatus

Abstract

The present application provides a memory management method and apparatus. The method comprises: a database system acquiring a reference capacity of a second memory and a number S of sample query statements in the database system, the second memory being used for columnar storage; on the basis of the reference capacity of the second memory and the S sample query statements, the database system determines an objective function, and a constraint condition of the objective function, the objective function being that time gains respectively corresponding to the S sample query statements satisfy a first condition; and on the basis of the constraint condition, the database system solves the objective function and obtains a data column set, the data column set comprising at least one data column, and the data column being a column of storage data selected from a persistent storage medium and used for loading into the second memory. The data column set determined by the described method helps ensure that time gains respectively corresponding to a number K of sub-query statements satisfy a first condition, thereby helping improve the execution efficiency of subsequent query statements.

Description

A memory management method and apparatus

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411550691.2, filed on October 31, 2024, entitled "A Memory Management Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and in particular to a memory management method and apparatus. Background Technology

[0004] With the advent of the big data era, enterprises' data processing needs are becoming increasingly complex. Currently, different types of database systems have emerged to address different data processing needs. Common database systems can be broadly categorized into two types: Online Transaction Processing (OLTP) systems and Online Analytical Processing (OLAP) systems. OLTP systems are primarily used for transaction processing (TP), such as bank transfers and sales order entry. OLTP systems typically use row-based storage in memory. OLAP systems are primarily used for analytical processing (AP), such as data analysis and querying. OLAP systems typically use column-based storage in memory.

[0005] As data processing tasks become increasingly blurred, often requiring both transactional (TP) and analytical (AP) processing capabilities, hybrid transactional and analytical processing (HTAP) systems that simultaneously support both have become a major trend and direction in database technology development. HTAP systems typically employ a combined row-based and column-based storage architecture, dividing the system's memory into separate memory segments for row-based and column-based storage. The column-based storage segment can store at least one column of data, selected from the data stored in persistent storage.

[0006] Currently, HTAP systems select frequently accessed data as data columns based on the access frequency of data stored in persistent storage media and load these columns into memory for columnar storage. However, this method of selecting data columns does not consider the execution efficiency of query statements. Therefore, how to select data columns to load into memory for columnar storage to improve query execution efficiency is a pressing technical problem that needs to be solved. Summary of the Invention

[0007] This application provides a memory management method and apparatus for more efficiently and accurately allocating the total memory capacity of an HTAP system.

[0008] In a first aspect, embodiments of this application provide a memory management method. The execution subject of this method is a database system, a module within a database system, a query engine, or a storage engine within a database system. Here, a database system is used as the execution subject for example. The method includes: the database system obtaining a reference capacity of a second memory and S sample query statements in the database system, wherein the second memory is used for columnar storage; the database system determining an objective function and constraints on the objective function based on the reference capacity of the second memory and the S sample query statements; wherein the objective function is that the time benefit corresponding to each of the S sample query statements satisfies a first condition; the time benefit of one sample query statement among the S sample query statements is determined based on the difference between the time required to read the data column corresponding to the sample query statement from the persistent storage medium and the time required to read it from the second memory; the database system solving the objective function according to the constraints to obtain a set of data columns; wherein the set of data columns includes at least one data column, which is a column of stored data selected from the persistent storage medium and used to be loaded into the second memory.

[0009] In the above method, the set of data columns determined by the database system includes at least one data column, wherein the data column is a column of data selected from the persistent storage medium. The set of data columns is subsequently loaded into the second memory. The determined set of data columns helps to ensure that the time benefits corresponding to the K subqueries meet the first condition, thereby improving the execution efficiency of subsequent query statements.

[0010] In one possible design, the constraints include at least one of the following: the total number of selected data columns to be loaded into the second memory and the reference capacity of the second memory satisfy the second condition; and the data column corresponding to one of the S sample query statements can be read from the second memory if all the data columns corresponding to one of the sample query statements are selected.

[0011] In one possible design, the database system determines the objective function and its constraints based on the reference capacity of the second memory and S sample query statements. This can include: the database system splits each of the S sample query statements into K subqueries; the database system then determines the objective function and its constraints based on the reference capacity of the second memory and the K subqueries; the objective function is that the time benefit corresponding to each of the K subqueries satisfies a first condition. In this design, the database system first splits the S sample query statements into K subqueries, and then determines the objective function and its constraints based on the reference capacity of the second memory and the K subqueries. The determined objective function and its constraints are more precise, resulting in a more precise set of data columns.

[0012] In one possible design, the constraints include at least one of the following: the total number of selected data columns to be loaded into the second memory and the reference capacity of the second memory satisfy a second condition; and the data column corresponding to one of the K subqueries can be read from the second memory if all the data columns corresponding to one of the subqueries are selected.

[0013] In one possible design, the above method may further include: the database system determining the total throughput of the database system based on the variable capacity of the first memory and the variable capacity of the second memory; the first memory is used for row-oriented storage; both the variable capacity of the first memory and the variable capacity of the second memory are continuously adjusted variables; the database system determines a reference capacity of the first memory corresponding to the variable capacity of the first memory, and a reference capacity of the second memory corresponding to the variable capacity of the second memory; both the reference capacity of the first memory and the reference capacity of the second memory are constants; the database system uses a first algorithm to calculate the first variable capacity based on the total throughput to obtain a first processing capacity, and obtains a first capacity based on the first reference capacity and the first processing capacity; the first capacity is used to maximize the total throughput of the database system; the first variable capacity is one of the variable capacity of the first memory and the variable capacity of the second memory; the first reference capacity is one of the reference capacity of the first memory and the reference capacity of the second memory; the first capacity is one of the capacity of the first memory and the capacity of the second memory; the database system obtains a second capacity based on the first capacity and the total memory capacity of the database system; the second capacity is one of the capacity of the first memory and the capacity of the second memory, and the second capacity is different from the first capacity. In this design, the process of dividing the total memory capacity of the database system into the capacity of the first memory and the capacity of the second memory does not rely on human experience, but is based on the total throughput of the database system. This allows for a more efficient and accurate division of the total memory capacity of the database system, thereby improving the efficiency and accuracy of memory management.

[0014] In one possible design, the database system determines its total throughput based on the variable capacity of the first memory and the variable capacity of the second memory. This can include: determining the database system's transaction throughput based on the variable capacity of the first memory and the total amount of data stored in the persistent storage medium; determining the database system's analytical processing throughput based on the variable capacity of the second memory and the set of data columns; and determining the database system's total throughput based on the transaction throughput and the analytical processing throughput.

[0015] In one possible design, the database system determines its analytical processing throughput based on the variable capacity of the second memory and the set of data columns. This can include: given the set of data columns, the database system determines the time cost corresponding to each of the S sample query statements; wherein the time cost of one of the S sample query statements is determined based on the time required to read the corresponding data column from persistent storage medium or second memory; and the database system determines the analytical processing throughput based on the time cost corresponding to each of the S sample query statements.

[0016] In one possible design, the database system uses a first algorithm to calculate a first variable capacity based on the total throughput to obtain a first unprocessed capacity, and obtains the first capacity based on a first reference capacity and the first unprocessed capacity. This can include iteratively executing the following steps until an iteration stopping condition is met: the database system calculates a gradient estimate of the total throughput with respect to the variable capacity of the first memory to obtain the first unprocessed capacity; the database system processes the first unprocessed capacity based on the reference capacity of the first memory to obtain an adjustment amount for the first memory; the database system obtains an updated reference capacity for the first memory based on the reference capacity and the adjustment amount; and the database system uses the reference capacity of the first memory that meets the iteration stopping condition as the first capacity. In this design, by calculating the adjustment amount of the first memory and then updating the reference capacity of the first memory based on the adjustment amount to obtain the first capacity, a method for obtaining the first capacity is provided.

[0017] In one possible design, the database system obtains an updated reference capacity for the first memory based on the reference capacity of the first memory and the adjustment amount of the first memory. This may include: the database system determining the updated reference capacity for the first memory based on the sum of the reference capacity of the first memory and the adjustment amount of the first memory.

[0018] In one possible design, the database system uses a first algorithm to calculate a first variable capacity based on the total throughput to obtain a first unprocessed capacity, and obtains the first capacity based on a first reference capacity and the first unprocessed capacity. This can include iteratively executing the following steps until an iteration stopping condition is met: the database system calculates a gradient estimate of the total throughput with respect to the variable capacity of the second memory to obtain the first unprocessed capacity; the database system processes the first unprocessed capacity based on the reference capacity of the second memory to obtain an adjustment amount for the second memory; the database system obtains an updated reference capacity for the second memory based on the reference capacity and the adjustment amount; and the database system uses the reference capacity of the second memory that meets the iteration stopping condition as the first capacity. In this design, by calculating the adjustment amount of the second memory and then updating the reference capacity of the second memory based on the adjustment amount to obtain the first capacity, a method for obtaining the first capacity is provided.

[0019] In one possible design, the database system obtains an updated reference capacity for the second memory based on the reference capacity of the second memory and the adjustment amount of the second memory. This may include: the database system determining the updated reference capacity of the second memory based on the sum of the reference capacity of the second memory and the adjustment amount of the second memory.

[0020] In one possible design, the above method may further include: the database system obtaining the query statement to be executed and determining the data columns corresponding to the query statement; if the data columns corresponding to the query statement are located in the data column set, the database system loads the data columns corresponding to the query statement from the persistent storage medium into the second memory; the database system executes the query statement based on the data columns corresponding to the query statement located in the second memory. During the execution of the query statement, if the data columns corresponding to the query statement are located in the data column set, loading the data columns corresponding to the query statement into the second memory eliminates the need for additional time to load the data column set, effectively achieving seamless loading of the data column set.

[0021] In one possible design, the above method may further include: the database system obtaining a first feature vector, wherein the first feature vector is used to mark the frequency information of the query statements to be executed within a first time period, the frequency information of the query statements to be executed including the frequency of the template query statement corresponding to the query statement to be executed and / or the frequency of the data column corresponding to the query statement to be executed; the database system obtaining a second feature vector, the second feature vector being used to mark the frequency information of the query statements to be executed within a second time period; the second time period is located before the first time period; if the similarity between the first feature vector and the second feature vector is less than a first threshold, the database system determines the capacity of the first memory and the capacity of the second memory. In this design, when a load shift occurs, the capacity of the first memory and the capacity of the second memory can be adjusted in a timely manner, which can more efficiently and accurately allocate the total memory capacity of the database system, thereby improving the efficiency and accuracy of memory management. At the same time, it can also determine the set of data columns with higher time benefits, which can help improve the execution efficiency of subsequent query statements.

[0022] Secondly, embodiments of this application also provide a memory management apparatus, which includes methods for performing the methods provided in the first aspect above.

[0023] Thirdly, embodiments of this application also provide a computer program product containing instructions that, when executed by a computing system, cause the computing system to perform the method provided in the first aspect above.

[0024] Fourthly, embodiments of this application also provide a computer-readable storage medium including computer program instructions, which, when executed by a computing system, enable the computing system to perform the method provided in the first aspect above.

[0025] Fifthly, embodiments of this application also provide a computing device, including a processor and a memory, wherein the memory stores computer-executable instructions; the processor is used to execute the computer-executable instructions stored in the memory, and the processor executes the method provided in the first aspect above.

[0026] Sixthly, embodiments of this application also provide a chip containing instructions that, when executed by a computing system, cause the computing system to perform the method provided in the first aspect above.

[0027] In a seventh aspect, embodiments of this application provide a chip system containing instructions that, when executed by a computing system, cause the computing system to perform the method provided in the first aspect above. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0028] Figure 1 is a schematic diagram of the structure of a database system provided in an embodiment of this application;

[0029] Figure 2 is a flowchart illustrating a memory management method provided in an embodiment of this application;

[0030] Figure 3 is a flowchart illustrating a method for determining the total throughput of a database according to an embodiment of this application;

[0031] Figure 4 is a flowchart illustrating a method for determining a set of data columns provided in an embodiment of this application;

[0032] Figure 5 is a flowchart illustrating a method for determining a set of data columns provided in an embodiment of this application;

[0033] Figure 6 is a flowchart illustrating a method for determining the analytical processing throughput of a database system according to an embodiment of this application;

[0034] Figure 7 is a flowchart illustrating a method for obtaining a first capacity according to an embodiment of this application;

[0035] Figure 8 is a flowchart illustrating an iterative process provided in an embodiment of this application;

[0036] Figure 9 is a flowchart illustrating a method for obtaining a first capacity according to an embodiment of this application;

[0037] Figure 10 is a flowchart illustrating a query statement execution method provided in an embodiment of this application;

[0038] Figure 11 is a flowchart illustrating a memory management method provided in an embodiment of this application;

[0039] Figure 12 is a schematic diagram of the structure of a database system provided in an embodiment of this application;

[0040] Figure 13 is a schematic diagram of the structure of a memory management device provided in an embodiment of this application. Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. The terms "first," "second," and corresponding reference numerals in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0042] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0043] 1. Matrix element-wise product (entry-wise product) refers to the element-wise multiplication of two matrices of the same size (i.e., the two matrices have the same number of rows and columns). For example, A and B are both p×q matrices. The matrix element-wise product of matrices A and B results in matrix C.

[0044] 2. Structured Query Language (SQL) is a special-purpose programming language, a database query and programming language used to access, query, update and manage relational database systems.

[0045] 3. Query statements are used to retrieve data from a database without modifying the data. The syntax of a simple query statement is `SELECT expression FROM table_name`, where the expression can be understood as a field name or column name in the table, and the table name can be understood as the identifier of the table. For example, `SELECT age FROM table1`, where `age` is the column name, `table1` is the table name, and the table corresponding to `table1` contains the data corresponding to the column name `age`.

[0046] 4. Transaction processing (TP) is a primary application of traditional relational databases, supporting basic, routine transaction processing. Examples include bank transfers and sales order entry.

[0047] 5. Analysis process (AP) is a primary application of database warehouses, supporting complex analytical operations and focusing on decision support. For example, analysis process can involve analyzing and querying data.

[0048] 6. Online transactional processing (OLTP) system: A database system used to process TP-type transactions.

[0049] 7. Online analytical processing (OLAP) system: a database system used to process AP-type transactions.

[0050] 8. Hybrid transactional and analytical processing (HTAP) system: A database system capable of handling both TP-type transactions and AP-type transactions simultaneously.

[0051] 9. Row-oriented storage: Data in a data table is stored in rows as the basic logical storage unit, and data in the same row exists in a contiguous storage form in the storage medium.

[0052] 10. Column-oriented storage: Data in a data table is stored in columns as the basic logical storage unit, and data in the same column exists in a contiguous storage form in the storage medium.

[0053] The following explanation uses a data table shown in Table 1 to illustrate row-based and column-based storage.

[0054] Table 1.

[0055] If the above data table is stored in row format, the content stored in the storage medium is: 1, Zhang San, male, 85; 2, Li Si, female, 90.

[0056] If the above data table is stored in columnar format, the contents stored in the storage medium are: 1, 2, Zhang San, Li Si, male, female, 85, 90.

[0057] 11. Transaction per second (TPS) refers to the total number of TP-type transactions that a database system can process per second. It is a standard used to measure the database system's ability to process TP-type transactions.

[0058] 12. Analytical Queries Per Second (QPS) refers to the total number of analytical (AP) transactions that a database system can process per second. It is a standard used to measure the database system's ability to handle AP transactions.

[0059] 13. Workload shift refers to a significant change in the workload of a database system caused by new applications, new users, or periodic changes in work patterns. This change may be a non-periodic or periodic load pattern switching.

[0060] The technical solutions provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0061] Figure 1 is a schematic diagram of a database system 100 provided in this application. The database system 100 includes a query engine 101 and a storage engine 102. The query engine 101 includes a parser 105, a preprocessor 106, a query optimizer 107, and a query execution engine 108. The storage engine 102 includes a row-level storage buffer 109 and a column-level storage buffer 110. The parser 105, preprocessor 106, query optimizer 107, query execution engine 108, row-level storage buffer 109, and column-level storage buffer 110 can be interconnected via communication lines. The communication lines can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication lines can be divided into address buses, data buses, control buses, etc.

[0062] The query engine 101 can receive query statements from the client 103. Specifically, the parser 105 in the query engine 101 receives the query statements from the client 103, and the parser 105 performs query analysis on the query statements to generate a parse tree. The query analysis includes at least lexical analysis, syntax analysis, and semantic analysis.

[0063] Parser 105 sends the generated parse tree to preprocessor 106, which rewrites the parse tree using a rule system and generates a new parse tree. The rule system consists of a series of rules, which can be categorized in two ways:

[0064] Based on the type of command used, rules can be divided into SELECT rules, UPDATE rules, INSERT rules, and DELETE rules.

[0065] The way actions are performed according to rules can be divided into INSTEAD rules and ALSO rules.

[0066] The preprocessor 106 sends the generated new parse tree to the query optimizer 107. The query optimizer 107 can determine the query plan tree corresponding to the new parse tree, select the query plan tree with the lowest time cost, and send the selected query plan tree with the lowest time cost to the query execution engine 108. The query execution engine 108 executes the query plan tree and retrieves the query results from the storage engine 102 according to the query plan tree. After that, the query results are sent to the client 103.

[0067] Storage engine 102 can be used to store data in order to provide query results to the query execution engine. Storage engine 102 can be connected to query engine 101 via a communication line, or it can be integrated with query engine 101. Specifically, storage engine 102 may include memory. Memory refers to the internal storage that directly exchanges data with query execution engine 108. It can read and write data at any time and is very fast, serving as temporary data storage for the operating system or other running programs. The data in memory can be obtained from external devices, input by the user, or generated by database system 100; there is no specific limitation.

[0068] The memory includes at least two types of storage, such as random access memory (RAM) and read-only memory (ROM). For example, RAM can be dynamic random access memory (DRAM) or storage class memory (SCM). Memory can also include other types of RAM, such as static random access memory (SRAM). For read-only memory, examples include programmable read-only memory (PROM) and erasable programmable read-only memory (EPROM). Additionally, the memory can be a dual in-line memory module (DIMM), i.e., a module composed of DRAM. In practical applications, the database system 100 can be configured with multiple memory modules of different types. This embodiment does not limit the number or type of memory.

[0069] The memory can be divided into a row-oriented storage buffer 109 and a column-oriented storage buffer 110. The row-oriented storage buffer 109 is used for row-based storage, meaning the data in the row-oriented storage buffer 109 is stored in a row-oriented manner. The column-oriented storage buffer 110 is used for column-oriented storage, meaning the data in the column-oriented storage buffer 110 is stored in a column-oriented manner.

[0070] When executing the query plan tree, the query execution engine 108 can read row-store data (i.e., data stored in a row-oriented storage mode) from the row-store buffer and / or read column-store data (i.e., data stored in a column-oriented storage mode) from the column-store buffer, and generate query results based on the obtained row-store data and / or column-store data.

[0071] Unlike RAM, persistent storage medium 104 reads and writes data more slowly and is typically used for persistent data storage. Persistent storage medium 104 can specifically be a disk or other type of storage medium, such as a solid-state drive (SSD), hard disk drive (HDD), shingled magnetic recording hard disk, compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but not limited to these.

[0072] The data in persistent storage medium 104 is stored in row-oriented storage. Query execution engine 108 can read row-oriented data from row-oriented storage buffer 109 and / or column-oriented data from column-oriented storage buffer 110. When row-oriented storage buffer 109 is not hit, query execution engine 108 can read data from persistent storage medium 104 into row-oriented storage buffer 109, and then read row-oriented data from row-oriented storage buffer 109.

[0073] When the column buffer 110 is not found, the query execution engine 108 can search in the row buffer 109. If the row buffer 109 is found, the query execution engine 108 can read data from the row buffer 109 into the column buffer 110, and then read the column data from the column buffer 110. If the row buffer 109 is not found, the query execution engine 108 can read data from the persistent storage medium 104 into the row buffer 109, and then read the data from the row buffer 109 into the column buffer 110, and then read the column data from the column buffer 110.

[0074] It should be noted that the structure of the database system 100 shown in Figure 1 is only an example. In actual applications, the database system 100 may have more or fewer components, and this application embodiment does not limit this.

[0075] The memory management method provided in this application embodiment will be described in detail below, taking the database system shown in Figure 1 as an example.

[0076] Figure 2 is a flowchart illustrating the memory management method provided in this embodiment. This method can be executed by a database system with memory management capabilities (such as database system 100), or by a module within that database system, or by the query engine 101 or storage engine 102 within that database system. For ease of explanation, the following description uses database system 100 as an example. As shown in Figure 2, the method includes:

[0077] S201, the database system determines the variable capacity of the first memory and the variable capacity of the second memory.

[0078] In this embodiment, the database system's memory can be divided into a first memory and a second memory. The first memory can also be referred to as the row storage buffer in Figure 1, where data is stored in a row-oriented manner. The second memory can also be referred to as the column storage buffer in Figure 1, where data is stored in a column-oriented manner.

[0079] Since the capacities of the first and second memory modules have multiple possibilities, the database first determines the variable capacities of both the first and second memory modules. These variable capacities are constantly being adjusted. Subsequently, the final capacities of the first and second memory modules are determined by executing the steps shown in Figure 2.

[0080] S202, the database system determines the reference capacity of the first memory corresponding to the variable capacity of the first memory, and the reference capacity of the second memory corresponding to the variable capacity of the second memory.

[0081] In the embodiments of this application, the reference capacity of the first memory and the reference capacity of the second memory are both constants.

[0082] In one possible implementation, the database system can determine the reference capacity of the first memory and the reference capacity of the second memory through implementation method A1 or A2.

[0083] In implementation method A1, the database system determines the reference capacity of the first memory and the reference capacity of the second memory through random initialization.

[0084] In the above-described embodiment A1, the reference capacity of the first memory and the reference capacity of the second memory can be any value. In order to facilitate subsequent adjustment of the reference capacity of the first memory and the reference capacity of the second memory, the reference capacity of the first memory and the reference capacity of the second memory can be any value close to 0.

[0085] The reference capacity of the first memory and the reference capacity of the second memory can be quickly determined through implementation method A1.

[0086] In implementation method A2, the database system divides the total memory capacity of the database system to determine a reference capacity for a first memory and a reference capacity for a second memory. The sum of the reference capacities for the first memory and the second memory is the total memory capacity.

[0087] In the above-described embodiment A2, the reference capacity of the first memory and the reference capacity of the second memory can be determined by embodiment B1, B2, or B3.

[0088] In implementation method B1, the database system divides the total memory capacity equally to obtain a reference capacity for a first memory and a reference capacity for a second memory. The reference capacity for the first memory and the reference capacity for the second memory are the same.

[0089] In implementation method B2, the database system randomly divides the total memory capacity to obtain a reference capacity for the first memory and a reference capacity for the second memory.

[0090] In implementation method B3, the database system uses the historical partitioning results of the total memory capacity as the reference capacity for both the first and second memory. The historical partitioning results of the total memory capacity include the historical capacity of the first memory and the historical capacity of the second memory; the historical capacity of the first memory serves as the reference capacity for the first memory, and the historical capacity of the second memory serves as the reference capacity for the second memory.

[0091] Since implementation method A2 determines the reference capacity of the first memory and the reference capacity of the second memory based on the total memory capacity of the database system, the determined reference capacity of the first memory will be closer to the actual capacity of the first memory, and the determined reference capacity of the second memory will also be closer to the actual capacity of the second memory. This facilitates the subsequent rapid determination of the capacity of the first memory based on the reference capacity of the first memory, and the determination of the capacity of the second memory based on the reference capacity of the second memory.

[0092] S203, the database system determines the total throughput of the database system based on the variable capacity of the first memory and the variable capacity of the second memory.

[0093] In one possible implementation, the database system can determine the total throughput of the database system through the following steps shown in Figure 3.

[0094] S301, the database system determines the transaction throughput of the database system based on the variable capacity of the first memory and the total amount of data stored in the persistent storage medium.

[0095] In one possible implementation, transaction throughput can be understood as TPS. The database system can determine the transaction throughput of the database system based on the ratio of the variable capacity of the first memory to the total amount of data stored in the persistent storage medium. The transaction throughput of the database system satisfies formula (1) with respect to the variable capacity of the first memory and the total amount of data stored in the persistent storage medium.

[0096] Among them, TPS(M r M represents the transaction processing throughput of a database system. r θ0, θ1, β1, and β0 represent the variable capacity of the first memory, DatabaseScale represents the total amount of data stored in the persistent storage medium, and θ0, θ1, β1, and β0 are all hyperparameters.

[0097] It should be understood that θ0, θ1, β1, and β0 can be preset values, or they can be determined by the database system through executing sample query statements within the database system. Sample query statements will be explained in detail later and will not be discussed here.

[0098] In the above formula (1), θ0, θ1, β1, β0, and DatabaseScale are all constants, and M r It is a variable, TPS(M) r With M r Adjustments are made accordingly.

[0099] In one possible implementation, the database system can determine the aforementioned hyperparameters by executing a sample query statement within the database system. Specifically, this can be achieved by: DatabaseScale being a constant, and M... r When set to different values, M can be determined by executing the sample query statement. r TPS(M) corresponding to different values r Then, based on DatabaseScale and different values ​​of M... r Different values ​​of M r Corresponding TPS(M) r By fitting the data, the specific values ​​of the hyperparameters θ0, θ1, β1, and β0 can be determined.

[0100] S302, the database system determines the analytical processing throughput of the database system based on the variable capacity of the second memory and the set of data columns.

[0101] In one possible implementation, the set of data columns in S302 can be determined by the variable capacity of the second memory and S sample query statements in the database system. The database system can generate S sample query statements based on N template query statements. Template query statements are non-executable query statements; one template query statement can generate one or more executable sample query statements.

[0102] For example, the template query statement is as follows:

[0103] The query is: `SELECT SUM(EXTENDED PRIVATE) FROM LINEItem WHERE DATE >= DATE'[DATE]'`, where DATE can be any January 1st from [2003, 2023].

[0104] In the above template query, `lineitem` is the table name, and `extendedprice` and `shipdate` are column names in the `lineitem` table. `extendedprice` is described as "total price," and `shipdate` is described as "shipping date." Furthermore, the `lineitem` table also includes data corresponding to the `extendedprice` column. `sum()` is an aggregate function; `sum(extendedprice)` calculates the sum of the data corresponding to the `extendedprice` column in the `lineitem` table. The `lineitem` table also includes data corresponding to the `shipdate` column. The condition `shipdate >= date[DATE]` is used to determine if the data corresponding to the `shipdate` column in the `lineitem` table is greater than or equal to `date[DATE]`.

[0105] A sample query statement generated from the above template query statement can be as follows:

[0106] select sum(extendedprice)from lineitem where shipdate>=date'2023-1-1'

[0107] After executing the above sample query, the query result is the sum of the data corresponding to the column name extendedprice in the data table lineitem, provided that the data corresponding to the column name shipdate in the data table lineitem is greater than or equal to date'2023-1-1'.

[0108] In this embodiment of the application, since the variable capacity of the second memory is a variable, the reference capacity of the second memory is one of the constants corresponding to the variable capacity of the second memory. Subsequently, the database system determines the data column set in S302 based on the reference capacity of the second memory and the S sample query statements in the database system. Specifically, the database system can determine the data column set through implementation method C1 or C2.

[0109] Implementation method C1 includes the following steps as shown in FIG4.

[0110] S401, the database system obtains the reference capacity of the second memory and S sample query statements in the database system.

[0111] S402, the database system determines the objective function and its constraints based on the reference capacity of the second memory and S sample query statements.

[0112] In one possible implementation, the objective function can be that the time benefit corresponding to each of the S sample query statements satisfies a first condition. Specifically, the time benefit of a single sample query statement is determined by the difference between the time required to read the corresponding data column from persistent storage and the time required to read it from secondary memory.

[0113] It should be understood that the time efficiency of any sample query statement can be directly determined by the query optimizer 107, which will not be elaborated on here.

[0114] The first condition can be any of the following:

[0115] (1) The first condition can be that the sum of the time benefits corresponding to the S sample query statements is maximized;

[0116] (2) The first condition can be that the sum of the time benefits corresponding to the S sample query statements is greater than the first preset value;

[0117] (3) The first condition can be that the sum of the time benefits corresponding to at least one of the S sample query statements is maximized;

[0118] (4) The first condition can be that the sum of the time benefits corresponding to at least one of the S sample query statements is greater than the first preset value.

[0119] It should be understood that, for the sake of clarity in the following description, the first condition is that the sum of the time benefits corresponding to the S sample query statements is maximized, which will not be elaborated upon in the following text.

[0120] In one possible implementation, the constraints may include at least one of the following:

[0121] (1) The total amount of the selected data columns to be loaded into the second memory satisfies the second condition with the reference capacity of the second memory; wherein, the second condition may be that the total amount of the selected data columns to be loaded into the second memory is less than or equal to the reference capacity of the second memory.

[0122] (2) If all the data columns corresponding to one of the S sample query statements are selected, the data column corresponding to that sample query statement can be read from the second memory.

[0123] In this embodiment of the application, the data column corresponding to one of the S sample query statements can only be read from the second memory if all of the data columns corresponding to that sample query statement are selected; if none of the data columns corresponding to one of the S sample query statements are selected, the data columns corresponding to that sample query statement cannot be read from the second memory.

[0124] In the embodiments of this application, the objective function satisfies formula (2), and the constraint conditions of the objective function satisfy formula (3).

[0125] In formula (2), S represents the number of sample query statements, and Q j Z represents the time benefit of the j-th sample query statement, where the time benefit of the j-th sample query statement is determined by the difference between the time required to read the data column corresponding to the j-th sample query statement from persistent storage and the time required to read it from secondary memory. j Indicates whether the j-th sample query statement is scanned from the second memory. If Z j =0, which means that the j-th sample query statement scans from the first memory; if Z = 0, it means that the j-th sample query statement scans from the first memory; j =1 indicates that the j-th sample query statement is scanned from the second memory.

[0126] In formula (3), N represents the number of columns storing data in the persistent storage medium, and M c (i) represents the storage size of the i-th data column, x i Indicates whether to select the i-th data column; if x i =0, which means that the i-th data column is not selected; if x i =1 indicates that the i-th data column is selected, M c This indicates the reference capacity of the second memory.

[0127] G j Z represents the data column corresponding to the j-th sample query statement, S represents the number of sample query statements, and Z represents the number of sample query statements.j Indicates whether the j-th sample query statement is scanned from the second memory. If Z j =0, which means that the j-th sample query statement scans from the first memory; if Z = 0, it means that the j-th sample query statement scans from the first memory; j =1 indicates that the j-th sample query statement is scanned from the second memory.

[0128] S403, the database system solves the objective function based on the constraints to obtain a set of data columns.

[0129] In S403, the database system uses a linear solver to solve the objective function according to the constraints to obtain a set of data columns. The linear solver can be generated by any method such as branch and bound method or cutting plane method, and there is no limitation here.

[0130] Through the steps shown in Figure 4, the data column set determined by the database system includes at least one data column, wherein the data column is a column of data selected from the persistent storage medium. The data column set is subsequently loaded into the second memory. The determined data column set helps to ensure that the time benefit corresponding to the S sample query statements meets the first condition, thereby improving the execution efficiency of subsequent query statements.

[0131] Furthermore, in the steps shown in Figure 4, the database system directly determines the objective function and its constraints based on the reference capacity of the second memory and the S sample query statements, without splitting the sample query statements. This allows for the rapid acquisition of the objective function and its constraints, and consequently, the rapid acquisition of the data column set.

[0132] Implementation method C2 includes the following steps as shown in FIG5.

[0133] S501, the database system obtains the reference capacity of the second memory and S sample query statements in the database system.

[0134] S502, the database system splits the S sample query statements into K subquery statements.

[0135] In S502 above, the S sample query statements can be split into K subquery statements by the preprocessor 106 in the database system. The detailed splitting process will not be described in detail here.

[0136] In one possible implementation, a sample query statement can be split into one or more subquery statements. When splitting a sample query statement, the database system can also generate a splitting relationship, which records the correspondence between each sample query statement and the K subquery statements split from each sample query statement.

[0137] S503, the database system determines the objective function and its constraints based on the reference capacity of the second memory and the K subqueries.

[0138] In one possible implementation, the objective function can be that the time benefit of each of the K subqueries satisfies a first condition, wherein the time benefit of one of the K subqueries is determined by the difference between the time required to read the corresponding data column from persistent storage and the time required to read it from secondary memory. Specifically, the time benefit of a subquery can be the difference between the time required to read the corresponding data column from persistent storage and the time required to read it from secondary memory.

[0139] It should be understood that the time benefit of any subquery statement can be directly determined by the query optimizer 107, which will not be elaborated on here.

[0140] The first condition can be any of the following:

[0141] (1) The first condition can be that the sum of the time benefits corresponding to the K subqueries is maximized;

[0142] (2) The first condition can be that the sum of the time benefits corresponding to the K subqueries is greater than the first preset value;

[0143] (3) The first condition can be that the sum of the time benefits corresponding to at least one of the K subquery statements is maximized;

[0144] (4) The first condition can be that the sum of the time benefits corresponding to at least one of the K subquery statements is greater than the first preset value.

[0145] It should be understood that, for the sake of clarity in the following description, the first condition is that the sum of the time benefits corresponding to the K subqueries is maximized, which will not be elaborated upon in the following text.

[0146] In one possible implementation, the constraints may include at least one of the following:

[0147] (1) The total amount of the selected data columns to be loaded into the second memory satisfies the second condition with the reference capacity of the second memory; wherein, the second condition may be that the total amount of the selected data columns to be loaded into the second memory is less than or equal to the reference capacity of the second memory.

[0148] (2) If all the data columns corresponding to one of the K subqueries are selected, the data columns corresponding to that subquery can be read from the second memory.

[0149] In this embodiment of the application, the data column corresponding to one of the K subqueries can only be read from the second memory if all of its data columns are selected to be loaded into the second memory; if none of its data columns are selected to be loaded into the second memory, the data column corresponding to the subquery cannot be read from the second memory.

[0150] In one possible implementation, the objective function satisfies formula (4), and the constraint conditions of the objective function satisfy formula (5).

[0151] In formula (4), K represents the number of subquery statements, and Q... j Z represents the time benefit of the j-th subquery statement, where the time benefit of the j-th subquery statement is determined by the difference between the time required to read the data column corresponding to the j-th subquery statement from persistent storage and the time required to read it from secondary memory. j Indicates whether the j-th subquery statement scans from the second memory region. If Z j =0, which means that the j-th subquery statement scans from the first memory location; if Z = 0, it means that the j-th subquery statement scans from the first memory location. j =1 indicates that the j-th subquery statement scans from the second memory.

[0152] In formula (5), N represents the number of columns storing data in the persistent storage medium, and M c (i) represents the storage size of the i-th data column, x i Indicates whether to select the i-th data column; if x i =0, which means that the i-th data column is not selected; if x i =1 indicates that the i-th data column is selected, M c This indicates the reference capacity of the second memory.

[0153] G j Z represents the data column corresponding to the j-th subquery statement, S represents the number of subquery statements, and Z represents the number of subquery statements. j Indicates whether the j-th subquery statement scans from the second memory region. If Z j =0, which means that the j-th subquery statement scans from the first memory location; if Z = 0, it means that the j-th subquery statement scans from the first memory location. j =1 indicates that the j-th subquery statement scans from the second memory.

[0154] S504, the database system solves the objective function based on the constraints to obtain a set of data columns.

[0155] In S504, the database system uses a linear solver to solve the objective function according to the constraints to obtain a set of data columns. The linear solver can be generated by any method such as branch and bound method or cutting plane method, and there is no limitation here.

[0156] Through the steps shown in Figure 5, the set of data columns determined by the database system includes at least one data column, wherein the data column is a column of data selected from the persistent storage medium. The set of data columns is subsequently loaded into the second memory. The determined set of data columns helps to ensure that the time benefits corresponding to the K subquery statements meet the first condition, thereby improving the execution efficiency of subsequent query statements.

[0157] Furthermore, in the steps shown in Figure 5, the database system first splits the S sample query statements to obtain K subquery statements. Then, based on the reference capacity of the second memory and the K subquery statements, it determines the objective function and the constraints of the objective function. The determined objective function and the constraints of the objective function are more accurate, and thus the determined set of data columns is more accurate.

[0158] In S302 above, the database system can determine the analytical processing throughput of the database system according to the following steps shown in Figure 6.

[0159] S601, Given a set of data columns, the database system determines the time cost corresponding to each of the S sample query statements.

[0160] The set of data columns in S601 above is determined through the steps shown in Figure 4 or Figure 5. The time cost of one of the S sample query statements is determined based on the time required to read the corresponding data column from the persistent storage medium or the second memory. If all data columns corresponding to a sample query statement are selected to be loaded into the second memory, then all data columns corresponding to that sample query statement are read from the second memory, and the time cost of that sample query statement is the time required to read the corresponding data column from the second memory. If no data column corresponding to a sample query statement is selected to be loaded into the second memory, then all data columns corresponding to that sample query statement are read from the persistent storage medium, and the time cost of that sample query statement is the time required to read the corresponding data column from the persistent storage medium.

[0161] S602, the database system determines the analysis and processing throughput based on the time cost corresponding to each of the S sample query statements.

[0162] In one possible implementation, the database system can determine the analysis and processing throughput based on the sum of the time costs corresponding to the S sample query statements; or, the database system can determine the analysis and processing throughput based on the sum of the time costs corresponding to at least one of the S sample query statements.

[0163] The following details how a database system determines the analysis and processing throughput based on the sum of the time costs corresponding to S sample query statements. The method for determining the analysis and processing throughput based on the sum of the time costs corresponding to at least one of the S sample query statements can be found in the documentation.

[0164] In one possible implementation, the throughput of the database system's analysis and processing is inversely proportional to the sum of the time costs corresponding to the S sample query statements; that is, the larger the sum of the time costs corresponding to the S sample query statements, the smaller the throughput of the database system's analysis and processing. The throughput of the database system's analysis and processing is inversely proportional to the sum of the time costs corresponding to the S sample query statements, as per formula (6).

[0165] Among them, QPS(M c M represents the analytical processing throughput of a database system. c Indicates the reference capacity of the second memory, C represents the data column set, S represents the number of sample query statements, and Cost(M) c (C,S) indicates that the reference capacity of the second memory is M. c And, given a data column set of C, the sum of the time costs corresponding to the S sample query statements, where the reference capacity M of the second memory... c The choice of the data column set C is influenced by both α and γ, which are hyperparameters.

[0166] It should be understood that α and γ can be preset values, or they can be determined by the database system through executing sample query statements in the database system.

[0167] In one possible implementation, the database system can determine the aforementioned hyperparameters by executing a sample query statement within the database system. Specifically, this can be achieved by: setting M... c When set to different values, M can be determined by executing the sample query statement. c Cost(M) corresponding to different values c (C,S), then, according to different values ​​of M c Different values ​​of M c Corresponding Cost (M) cBy fitting the hyperparameters α and γ to C and S, the specific values ​​of the hyperparameters α and γ can be determined.

[0168] S303, the database system determines the total throughput of the database system based on the transaction processing throughput and the analytical processing throughput.

[0169] In S303 above, the database system can determine the total throughput of the database system through implementation method D1 or D2.

[0170] In implementation method D1, the database system determines the total throughput of the database system based on the product of transaction processing throughput and analysis processing throughput.

[0171] In one possible implementation, the total throughput of the database system is directly proportional to both transaction processing throughput and analytical processing throughput. The total throughput of the database system satisfies any one of the following four formulas in relation to transaction processing throughput and analytical processing throughput: f = TPS(M r )*QPS(M c (7) f=δ0*TPS(M r )*QPS(M c )+δ1 (8)

[0172] Where f represents the total throughput of the database system, TPS(M r QPS (Quantum Per Second) represents the transaction throughput of a database system. c ) represents the analytical processing throughput of the database system, where δ0 and δ1 are hyperparameters.

[0173] In implementation method D2, the database system determines the total throughput of the database system based on the sum of transaction processing throughput and analytical processing throughput.

[0174] In one possible implementation, the total throughput of the database system is directly proportional to both transaction processing throughput and analytical processing throughput. The total throughput of the database system satisfies any one of the following four formulas in relation to transaction processing throughput and analytical processing throughput: f = TPS(M r )+QPS(M c (13) f=δ0(TPS(M) r )+QPS(M c ))+δ1 (14)

[0175] Where f represents the total throughput of the database system, TPS(M r QPS (Quantum Per Second) represents the transaction throughput of a database system. c ) represents the analytical processing throughput of the database system, where δ0 and δ1 are hyperparameters.

[0176] S204, the database system uses a first algorithm to calculate the first variable capacity based on the total throughput to obtain the first unprocessed capacity, and obtains the first capacity based on the first reference capacity and the first unprocessed capacity, wherein the first capacity is used to maximize the total throughput of the database system.

[0177] In the embodiments of this application, the first variable capacity is one of the variable capacity of the first memory and the variable capacity of the second memory; the first reference capacity is one of the reference capacity of the first memory and the reference capacity of the second memory; and the first capacity is one of the capacity of the first memory and the capacity of the second memory.

[0178] When the first variable capacity is the variable capacity of the first memory, the first reference capacity is the reference capacity of the first memory, and the first capacity is the capacity of the first memory. When the first variable capacity is the variable capacity of the second memory, the first reference capacity is the reference capacity of the second memory, and the first capacity is the capacity of the second memory.

[0179] In S204 above, the first algorithm can be gradient ascent or other methods, which are not limited here.

[0180] In one possible implementation, if the first algorithm is the gradient ascent method, the database system can obtain the first capacity through implementation method E1 or E2.

[0181] Implementation method E1 includes the following steps as shown in FIG7.

[0182] S701, the database system iteratively executes the following steps until the iteration stop condition is met: the database system calculates the gradient estimate of the total throughput with respect to the variable capacity of the first memory to obtain the first unprocessed capacity; the database system processes the first unprocessed capacity based on the reference capacity of the first memory to obtain the adjustment amount of the first memory; the database system obtains the updated reference capacity of the first memory based on the reference capacity of the first memory and the adjustment amount of the first memory.

[0183] In S701 above, an iteration process may include the following steps as shown in Figure 8.

[0184] S801, the database system uses the forward difference of the total throughput with respect to the variable capacity of the first memory as the gradient estimate of the total throughput with respect to the variable capacity of the first memory. This gradient estimate of the total throughput with respect to the variable capacity of the first memory is the first processing capacity. The first processing capacity includes the variable capacity of the first memory, and the first processing capacity adjusts as the variable capacity of the first memory is adjusted.

[0185] S802, the database system replaces the variable capacity of the first memory in the first unprocessed capacity with the reference capacity of the first memory to obtain the adjustment amount of the first memory.

[0186] S803, the database system uses the sum of the reference capacity of the first memory and the adjustment amount of the first memory as the updated reference capacity of the first memory.

[0187] In the above S803, if the sum of the reference capacity of the first memory and the adjustment amount of the first memory is greater than the total memory capacity, then the total memory capacity is used as the updated reference capacity of the first memory; otherwise, the sum of the reference capacity of the first memory and the adjustment amount of the first memory is used as the updated reference capacity of the first memory.

[0188] The steps shown in Figure 8 provide a specific iterative process.

[0189] In S701 above, the iteration stopping condition includes at least one of the following:

[0190] (1) The adjustment amount of the first memory is less than the second preset value;

[0191] (2) The number of iterations is greater than or equal to the third preset value.

[0192] S702, the database system uses the reference capacity of the first memory that meets the iteration stopping condition as the first capacity.

[0193] After executing S702, if the difference between the total memory capacity and the first capacity is less than 0, then 0 is used as the second capacity; otherwise, the difference between the total memory capacity and the first capacity is used as the second capacity.

[0194] In the steps shown in Figure 7, the adjustment amount of the first memory is calculated, and then the reference capacity of the first memory is updated according to the adjustment amount of the first memory in order to obtain the first capacity. The steps shown in Figure 7 provide a method for obtaining the first capacity.

[0195] Implementation method E2 includes the following steps as shown in FIG9.

[0196] S901, the database system iteratively executes the following steps until the iteration stop condition is met: the database system calculates the gradient estimate of the total throughput with respect to the variable capacity of the second memory to obtain the first unprocessed capacity; the database system processes the first unprocessed capacity based on the reference capacity of the second memory to obtain the adjustment amount of the second memory; the database system obtains the updated reference capacity of the second memory based on the reference capacity of the second memory and the adjustment amount of the second memory.

[0197] The iterative process and iterative stopping condition in S901 above can be referred to the iterative process and iterative stopping condition in S701, and will not be repeated here.

[0198] S902, the database system uses the reference capacity of the second memory that meets the iteration stopping condition as the first capacity.

[0199] After executing S902, if the difference between the total memory capacity and the first capacity is less than 0, then 0 is used as the second capacity; otherwise, the difference between the total memory capacity and the first capacity is used as the second capacity.

[0200] In the steps shown in Figure 9, the adjustment amount of the second memory is calculated, and then the reference capacity of the second memory is updated according to the adjustment amount of the second memory in order to obtain the first capacity. The steps shown in Figure 9 provide a method for obtaining the first capacity.

[0201] S205, the database system obtains a second capacity based on the first capacity and the total memory capacity of the database system; the second capacity is one of the first memory capacity and the second memory capacity, and the second capacity is different from the first capacity.

[0202] In S205, after determining the first capacity, the database system uses the difference between the total memory capacity of the database system and the first capacity as the second capacity.

[0203] In the steps shown in Figure 2, the process of dividing the total memory capacity of the database system into the capacity of the first memory and the capacity of the second memory does not rely on human experience, but is based on the total throughput of the database system. This can divide the total memory capacity of the database system more efficiently and accurately, thereby improving the efficiency and accuracy of memory management.

[0204] In this embodiment of the application, after determining the capacity of the first memory and the capacity of the second memory through the memory management method shown in Figure 2, the data column set can be loaded during the execution of the query statement. Therefore, this embodiment of the application also provides a query statement execution method, including the following steps shown in Figure 10.

[0205] S1001, the database system obtains the query statement to be executed and determines the data columns corresponding to the query statement to be executed.

[0206] For example, the query statement to be executed is as follows:

[0207] select sum(extendedprice)from lineitem where shipdate>=date'2023-1-1'

[0208] The data columns corresponding to the query statement to be executed include the data column corresponding to the column name extendedprice and the data column corresponding to the column name shipdate.

[0209] S1002, if the data column corresponding to the query statement to be executed is located in the data column set, the database system will load the data column corresponding to the query statement to be executed from persistent storage into the second memory.

[0210] S1003, the database system executes the query statement to be executed based on the data columns corresponding to the query statement to be executed located in the second memory.

[0211] In the steps shown in Figure 10 above, if the data column corresponding to the query statement to be executed is located in the data column set during the execution of the query statement, the data column corresponding to the query statement to be executed is loaded into the second memory. There is no need to use extra time to load the data column set, which can effectively realize the seamless loading of the data column set.

[0212] In this embodiment of the application, after determining the capacity of the first memory and the capacity of the second memory through the memory management method shown in FIG2, the capacity of the first memory and the capacity of the second memory can be re-determined according to the execution status of the query statement to be executed within the time period. Therefore, this application also provides a memory management method, including the following steps shown in FIG11.

[0213] S1101, the database system obtains the first feature vector.

[0214] In one possible implementation, the first feature vector is used to mark the frequency information of the query statement to be executed within the first time period. The frequency information of the query statement to be executed includes the frequency of the template query statement corresponding to the query statement to be executed and / or the frequency of the data column corresponding to the query statement to be executed.

[0215] For any query statement to be executed, the template query statement corresponding to the query statement to be executed can be determined, and the frequency of the template query statement corresponding to the query statement to be executed can be increased by 1; the data column corresponding to the query statement to be executed can also be determined, and the frequency of the data column corresponding to the query statement to be executed can be increased by 1.

[0216] After performing the above operations on all the query statements to be executed in the first time period, we can obtain a template query statement vector composed of the frequency of each template query statement, and a data column vector composed of the frequency of each data column.

[0217] It should be understood that the database system can update the template query statement vector and data column vector during the execution of the query statement to be executed, so as to improve the efficiency of generating the first feature vector later.

[0218] In one possible implementation, the database system can generate a first feature vector based on a template query statement vector and / or a data column vector through any of the implementations F1, F2, F3, or F4.

[0219] In implementation F1, the database system uses the template query statement vector as the first feature vector.

[0220] In implementation F2, the database system uses the data column vector as the first feature vector.

[0221] In implementation method F3, the database system can concatenate the template query statement vector and the data column vector to obtain the first feature vector.

[0222] In implementation F3, the template query statement vector is a 1×S vector, and when the persistent storage medium includes N columns of stored data, the data column vector is a 1×N vector. The template query statement vector and the data column vector are concatenated to obtain a first feature vector of 1×(S+N). The template query statement vector in the first feature vector can be located before or after the data column vector; this is not limited here.

[0223] In implementation F4, the database system can perform matrix multiplication on the template query statement vector and the data column vector to obtain the first feature vector.

[0224] In implementation F4, the template query statement vector is a 1×S vector and the data column vector is a 1×N vector. When the dimensions of the template query statement vector and the data column vector are the same, the database system can perform matrix dot product on the template query statement vector and the data column vector to obtain the first feature vector.

[0225] S1102, the database system obtains the second feature vector.

[0226] In one possible implementation, the second feature vector is used to mark the frequency information of the query statements to be executed within the second time period. The generation of the second feature vector can refer to the generation of the first feature vector, and will not be elaborated here.

[0227] In one possible implementation, the second time period and the first time period satisfy the following relationship: the second time period can be any time period preceding the first time period. The second time period can be adjacent to the first time period or not, which is not limited here.

[0228] S1103, if the similarity between the first feature vector and the second feature vector is less than a first threshold, the database system determines the capacity of the first memory and the capacity of the second memory.

[0229] In one possible implementation, the similarity between the first feature vector and the second feature vector can be determined by at least one of the following:

[0230] (1) Euclidean distance between the first eigenvector and the second eigenvector;

[0231] (2) The cosine of the angle between the first eigenvector and the second eigenvector;

[0232] (3) The Manhattan distance between the first eigenvector and the second eigenvector;

[0233] (4) Chebyshev distance between the first eigenvector and the second eigenvector;

[0234] (5) Hamming distance between the first eigenvector and the second eigenvector.

[0235] It should be understood that the method for determining the capacity of the first memory and the capacity of the second memory in S1103 can refer to the steps shown in Figure 2, and will not be repeated here.

[0236] In the steps shown in Figure 11, when a load shift occurs, the capacity of the first memory and the capacity of the second memory can be adjusted in a timely manner. This allows for a more efficient and accurate allocation of the total memory capacity of the database system, thereby improving the efficiency and accuracy of memory management. Simultaneously, it can identify the set of data columns with higher time benefits, which can further improve the execution efficiency of subsequent query statements.

[0237] Based on the steps shown in Figure 2, this application embodiment provides an updated database system. In addition to the various units shown in the database system of Figure 1, the updated database system may further include a monitoring module 111 and a memory allocation module 112. The monitoring module 111 can communicate with the memory allocation module 112. The memory allocation module 112 includes a data column configuration module 113. The updated database system is shown in Figure 12.

[0238] The monitoring module 111 is used to acquire the first feature vector and the second feature vector, and determine whether the similarity between the first feature vector and the second feature vector is less than a first threshold. If it is determined that the similarity between the first feature vector and the second feature vector is less than the first threshold, the monitoring module 111 sends a memory allocation instruction to the memory allocation module 112. The memory allocation instruction is used to instruct the memory allocation module 112 to redetermine the capacity of the first memory and the capacity of the second memory.

[0239] The memory allocation module 112 is used to receive memory allocation instructions and determine the capacity of the first memory and the capacity of the second memory according to the memory allocation instructions.

[0240] Specifically, the memory allocation module 112 is used to determine the total throughput of the database system based on the variable capacity of the first memory and the variable capacity of the second memory; the first memory is used for row-oriented storage; both the variable capacity of the first memory and the variable capacity of the second memory are constantly adjusted variables; a reference capacity of the first memory corresponding to the variable capacity of the first memory and a reference capacity of the second memory corresponding to the variable capacity of the second memory are determined; both the reference capacity of the first memory and the reference capacity of the second memory are constants; a first algorithm is used to calculate the first variable capacity based on the total throughput to obtain a first processing capacity, and a first capacity is obtained based on the first reference capacity and the first processing capacity; the first capacity is used to maximize the total throughput of the database system; the first variable capacity is one of the variable capacity of the first memory and the variable capacity of the second memory; the first reference capacity is one of the reference capacity of the first memory and the reference capacity of the second memory; the first capacity is one of the capacity of the first memory and the capacity of the second memory; the second capacity is one of the capacity of the first memory and the capacity of the second memory, and the second capacity is different from the first capacity.

[0241] Specifically, the data column configuration module 113 is used to determine the data column set based on the reference capacity of the second memory and S sample query statements.

[0242] Based on the above embodiments, this application also provides a memory management device, which is applied to the database system shown in FIG12 to implement the memory management method provided in the embodiments shown in FIG2 and FIG4. Referring to FIG13, the memory management device includes: an acquisition module 1301, a determination module 1302, a calculation module 1303, a loading module 1304, and an execution module 1305. Wherein:

[0243] The acquisition module 1301 is used to acquire the reference capacity of the second memory and S sample query statements in the database system; the second memory is used for columnar storage.

[0244] The determination module 1302 is used to determine the objective function and the constraints of the objective function based on the reference capacity of the second memory and the S sample query statements; the objective function is that the time benefit corresponding to each of the S sample query statements satisfies the first condition; the time benefit of one sample query statement among the S sample query statements is determined based on the difference between the time required to read the data column corresponding to the sample query statement from the persistent storage medium and the time required to read it from the second memory;

[0245] The acquisition module 1301 is also used to solve the objective function according to the constraints to obtain a set of data columns; the set of data columns includes at least one data column, which is a column of stored data selected from the persistent storage medium and used to be loaded into the second memory.

[0246] In one possible implementation, the constraints include at least one of the following: the total number of selected data columns to be loaded into the second memory and the reference capacity of the second memory satisfy a second condition; and the data column corresponding to one of the S sample query statements can be read from the second memory when all the data columns corresponding to one of the sample query statements are selected.

[0247] In one possible implementation, the determining module 1302 is specifically used to: split the S sample query statements to obtain K subquery statements; determine the objective function and the constraints of the objective function based on the reference capacity of the second memory and the K subquery statements; the objective function is that the time benefits corresponding to the K subquery statements satisfy the first condition.

[0248] In one possible implementation, the constraints include at least one of the following: the total number of selected data columns to be loaded into the second memory and the reference capacity of the second memory satisfy a second condition; and the data column corresponding to one of the K subqueries can be read from the second memory if all the data columns corresponding to one of the subqueries are selected.

[0249] In one possible implementation, the determining module 1302 is further configured to determine the total throughput of the database system based on the variable capacity of the first memory and the variable capacity of the second memory; the first memory is used for row-oriented storage; both the variable capacity of the first memory and the variable capacity of the second memory are continuously adjusted variables; the determining module 1302 is further configured to determine a reference capacity of the first memory corresponding to the variable capacity of the first memory, and a reference capacity of the second memory corresponding to the variable capacity of the second memory; both the reference capacity of the first memory and the reference capacity of the second memory are constants; the calculation module 1303 is configured to calculate the first variable capacity based on the total throughput using a first algorithm. The system obtains a first processing capacity and a first capacity based on a first reference capacity and the first processing capacity; the first capacity is used to maximize the total throughput of the database system; the first variable capacity is one of the variable capacity of the first memory and the variable capacity of the second memory; the first reference capacity is one of the reference capacity of the first memory and the reference capacity of the second memory; the first capacity is one of the capacity of the first memory and the capacity of the second memory; the calculation module 1303 is also used to obtain a second capacity based on the first capacity and the total memory capacity of the database system; the second capacity is one of the capacity of the first memory and the capacity of the second memory, and the second capacity is different from the first capacity.

[0250] In one possible implementation, the determining module 1302 is specifically used to: determine the transaction processing throughput of the database system based on the variable capacity of the first memory and the total amount of data stored in the persistent storage medium; determine the analysis processing throughput of the database system based on the variable capacity of the second memory and the data column set; and determine the total throughput of the database system based on the transaction processing throughput and the analysis processing throughput.

[0251] In one possible implementation, the determining module 1302 is specifically used to: determine the time cost corresponding to each of the S sample query statements in the case of a set of data columns; the time cost of one of the S sample query statements is determined based on the time required to read the data column corresponding to the sample query statement from the persistent storage medium or the second memory; and determine the analysis processing throughput based on the time cost corresponding to each of the S sample query statements.

[0252] In one possible implementation, the calculation module 1303 is specifically used to: iteratively execute the following steps until the iteration stopping condition is met: calculate the gradient estimate of the total throughput with respect to the variable capacity of the first memory to obtain the first capacity to be processed; process the first capacity to be processed based on the reference capacity of the first memory to obtain the adjustment amount of the first memory; obtain the updated reference capacity of the first memory based on the reference capacity of the first memory and the adjustment amount of the first memory; and use the reference capacity of the first memory that meets the iteration stopping condition as the first capacity.

[0253] In one possible implementation, the calculation module 1303 is specifically used to: determine the updated reference capacity of the first memory based on the sum of the reference capacity of the first memory and the adjustment amount of the first memory.

[0254] In one possible implementation, the calculation module 1303 is specifically used to: iteratively execute the following steps until the iteration stopping condition is met: calculate the gradient estimate of the total throughput with respect to the variable capacity of the second memory to obtain a first capacity to be processed; process the first capacity to be processed based on the reference capacity of the second memory to obtain an adjustment amount of the second memory; obtain an updated reference capacity of the second memory based on the reference capacity of the second memory and the adjustment amount of the second memory; and use the reference capacity of the second memory that meets the iteration stopping condition as the first capacity.

[0255] In one possible implementation, the calculation module 1303 is specifically used to: determine the updated reference capacity of the second memory based on the sum of the reference capacity of the second memory and the adjustment amount of the second memory.

[0256] In one possible implementation, the acquisition module 1301 is further configured to acquire the query statement to be executed and determine the data column corresponding to the query statement to be executed; the loading module 1304 is configured to load the data column corresponding to the query statement to be executed from the persistent storage medium into the second memory if the data column corresponding to the query statement to be executed is located in the data column set; and the execution module 1305 is configured to execute the query statement to be executed according to the data column corresponding to the query statement to be executed located in the second memory.

[0257] In one possible implementation, the acquisition module 1301 is further configured to acquire a first feature vector, which is used to mark the frequency information of the query statement to be executed within a first time period. The frequency information of the query statement to be executed includes the frequency of the template query statement corresponding to the query statement to be executed and / or the frequency of the data column corresponding to the query statement to be executed. The acquisition module 1301 is further configured to acquire a second feature vector, which is used to mark the frequency information of the query statement to be executed within a second time period. The second time period is located before the first time period. The determination module 1302 is further configured to determine the capacity of the first memory and the capacity of the second memory if the similarity between the first feature vector and the second feature vector is less than a first threshold.

[0258] In the embodiments of this application, the acquisition module, determination module, calculation module, loading module, and execution module can all be implemented in software or in hardware. For example, the implementation of the acquisition module will be described below. Similarly, the implementation of the determination module, calculation module, loading module, and execution module can refer to the implementation of the acquisition module.

[0259] As an example of a software functional unit, a module can include code running on a computing instance. A computing instance can include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instance can be one or more. For example, a module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.

[0260] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0261] As an example of a hardware functional unit, an acquisition module may include at least one computing device, such as a server. Alternatively, an acquisition module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0262] The acquisition module includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the acquisition module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the acquisition module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0263] It should be noted that, in other embodiments, the acquisition module is used to acquire the reference capacity of the second memory and S sample query statements in the database system; the second memory is used for columnar storage; the determination module is used to determine the objective function and the constraints of the objective function based on the reference capacity of the second memory and the S sample query statements; the objective function is that the time benefit corresponding to each of the S sample query statements satisfies the first condition; the time benefit of one sample query statement among the S sample query statements is determined based on the difference between the time required to read the data column corresponding to the sample query statement from the persistent storage medium and the time required to read it from the second memory; the acquisition module is also used to solve the objective function according to the constraints to obtain a set of data columns; the set of data columns includes at least one data column, which is a column of stored data selected from the persistent storage medium and used to be loaded into the second memory.

[0264] The steps implemented by the acquisition module, determination module, calculation module, loading module, and execution module can be specified as needed. By implementing different steps in the memory management method through the acquisition module, determination module, calculation module, loading module, and execution module, the full functionality of the memory management device can be achieved.

[0265] This application also provides a computing device, which includes a processor and a memory, wherein the memory stores computer-executable instructions; the processor is used to execute the computer-executable instructions stored in the memory, and the processor can execute memory management methods.

[0266] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing system or stored on any usable medium. When the computer program product runs on at least one computing system, it causes the at least one computing system to perform a memory management method.

[0267] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing system can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing system to perform a memory management method.

[0268] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

[0269] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0270] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0271] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0272] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0273] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A memory management method, characterized in that, include: Obtain the reference capacity of the second memory and S sample query statements in the database system; The second memory is used for columnar storage; Based on the reference capacity of the second memory and the S sample query statements, determine the objective function and the constraints of the objective function; The objective function is that the time benefit corresponding to each of the S sample query statements satisfies the first condition; the time benefit of one of the S sample query statements is determined based on the difference between the time required to read the data column corresponding to the sample query statement from the persistent storage medium and the time required to read it from the second memory; The objective function is solved according to the constraints to obtain the data column set; the data column set includes at least one data column, which is a column of stored data selected from the persistent storage medium and used to be loaded into the second memory.

2. The method as described in claim 1, characterized in that, The constraints include at least one of the following: The total amount of the selected data columns to be loaded into the second memory satisfies the second condition with respect to the reference capacity of the second memory; If all the data columns corresponding to one of the S sample query statements are selected, the data column corresponding to that sample query statement can be read from the second memory.

3. The method as described in claim 1, characterized in that, The step of determining the objective function and its constraints based on the reference capacity of the second memory and the S sample query statements includes: Each of the S sample query statements is split into K sub-query statements; Based on the reference capacity of the second memory and the K subqueries, the objective function and its constraints are determined; the objective function is that the time benefit corresponding to each of the K subqueries satisfies the first condition.

4. The method as described in claim 3, characterized in that, The constraints include at least one of the following: The total amount of the selected data columns to be loaded into the second memory satisfies the second condition with respect to the reference capacity of the second memory; If all the data columns corresponding to one of the K subqueries are selected, the data column corresponding to that subquery can be read from the second memory.

5. The method according to any one of claims 1-4, characterized in that, Also includes: The total throughput of the database system is determined based on the variable capacity of the first memory and the variable capacity of the second memory. The first memory is used for row-oriented storage; The variable capacity of both the first memory and the second memory are constantly being adjusted variables. Determine the reference capacity of the first memory corresponding to the variable capacity of the first memory, and the reference capacity of the second memory corresponding to the variable capacity of the second memory; The reference capacity of the first memory and the reference capacity of the second memory are both constants; The first algorithm is used to calculate the first variable capacity based on the total throughput to obtain the first processing capacity, and the first capacity is obtained based on the first reference capacity and the first processing capacity. The first capacity is used to maximize the total throughput of the database system; the first variable capacity is one of the variable capacity of the first memory and the variable capacity of the second memory; the first reference capacity is one of the reference capacity of the first memory and the reference capacity of the second memory. The first capacity is one of the capacity of the first memory and the capacity of the second memory; Based on the first capacity and the total memory capacity of the database system, the second capacity is obtained; The second capacity is one of the capacity of the first memory and the capacity of the second memory, and the second capacity is different from the first capacity.

6. The method as described in claim 5, characterized in that, Determining the total throughput of the database system based on the variable capacity of the first memory and the variable capacity of the second memory includes: The transaction throughput of the database system is determined based on the variable capacity of the first memory and the total amount of data stored in the persistent storage medium. The analytical processing throughput of the database system is determined based on the variable capacity of the second memory and the set of data columns. The total throughput of the database system is determined based on the transaction processing throughput and the analysis processing throughput.

7. The method as described in claim 6, characterized in that, The step of determining the analytical processing throughput of the database system based on the variable capacity of the second memory and the data column set includes: In the case of the data column set, the time cost corresponding to each of the S sample query statements is determined; the time cost of one of the S sample query statements is determined based on the time required to read the data column corresponding to the one sample query statement from the persistent storage medium or the second memory; The analysis and processing throughput is determined based on the time cost corresponding to the S sample query statements.

8. The method as described in any one of claims 5-7, characterized in that, The step of using a first algorithm to calculate the first variable capacity based on the total throughput to obtain a first processing capacity, and obtaining the first capacity based on the first reference capacity and the first processing capacity, includes: The following steps are performed iteratively until the iteration stopping condition is met: calculate the gradient estimate of the total throughput with respect to the variable capacity of the first memory to obtain the first capacity to be processed; process the first capacity to be processed based on the reference capacity of the first memory to obtain the adjustment amount of the first memory; obtain the updated reference capacity of the first memory based on the reference capacity of the first memory and the adjustment amount of the first memory. The reference capacity of the first memory that satisfies the iteration stopping condition is taken as the first capacity.

9. The method as described in claim 8, characterized in that, Obtaining the updated reference capacity of the first memory based on the reference capacity of the first memory and the adjustment amount of the first memory includes: The updated reference capacity of the first memory is determined based on the sum of the reference capacity of the first memory and the adjustment amount of the first memory.

10. The method according to any one of claims 5-7, characterized in that, The step of using a first algorithm to calculate the first variable capacity based on the total throughput to obtain a first processing capacity, and obtaining the first capacity based on the first reference capacity and the first processing capacity, includes: The following steps are performed iteratively until the iteration stopping condition is met: calculate the gradient estimate of the total throughput with respect to the variable capacity of the second memory to obtain the first capacity to be processed; process the first capacity to be processed based on the reference capacity of the second memory to obtain the adjustment amount of the second memory; obtain the updated reference capacity of the second memory based on the reference capacity of the second memory and the adjustment amount of the second memory. The reference capacity of the second memory that satisfies the iteration stopping condition is used as the first capacity.

11. The method as described in claim 10, characterized in that, Obtaining the updated reference capacity of the second memory based on the reference capacity of the second memory and the adjustment amount of the second memory includes: The updated reference capacity of the second memory is determined based on the sum of the reference capacity of the second memory and the adjustment amount of the second memory.

12. The method according to any one of claims 1-11, characterized in that, Also includes: Obtain the query statement to be executed and determine the data column corresponding to the query statement to be executed; If the data column corresponding to the query statement to be executed is located in the data column set, then the data column corresponding to the query statement to be executed is loaded from the persistent storage medium into the second memory; The query statement to be executed is executed based on the data column corresponding to the query statement to be executed located in the second memory.

13. The method according to any one of claims 1-12, characterized in that, Also includes: Obtain a first feature vector, which is used to mark the frequency information of the query statement to be executed within a first time period. The frequency information of the query statement to be executed includes the frequency of the template query statement corresponding to the query statement to be executed and / or the frequency of the data column corresponding to the query statement to be executed. Obtain a second feature vector, which is used to mark the frequency information of the query statement to be executed within a second time period; the second time period is located before the first time period. If the similarity between the first feature vector and the second feature vector is less than a first threshold, the capacity of the first memory and the capacity of the second memory are determined.

14. A memory management device, characterized in that, include: The acquisition module is used to obtain the reference capacity of the second memory and S sample query statements in the database system; The second memory is used for columnar storage; The determination module is used to determine the objective function and the constraints of the objective function based on the reference capacity of the second memory and the S sample query statements; The objective function is that the time benefit corresponding to each of the S sample query statements satisfies the first condition; the time benefit of one of the S sample query statements is determined based on the difference between the time required to read the data column corresponding to the sample query statement from the persistent storage medium and the time required to read it from the second memory; The acquisition module is further configured to solve the objective function according to the constraints to obtain the data column set; the data column set includes at least one data column, which is a column of stored data selected from the persistent storage medium and used to be loaded into the second memory.

15. The apparatus as claimed in claim 14, characterized in that, The constraints include at least one of the following: The total amount of the selected data columns to be loaded into the second memory satisfies the second condition with respect to the reference capacity of the second memory; If all the data columns corresponding to one of the S sample query statements are selected, the data column corresponding to that sample query statement can be read from the second memory.

16. The apparatus as claimed in claim 14, characterized in that, The determining module is specifically used for: Each of the S sample query statements is split into K sub-query statements; Based on the reference capacity of the second memory and the K sub-queries, determine the objective function and the constraints of the objective function; The objective function is that the time benefits corresponding to the K subqueries satisfy the first condition.

17. The apparatus as claimed in claim 16, characterized in that, The constraints include at least one of the following: The total amount of the selected data columns to be loaded into the second memory satisfies the second condition with respect to the reference capacity of the second memory; If all the data columns corresponding to one of the K subqueries are selected, the data column corresponding to that subquery can be read from the second memory.

18. The apparatus according to any one of claims 14-17, characterized in that, It also includes a computing module; The determining module is further configured to determine the total throughput of the database system based on the variable capacity of the first memory and the variable capacity of the second memory; the first memory is used for row-based storage; the variable capacity of the first memory and the variable capacity of the second memory are both continuously adjusted variables; The determining module is further configured to determine the reference capacity of the first memory corresponding to the variable capacity of the first memory, and the reference capacity of the second memory corresponding to the variable capacity of the second memory. The reference capacity of the first memory and the reference capacity of the second memory are both constants; The calculation module is used to calculate the first variable capacity based on the total throughput using a first algorithm to obtain a first processing capacity, and to obtain a first capacity based on the first reference capacity and the first processing capacity. The first capacity is used to maximize the total throughput of the database system; the first variable capacity is one of the variable capacity of the first memory and the variable capacity of the second memory; the first reference capacity is one of the reference capacity of the first memory and the reference capacity of the second memory. The first capacity is one of the capacity of the first memory and the capacity of the second memory; The calculation module is further configured to obtain a second capacity based on the first capacity and the total memory capacity of the database system; The second capacity is either the capacity of the first memory or the capacity of the second memory, and the second capacity is different from the first capacity.

19. The apparatus as claimed in claim 18, characterized in that, The determining module is specifically used for: The transaction throughput of the database system is determined based on the variable capacity of the first memory and the total amount of data stored in the persistent storage medium. The analytical processing throughput of the database system is determined based on the variable capacity of the second memory and the set of data columns. The total throughput of the database system is determined based on the transaction processing throughput and the analysis processing throughput.

20. The apparatus as claimed in claim 19, characterized in that, The determining module is specifically used for: In the case of the data column set, the time cost corresponding to each of the S sample query statements is determined; the time cost of one of the S sample query statements is determined based on the time required to read the data column corresponding to the one sample query statement from the persistent storage medium or the second memory; The analysis and processing throughput is determined based on the time cost corresponding to the S sample query statements.

21. The apparatus according to any one of claims 18-20, characterized in that, The computing module is specifically used for: The following steps are performed iteratively until the iteration stopping condition is met: calculate the gradient estimate of the total throughput with respect to the variable capacity of the first memory to obtain the first capacity to be processed; The first capacity to be processed is processed based on the reference capacity of the first memory to obtain the adjustment amount of the first memory; Based on the reference capacity of the first memory and the adjustment amount of the first memory, the updated reference capacity of the first memory is obtained. The reference capacity of the first memory that satisfies the iteration stopping condition is taken as the first capacity.

22. The apparatus as claimed in claim 21, characterized in that, The computing module is specifically used for: The updated reference capacity of the first memory is determined based on the sum of the reference capacity of the first memory and the adjustment amount of the first memory.

23. The apparatus according to any one of claims 18-20, characterized in that, The computing module is specifically used for: The following steps are performed iteratively until the iteration stopping condition is met: calculate the gradient estimate of the total throughput with respect to the variable capacity of the second memory to obtain the first capacity to be processed; The first capacity to be processed is processed based on the reference capacity of the second memory to obtain the adjustment amount of the second memory; Based on the reference capacity of the second memory and the adjustment amount of the second memory, the updated reference capacity of the second memory is obtained. The reference capacity of the second memory that satisfies the iteration stopping condition is used as the first capacity.

24. The apparatus as claimed in claim 23, characterized in that, The computing module is specifically used for: The updated reference capacity of the second memory is determined based on the sum of the reference capacity of the second memory and the adjustment amount of the second memory.

25. The apparatus according to any one of claims 14-24, characterized in that, It also includes a loading module and an execution module; The acquisition module is further configured to acquire the query statement to be executed and determine the data column corresponding to the query statement to be executed; The loading module is used to load the data column corresponding to the query statement to be executed from the persistent storage medium into the second memory if the data column corresponding to the query statement to be executed is located in the data column set; The execution module is used to execute the query statement to be executed based on the data column corresponding to the query statement to be executed located in the second memory.

26. The apparatus as described in any one of claims 14-25, characterized in that, The acquisition module is further configured to acquire a first feature vector, which is used to mark the frequency information of the query statement to be executed within a first time period. The frequency information of the query statement to be executed includes the frequency of the template query statement corresponding to the query statement to be executed and / or the frequency of the data column corresponding to the query statement to be executed. The acquisition module is further configured to acquire a second feature vector, which is used to mark the frequency information of the query statement to be executed within a second time period; the second time period is located before the first time period. The determining module is further configured to determine the capacity of the first memory and the capacity of the second memory when the similarity between the first feature vector and the second feature vector is less than a first threshold.

27. A computing device, characterized in that, The device includes a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1-13.

28. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing system, perform the method as described in any one of claims 1-13.

29. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing system, it causes the computing system to perform the method as described in any one of claims 1-13.

Images