Method and apparatus for columnar data storage in database, device, and medium

Abstract

Disclosed in the present invention are a method and apparatus for columnar data storage in a database, a device, and a storage medium. The method comprises: writing a data table to be stored; storing data of each column in said data table according to a column header of the column to form an original storage column; generating fused data on the basis of data corresponding to each row in said data table, and performing column-based storage on the fused data to form a fused storage column; and arranging the fused storage column as a new column after the original storage column to generate a fused column-based storage table, and storing the fused column-based storage table into a database. By using the method, on the basis of original storage columns, a new fused column is introduced, in which multiple columns of a data table are fused, such that during multi-column query, data can be preferentially retrieved from the fused column, thereby improving access performance of multi-column query in a column-based storage mode.

Description

A method, apparatus, device, and medium for storing columnar data in a database.

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 2024119723792, filed on December 30, 2024, entitled "A method, apparatus, device and medium for storing data in a database", the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to the field of database technology, and in particular to a method, apparatus, device and storage medium for columnar data storage in a database. Background Technology

[0004] In data analysis scenarios, more and more data analysis scenarios use columnar storage. Traditional columnar storage databases perform very well for analysis with a small number of columns, but when performing ad-hoc queries with multiple columns, the scattered storage of different columns leads to very high random I / O and poor performance. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for storing columnar data in a database, which solves the problem that columnar databases suffer from high random I / O and poor performance when performing ad-hoc queries on multiple columns due to the dispersed storage of different columns.

[0006] In a first aspect, embodiments of the present invention provide a method for storing columnar data in a database, comprising:

[0007] Write the data to be stored in the data table;

[0008] Store one column of data according to the column header of each column in the data table to be stored, forming the original storage column;

[0009] Based on the data corresponding to each row in the data table to be stored, fused data is generated, and the fused data is stored in columns to form a fused storage column;

[0010] The merged storage column is placed as a new column after the original storage column to generate a merged column storage table, and the merged column storage table is stored in the database.

[0011] Secondly, embodiments of the present invention also provide an apparatus for storing columnar data in a database, the apparatus comprising:

[0012] The data writing module is used to write data to the table to be stored;

[0013] The raw column data generation module is used to store one column of data according to the column header of each column in the data table to be stored, forming the raw storage column;

[0014] The merged column data generation module is used to generate merged data based on the data corresponding to each row in the data table to be stored, and to store the merged data in columns to form a merged storage column;

[0015] The data storage module is used to place the merged storage column as a new column after the original storage column to generate a merged column storage table, and to store the merged column storage table in the database.

[0016] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0017] One or more processors;

[0018] Storage device for storing one or more programs.

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the database columnar data storage method provided in the embodiments of this disclosure.

[0020] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a method for storing columnar data in a database as provided in embodiments of this disclosure.

[0021] Fifthly, embodiments of this disclosure also provide a computer program product, which includes a computer program that, when executed by a processor, implements the method for storing columnar data in a database provided in embodiments of this disclosure.

[0022] This invention discloses a method, apparatus, device, and storage medium for column-oriented data storage in a database. The method includes: writing data into a data table to be stored; storing data in one column according to the column header of each column in the data table to form an original storage column; generating fused data based on the data corresponding to each row in the data table to be stored, and storing the fused data in columns to form a fused storage column; placing the fused storage column as a new column after the original storage column to generate a fused column-oriented storage table, and storing the fused column-oriented storage table in the database. Using this method, a new fused column is introduced on the basis of the original column-oriented data. The new fused column merges multiple columns of the data table together. When performing multi-column queries, data can be preferentially retrieved from this fused column, improving the access performance of multi-column queries in column-oriented storage mode. Attached Figure Description

[0023] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0024] Figure 1 is a flowchart of a method for storing columnar data in a database according to an embodiment of this disclosure;

[0025] Figure 2 is an example diagram of a data storage method provided in an embodiment of this disclosure;

[0026] Figure 3 is an example diagram of another data storage method provided in an embodiment of this disclosure;

[0027] Figure 4 is an example diagram of another data storage method provided in an embodiment of this disclosure;

[0028] Figure 5 is an example diagram of another data storage method provided in an embodiment of this disclosure;

[0029] Figure 6 is a schematic diagram of the structure of a database column storage device provided in an embodiment of this disclosure;

[0030] Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0031] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0032] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0033] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0034] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0035] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0036] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0037] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0038] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0039] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0040] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0041] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0042] Example 1

[0043] Figure 1 is a flowchart of a method for storing columnar data in a database according to an embodiment of this disclosure. This embodiment of the disclosure is applicable to situations where, when performing ad-hoc queries on multiple columns in a columnar database, the scattered storage of different columns leads to very high random I / O and poor performance. This method can be executed by a device for storing columnar data in a database. This device can be implemented in the form of software and / or hardware, and optionally, it can be implemented by an electronic device, such as a mobile terminal, a PC, or a server.

[0044] As shown in Figure 1, this disclosure provides a method for storing columnar data in a database, which may specifically include the following steps:

[0045] S110, Write the data to be stored into the data table.

[0046] In this embodiment, the data table to be stored can be a data table waiting to be stored in the database.

[0047] This step is used to write the data to be stored in the table.

[0048] S120. Store one column of data according to the column header of each column in the data table to be stored, forming the original storage column.

[0049] In this embodiment, the column header can be an attribute of each column or the name of the column. Figure 2 is an example diagram of a data storage method provided by an embodiment of this disclosure. As shown in Figure 2, each column of data in the data table to be stored is stored according to its column header to form the original storage columns. The first four columns in Figure 2 are the original storage columns.

[0050] Based on the above embodiments, storing one column of data according to the column header of each column in the data table to be stored, forming the original storage column, includes the following steps:

[0051] a1) Obtain the preset first column header set; the first column header set includes at least one column header from the data table to be stored.

[0052] b1) Use the column data corresponding to each column header in the first column header set of the data table to be stored as the original column data.

[0053] c1) Store each column of original data in a separate column to form the original storage column.

[0054] In this embodiment, the first column header set may be a set of column headers of the columns used for data analysis.

[0055] Specifically, a preset first column header set is obtained; the first column header set includes at least one column header from the data table to be stored. For each row of data written, the column data corresponding to each column header in the data table to be stored and the column data in the first column header set are used as the original column data. The original storage column is stored according to the column header of each column of the original column data to form the original storage column.

[0056] For example, Figure 2 is an example diagram of another data storage method provided by an embodiment of the present disclosure. As shown in Figure 2, the first, second, third and fourth columns are all original columns, the fifth column is a fusion column of the first and second columns, and the sixth column is a fusion column of the third and fourth columns.

[0057] For example, Figure 3 is an example diagram of another data storage method provided by the present disclosure. As shown in Figure 3, the first column header set includes first and third column headers. The column data in the data table to be stored that corresponds to each column header in the first column header set is used as the original column data. The original storage column is stored according to the column header of each column of the original column data to form an original storage column.

[0058] Using this method, only the columns that need to be analyzed can be stored column by column, while the data that does not need to be analyzed column by column can be stored in the merged column. The sum of all the data in all columns cannot be missing any column of the original data, thus saving storage space.

[0059] S130. Generate fused data based on the data corresponding to each row in the data table to be stored, and store the fused data in columns to form a fused storage column.

[0060] In this embodiment, the data corresponding to each row in the data table to be stored is sequentially connected to generate fused data, and the fused data is then stored in the corresponding fused storage columns.

[0061] For example, as shown in Figure 3, the fifth column is the converged storage column.

[0062] Based on the above embodiments, generating fused data based on the data corresponding to each row in the data table to be stored, and storing the fused data in columns to form a fused storage column includes the following steps:

[0063] a2) Obtain the set of column headers from the data table to be stored that are pre-set for each merged storage column.

[0064] b2) For each row of data written, group each row of data in the data table to be stored according to the set of column headers, and generate group data corresponding to each fused storage column.

[0065] c2) Organize the data from each group in sequence to generate the corresponding fused data.

[0066] d2) Store each fused data in its corresponding column into the respective fused storage column.

[0067] Specifically, the process involves retrieving a pre-defined set of column headers from the data table to be stored, pre-defined by each merged storage column. Each row of data in the data table is then grouped according to its column header set, generating corresponding group data for each merged storage column. These groups are then concatenated to generate the merged data. Finally, each merged data is stored as a column in its respective merged storage column. Data can be organized in various ways in the database, such as JSON format, delimiter-based delimitation, XML format, and TLV (Time-Based Value) methods.

[0068] For example, Figure 5 is an example diagram of another data storage method provided by the present disclosure. The data in columns 1-4 are fused according to a preset method. The data in columns 1 and 2 of the same row are fused to generate a fifth column as a fused storage column. The data in columns 3 and 4 of the same row are fused to generate a fifth column as another fused storage column. As shown in Figure 5, there can be multiple fused columns. The fifth and sixth columns are fused data columns.

[0069] Using this method, data is divided into multiple merge columns based on the frequency of use. The sum of all columns must not be missing any column of the original data, thus reducing unnecessary disk I / O overhead during access.

[0070] S140. Place the merged storage column as a new column after the original storage column to generate a merged column storage table, and store the merged column storage table in the database.

[0071] Specifically, the merged storage column is placed as a new column after the original storage column to generate a merged columnstore table, and the merged columnstore table is stored in the database. For example, Figures 2, 3, 4 and 5 are all examples of merged columnstore tables.

[0072] Based on the above embodiments, before generating fused data based on the data corresponding to each row in the data table to be stored, and before forming the fused storage column, the following steps are also included:

[0073] a3) Obtain the preset second column header set; the second column header set includes at least one column header from the data table to be stored.

[0074] b3) Take the column data corresponding to each column header in the second column header set from the data table to be stored as a new data table to be stored.

[0075] In this embodiment, the first column header set can be a set of column headers for data columns that do not require column-by-column analysis.

[0076] Specifically, a preset second column header set is obtained; the second column header set includes at least one column header from the data table to be stored. The column data in the data table to be stored that corresponds to each column header in the second column header set is used as a new data table to be stored.

[0077] Based on the above embodiments, ad-hoc queries on data in the database include the following steps:

[0078] a4) Obtain ad-hoc query information generated based on user needs.

[0079] b4) Determine the merged column table corresponding to the ad-hoc query information.

[0080] c4) I / O overhead of performing ad-hoc query information when the accessed objects are determined to be merged storage columns and original storage columns.

[0081] d4) Select the access object with the lowest I / O overhead as the one corresponding to the ad-hoc query information.

[0082] In this embodiment, ad-hoc query information refers to non-predefined queries made by users in a data warehouse or database based on their immediate needs.

[0083] Specifically, retrieve ad-hoc query information generated based on user needs. Determine the fused columnstore table corresponding to the ad-hoc query information. Determine the I / O overhead of executing the ad-hoc query information when the access objects are fused storage columns and original storage columns. Select the access object with the lower I / O overhead as the access object corresponding to the execution of the ad-hoc query information.

[0084] Based on the above embodiments, determining the fused columnar storage table corresponding to the ad-hoc query information includes the following steps:

[0085] b41) Extract the SQL statements from the ad-hoc query information;

[0086] b42) Parse the SQL statement according to the preset SQL parsing engine to obtain the corresponding fused column table in the SQL statement.

[0087] Specifically, the SQL statements in the ad-hoc query information are extracted, an SQL parsing engine is generated based on the SQL syntax, and the SQL statements are parsed by the SQL parsing engine to obtain the fused column table in the SQL statements.

[0088] This invention discloses a method for storing columnar data in a database. The method includes: writing data into a data table to be stored; storing data in a column according to the column headers of each column in the data table to form an original storage column; generating fused data based on the data corresponding to each row in the data table to be stored, and storing the fused data in columns to form a fused storage column; placing the fused storage column as a new column after the original storage column to generate a fused columnar storage table, and storing the fused columnar storage table in the database. Using this method, a new fused column is introduced on the basis of the original columnar data. This new fused column merges multiple columns of the data table, allowing for better data retrieval from this fused column during multi-column queries, thus improving the access performance of multi-column queries in the columnar storage mode.

[0089] Figure 6 is a schematic diagram of the structure of a method and apparatus for storing columnar data in a database, as provided in an embodiment of the present invention. As shown in Figure 6, there is a data writing module 210, a raw column data generation module 220, a fused column data generation module 230, and a data storage module 240.

[0090] Data writing module 210 is used to write data to be stored in the data table;

[0091] The raw column data generation module 220 is used to store one column of data according to the column header of each column of data in the data table to be stored, forming the raw storage column;

[0092] The fused column data generation module 230 is used to generate fused data based on the data corresponding to each row in the data table to be stored, and to store the fused data in columns to form a fused storage column;

[0093] The data storage module 240 is used to place the fused storage column as a new column after the original storage column to generate a fused column storage table, and to store the fused column storage table in the database.

[0094] The technical solution provided in this disclosure uses a method that introduces a new fusion column on the basis of the original columnar data. The new fusion column merges multiple columns of the data table together. When querying multiple columns, data can be preferentially obtained from this fusion column, thereby improving the access performance of querying multiple columns in the columnar storage mode.

[0095] Furthermore, the raw column data generation module 220 can be used for:

[0096] Obtain a preset first column header set; the first column header set includes at least one column header from the data table to be stored;

[0097] The column data corresponding to each column header in the first column header set of the data table to be stored is used as the original column data;

[0098] The original storage columns are formed by storing each column of original data according to its column header.

[0099] Furthermore, the merged column data generation module 230 can be used for:

[0100] Obtain the set of column headers from the data table to be stored that are pre-set for each of the fused storage columns;

[0101] Group each row of data in the data table to be stored according to the set of column headers to generate group data corresponding to each of the fused storage columns;

[0102] Each of the aforementioned groups of data is sequentially concatenated to generate corresponding fused data.

[0103] The fused data is stored in columns corresponding to each of the fused storage columns.

[0104] Furthermore, the device can also be used for:

[0105] Before generating fused data based on the data corresponding to each row in the data table to be stored, and storing the fused data in columns to form fused storage columns, the method further includes:

[0106] Obtain a preset second column header set; the second column header set includes at least one column header from the data table to be stored;

[0107] The column data corresponding to each column header in the second column header set in the data table to be stored is used as the new data table to be stored.

[0108] Furthermore, the device can also be used for:

[0109] It also includes ad-hoc queries on data in the database, including:

[0110] Obtain ad-hoc query information generated based on user needs;

[0111] Determine the fused column table corresponding to the ad-hoc query information;

[0112] The I / O overhead of executing the ad-hoc query information when the access objects are determined to be the fused storage column and the original storage column;

[0113] The access object with the lowest I / O overhead is selected as the one corresponding to the ad-hoc query information.

[0114] Furthermore, the device can also be used for:

[0115] Extract the SQL statements from the ad-hoc query information;

[0116] The SQL statement is parsed using a preset SQL parsing engine to obtain the corresponding fused columnar storage table in the SQL statement.

[0117] The above-described apparatus can execute the methods provided in all the foregoing embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the above methods. Technical details not described in detail in this embodiment can be found in the methods provided in all the foregoing embodiments of the present invention.

[0118] Figure 7 shows a schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0119] As shown in Figure 7, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0120] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0121] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the method of columnar data storage in a database.

[0122] In some embodiments, the method for storing columnar data in a database can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for storing columnar data in a database described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the method for storing columnar data in a database by any other suitable means (e.g., by means of firmware).

[0123] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0125] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0126] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0127] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0128] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0129] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0130] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

A method for storing columnar data in a database, characterized in that, include: Write the data to be stored in the data table; Store one column of data according to the column header of each column in the data table to be stored, forming the original storage column; Based on the data corresponding to each row in the data table to be stored, fused data is generated, and the fused data is stored in columns to form a fused storage column; The merged storage column is placed as a new column after the original storage column to generate a merged column storage table, and the merged column storage table is stored in the database. The method according to claim 1, characterized in that, The step of storing data column by column header according to the data in the data table to be stored, forming the original storage column, includes: Obtain a preset first column header set; the first column header set includes at least one column header from the data table to be stored; The column data corresponding to each column header in the first column header set of the data table to be stored is used as the original column data; For each row of data written, a column is stored according to the column header of each original column data to form the original storage column. The method according to claim 1, characterized in that, The process of generating fused data based on the data corresponding to each row in the data table to be stored, and storing the fused data in columns to form fused storage columns, includes: Obtain the set of column headers from the data table to be stored that are pre-set for each of the fused storage columns; For each row of data written, each row of data in the data table to be stored is grouped according to the set of column headers to generate group data corresponding to each of the fused storage columns; The data from each group are organized sequentially to generate corresponding fused data. The fused data is stored in columns corresponding to each of the fused storage columns. The method according to claim 1, characterized in that, Before generating fused data based on the data corresponding to each row in the data table to be stored, and storing the fused data in columns to form fused storage columns, the method further includes: Obtain a preset second column header set; the second column header set includes at least one column header from the data table to be stored; The column data corresponding to each column header in the second column header set in the data table to be stored is used as the new data table to be stored. The method according to claim 1, characterized in that, It also includes ad-hoc queries on data in the database, including: Obtain ad-hoc query information generated based on user needs; Determine the fused column table corresponding to the ad-hoc query information; The I / O overhead of executing the ad-hoc query information when the access objects are determined to be the fused storage column and the original storage column; The access object with the lowest I / O overhead is selected as the one corresponding to the ad-hoc query information. The method according to claim 5, characterized in that, The determination of the fused columnar storage table corresponding to the ad-hoc query information includes: Extract the SQL statements from the ad-hoc query information; The SQL statement is parsed using a preset SQL parsing engine to obtain the corresponding fused columnar storage table in the SQL statement. An apparatus for storing columnar data in a database, characterized in that, include: The data writing module is used to write data to the data table to be stored; The raw column data generation module is used to store one column of data according to the column header of each column in the data table to be stored, forming the raw storage column; The merged column data generation module is used to generate merged data based on the data corresponding to each row in the data table to be stored, and to store the merged data in columns to form a merged storage column; The data storage module is used to place the merged storage column as a new column after the original storage column to generate a merged column storage table, and to store the merged column storage table in the database. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method for storing columnar data in a database as described in any one of claims 1-6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for storing columnar data in a database as described in any one of claims 1-6. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method for storing columnar data in a database as described in any one of claims 1-6.

Images