Data system, data management and data query

Abstract

Provided in the present specification are a data system, a data management method and apparatus, and a data query method and apparatus. The data system comprises: an object storage layer, which stores at least one sorted string table (SSTable), wherein a plurality of data blocks and an index tree are stored in the SSTable, data and index information of the data are stored in the data blocks, the index tree comprises a plurality of nodes in a tree structure, leaf nodes among the plurality of nodes correspond to the plurality of data blocks on a one-to-one basis, and the leaf nodes store the index information in the corresponding data blocks; and at least one database, which is in communication connection with the object storage layer, and is used for externally providing data services on the basis of the data stored in the object storage layer.

Description

Data systems, data management, data query Technical Field

[0001] This specification relates to the field of database technology, and more particularly to a data system, data management method and apparatus, and data query method and apparatus. Background Technology

[0002] Object storage offers advantages such as low cost (a fraction of the price of cloud disks), pay-as-you-go pricing (based on actual usage, including capacity and read / write requests), high scalability, high durability, high availability, and shareability. Using object storage as shared storage can significantly reduce storage costs. Because object storage handles large amounts of data, it primarily utilizes relatively low-cost storage media such as hard disk drives (HDDs).

[0003] Data systems used for data management in related technologies for object storage have low performance. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide a data system, a data management method and apparatus, and a data query method and apparatus.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions.

[0006] According to a first aspect of one or more embodiments of this specification, a data system is proposed, the system comprising: an object storage layer storing at least one ordered string table SSTable, the SSTable storing multiple data blocks and an index tree, the data blocks storing data and index information of the data, the index tree including multiple nodes in a tree structure, the leaf nodes of the multiple nodes corresponding one-to-one with the multiple data blocks, and the leaf nodes storing the index information of the corresponding data blocks; and at least one database communicatively connected to the object storage layer for providing data services to external parties based on the data stored in the object storage layer.

[0007] In one or more of the descriptions in this specification, the data block contains multiple data segments, and the index information within the data block includes multiple index rows, with each data segment corresponding to one of the multiple index rows.

[0008] In one or more of these specifications, the non-leaf nodes among the plurality of nodes store index information of their child nodes.

[0009] In one or more of these specifications, the root node of the plurality of nodes is stored in the metadata of the SSTable.

[0010] In one or more of the embodiments of this specification, the object storage layer is used to merge the SSTable in the following manner: sequentially determining whether the data in each data block has incremental updates, and updating the data and index information in the data blocks with incremental updates to obtain update results; generating the merged result of the SSTable based on the update results of each data block with incremental updates and each data block without incremental updates; and constructing an index tree in the merged result of the SSTable based on the index information in the update results of each data block with incremental updates and the index information in each data block without incremental updates.

[0011] In one or more of the embodiments of this specification, when the object storage layer is used to construct an index tree in the merged result of the SSTable based on the index information in the update result of each data block with incremental updates and the index information in each data block without incremental updates, the layer is used to: load the index information in the data blocks without incremental updates into memory; load the index information in the update result of the data blocks with incremental updates into memory; write the index information in memory to disk into the merged result of the SSTable, and construct an index tree based on the index information written to disk.

[0012] In one or more of the embodiments of this specification, when the object storage layer is used to write the index information in memory to disk into the merged result of the SSTable and to build an index tree based on the index information written to disk, it is used to: in response to the index information in memory reaching the disk write threshold, write the index information in memory to disk into the merged result of the SSTable; in response to the index information of all data blocks in the merged result of the SSTable having been written to disk, build an index tree based on the index information written to disk.

[0013] In one or more of these specifications, the object storage layer is also used to load the index tree into each database when the index tree in the SSTable is updated.

[0014] In one or more of these specifications, the database is further configured to determine the data block containing the data targeted by the SQL request based on the index tree within the SSTable, and to read the data targeted by the SQL request within the data block.

[0015] According to a second aspect of one or more embodiments of this specification, a data management method is proposed, the method comprising: sequentially determining whether the data in each data block of an ordered string table (SSTable) has incremental updates, and updating the data and index information in the data blocks with incremental updates to obtain an update result; generating a merged result of the SSTable based on the data in the update result of each data block with incremental updates and the data in each data block without incremental updates; and constructing an index tree in the merged result of the SSTable based on the index information in the update result of each data block with incremental updates and the index information in each data block without incremental updates.

[0016] In one or more of these specifications, the construction of an index tree within the merged result of the SSTable based on the index information within the update result of each data block with incremental updates and the index information within each data block without incremental updates includes: loading the index information within the data blocks without incremental updates into memory; loading the index information within the update result of the data blocks with incremental updates into memory; writing the index information in memory to disk into the merged result of the SSTable, and constructing an index tree based on the written index information.

[0017] In one or more of these specifications, the step of writing the index information in memory to disk into the merged result of the SSTable and constructing an index tree based on the written index information includes: in response to the index information in memory reaching the disk writing threshold, writing the index information in memory to disk into the merged result of the SSTable; and in response to the index information of all data blocks in the merged result of the SSTable having been written to disk, constructing an index tree based on the written index information.

[0018] In one or more of these specifications, the method further includes: loading the index tree within the merged result or DDL operation result of the SSTable into each database by the object storage layer.

[0019] According to a third aspect of one or more embodiments of this specification, a data query method is proposed, the method comprising: determining, based on an index tree in an ordered string table SSTable, a data block containing data targeted by an SQL request, and reading the data targeted by the SQL request within the data block.

[0020] According to a fourth aspect of one or more embodiments of this specification, a data management apparatus is provided, the apparatus comprising: an update module, configured to sequentially determine whether the data in each data block of an ordered string table (SSTable) has an incremental update, and update the data and index information in the data blocks with incremental updates to obtain an update result; a merging module, configured to generate a merging result of the SSTable based on the data in the update result of each data block with incremental updates and the data in each data block without incremental updates; and an indexing module, configured to construct an index tree in the merging result of the SSTable based on the index information in the update result of each data block with incremental updates and the index information in each data block without incremental updates.

[0021] According to a fifth aspect of one or more embodiments of this specification, a data query apparatus is provided, the apparatus comprising: a query module configured to determine, based on an index tree in an ordered string table SSTable, a data block containing data targeted by an SQL request, and to read the data targeted by the SQL request within the data block.

[0022] According to a sixth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in the second or third aspect.

[0023] According to a seventh aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor executes the executable instructions to implement the method as described in the second or third aspect.

[0024] According to an eighth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the second or third aspect.

[0025] The technical solutions provided by the embodiments of this specification may include the following beneficial effects.

[0026] The data system provided in the embodiments of this specification includes an object storage layer and at least one database communicatively connected to the object storage layer. These databases can provide data services to external entities based on the data stored in the object storage layer, thereby achieving sharing of the object storage layer. The object storage layer stores at least one SSTable, which contains multiple data blocks and an index tree. Each data block stores data and its index information. The index tree includes multiple nodes in a tree structure, with leaf nodes corresponding one-to-one with the data blocks, and each leaf node storing the index information within the corresponding data block. The index information within a data block is stored locally, providing self-interpretation capabilities for the data block and supporting database DDL (Data Definition Language) operations and write operations such as SSTable merging and reconstruction, thereby improving the performance of these write operations. The index tree aggregates the index information of data blocks within the SSTable, facilitating IO (Input / Output) pipelined operations and prefetching during query operations, better utilizing the advantages of sequential and batch reads of object storage, reducing random IO occurrences, and thus improving query performance. Attached Figure Description

[0027] Figure 1 is a schematic diagram of the architecture of a data system provided in an exemplary embodiment.

[0028] Figure 2 is a schematic diagram of the structure of an index tree provided in an exemplary embodiment.

[0029] Figure 3 is a flowchart of a data management method provided in an exemplary embodiment.

[0030] Figure 4 is a schematic diagram of the structure of a device provided in an exemplary embodiment.

[0031] Figure 5 is a block diagram of a data management device provided in an exemplary embodiment. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0033] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0034] First, some of the concepts involved in this instruction manual will be explained.

[0035] LSM-Tree: A high-efficiency storage structure widely used in database systems, designed to optimize write performance and disk space utilization. It reduces the overhead of random writes by caching write operations in memory and periodically merging data to disk.

[0036] Object storage: Object storage is a data management architecture that efficiently and flexibly stores and retrieves massive amounts of unstructured data by treating data as objects rather than files or blocks. Its "everything is an object" concept emphasizes that all data, whether documents, images, videos, or logs, can be processed as objects, which helps improve access efficiency and scalability.

[0037] In view of the above-mentioned technical problems in the background art, at least one embodiment of this specification provides a data system that can be implemented based on the LSM-Tree storage engine and is in the form of a distributed cluster, that is, it contains multiple databases.

[0038] Referring to Figure 1, the data system includes an object storage layer and at least one database communicatively connected to the object storage layer. The object storage layer stores at least one Sorted String Table (SSTable). Each SSTable stores multiple data blocks and an index tree. Each data block stores data and its index information. The index tree includes multiple nodes in a tree structure, with leaf nodes corresponding one-to-one with each data block, and each leaf node storing the index information within the corresponding data block. The database is used to provide data services based on the data stored in the object storage layer.

[0039] Each database exists as an instance or copy of the data system.

[0040] The object storage layer contains at least one SSTable for storing data written within the data system, and is formed during the data writing process and the data merging process.

[0041] For example, when a data system writes data, it first inserts it into a memory table (MemTable). When the MemTable reaches a certain amount of data, it is marked as immutable, thus forming an Immutable MemTable. The system creates a new MemTable in memory to receive data writes and writes the Immutable MemTable to disk to generate a low-level SSTable. The number of low-level SSTables created on disk gradually increases as the data system runs. When the number of low-level SSTables created on disk becomes large (e.g., exceeding a certain threshold), the system performs Minor Compaction on these low-level SSTables to merge them into a neighboring low-level SSTable. The number of neighboring low-level SSTables created by merging them gradually increases as the data system runs. When the number of neighboring low-level SSTables becomes large (e.g., exceeding a certain threshold), the system performs Major Compaction on these neighboring low-level SSTables to merge them into a higher-level SSTable. As the system runs, the number of SSTable levels increases, gradually forming a complex multi-level structure.

[0042] In addition, the data system periodically performs Major Compaction, which merges SSTables at different levels within the system. This typically involves rewriting the entire dataset to thoroughly remove expired data and optimize the storage structure. The periodically performed Major Compaction uses the SSTable obtained from the previous merge as the baseline SSTable and the newly generated SSTable as the incremental SSTable for compaction.

[0043] The data blocks within the SSTable can be the underlying SSTable formed after the MemTable is written to disk. This underlying SSTable is maintained as a unit in the form of data blocks during subsequent compaction processes. For example, the data block contains multiple data fragments, and the index information within the data block includes multiple index rows, with each data fragment corresponding to one of the multiple index rows.

[0044] The index information within the data block is stored nearby, providing the data block with self-interpretation capabilities. This supports database DDL operations and write operations such as SSTable merging and reconstruction. For example, a reused data block can directly obtain various information about the data stored within it through the index information stored within it, without needing to send I / O to read all the data within the data block to obtain its various information, thereby improving the performance of these write operations.

[0045] The index tree is a logically ordered tree structure composed of index information from all data blocks within the SSTable. For example, non-leaf nodes among the multiple nodes store the index information of their child nodes. For example, the root node among the multiple nodes is stored in the metadata of the SSTable.

[0046] Please refer to Figure 2, which exemplarily illustrates the structure of an index tree. Figure 2 schematically shows the index information such as last_key, min_value, and max_value for each node; it should be understood that the index information also includes pointer information to child nodes or corresponding data blocks.

[0047] The index information within each node of the index tree is stored in a data block, forming an index block. That is, the data block storing the index information within a node is called an index block. All index blocks in the index tree can be stored sequentially in several physical blocks of the SSTable. This physical organization method can effectively utilize the characteristics of storage media such as hard disk drives in object storage, thereby improving the query performance of the data system. For example, it supports I / O pipelining and prefetching when the data system performs query operations, thus better utilizing the advantages of sequential and batch reads in object storage.

[0048] For example, when the data system executes a query operation, the database determines the data block containing the data targeted by the SQL (Structured Query Language) request based on the index tree within the SSTable, and reads the data targeted by the SQL request from within that data block. After receiving the SQL request, the database can determine the data block containing the data targeted by the SQL request within the index tree based on query information such as key values ​​within the SQL request, further read the data targeted by the SQL request from within that data block, and then return the read data to the client.

[0049] For example, if the index tree has been pre-loaded onto the database's local disk, i.e., pre-warmed up, the database can directly query the index tree locally when it receives an SQL request to determine the data block where the data targeted by the SQL request is located, and then read the data targeted by the SQL request from that data block.

[0050] For example, if the index tree is not preloaded to the database's local disk, i.e., the prediction is not completed, the data block can read the index tree from the object storage layer to the local disk when it receives an SQL request, and then query the index tree locally to determine the data block where the data targeted by the SQL request is located, and then read the data targeted by the SQL request from that data block.

[0051] The data system provided in the embodiments of this specification includes an object storage layer and at least one database communicatively connected to the object storage layer. These databases can provide data services to external entities based on the data stored in the object storage layer, thereby achieving sharing of the object storage layer. The object storage layer stores at least one SSTable, which contains multiple data blocks and an index tree. Each data block stores data and its index information. The index tree includes multiple nodes in a tree structure, with leaf nodes corresponding one-to-one with the data blocks, and each leaf node storing the index information within the corresponding data block. The index information within a data block is stored locally, providing self-interpretation capabilities for the data block and supporting database DDL operations and write operations such as SSTable merging and reconstruction, thereby improving the performance of these write operations. The index tree aggregates the index information of data blocks within the SSTable, facilitating IO pipelining and prefetching during query operations, better utilizing the advantages of sequential and batch reads of object storage, reducing random IO, and thus improving query performance.

[0052] In some embodiments of this disclosure, the object storage layer is used to merge the SSTable in the following manner.

[0053] First, determine whether the data in each data block has incremental updates, and then update the data and index information in the data blocks with incremental updates to obtain the update results.

[0054] For example, metadata such as last_rowkey recorded in the index tree iterator can be used to determine whether a data block has been incrementally updated.

[0055] Next, the merged result of the SSTable is generated based on the update results of each data block with incremental updates and each data block without incremental updates.

[0056] For example, data rows with the same key value are merged to remove expired data rows.

[0057] Finally, an index tree is constructed within the merged result of the SSTable based on the index information within the update results of each data block with incremental updates and the index information within each data block without incremental updates.

[0058] The index information within a data block records the distribution and metadata of the data stored within that block. If the data stored within a data block does not undergo incremental updates, the index information within that block does not need to be updated. The index tree, on the other hand, records the distribution and metadata of all data within the SSTable. Therefore, when merging the SSTables, the index tree needs to be updated, meaning the index tree needs to be rebuilt within the SSTables.

[0059] For example, the index information within data blocks without incremental updates is loaded into memory; the index information within the update results of data blocks with incremental updates is loaded into memory; then the index information in memory is written to disk into the merged result of the SSTable, and an index tree is constructed based on the written index information. For example, in response to the index information in memory reaching the disk writing threshold, the index information in memory is written to disk into the merged result of the SSTable; in response to the index information of all data blocks in the merged result of the SSTable being written to disk, an index tree is constructed based on the written index information. As another example, when constructing the index tree based on the written index information, the written index information can be used as the leaf node layer, building a tree structure from the bottom layer to the top layer until the root node of the tree structure is less than a certain data size. The root node is then stored in the metadata block of the merged result of the SSTable, where the metadata block is used to store metadata.

[0060] The above merging process can be performed on multiple underlying SSTables; or on multiple non-underlying SSTables, for example, when the data system periodically performs Major Compaction, the above merging process can be performed on all SSTables.

[0061] The following example, using the periodic execution of Major Compaction by the data system, will illustrate the above merging process in more detail. After merging begins, each data block within the baseline SSTable (hereinafter referred to as the baseline data block, to distinguish it from the data blocks within the incremental SSTable) is traversed sequentially. While traversing each baseline data block, it is determined whether the data within the baseline data block has been incrementally updated, based on the data blocks within the incremental SSTable. If no update is found, the baseline data block and its index information are directly loaded into the corresponding MemTable. If an update is found, the baseline data block is updated, and the update result and its index information are then loaded into the corresponding MemTable. As each baseline data block is traversed, the MemTable storing the SSTable merging results generates the SSTable merging result during creation and disk persistence, while the MemTable storing the new index tree constructs the index tree during creation and disk persistence.

[0062] This embodiment enables the reuse of data blocks that do not have incremental updates during SSTable merging. The reused data blocks do not need to be rewritten (i.e., their data and index information need to be updated), and the reused data blocks only need to read the information of the data stored in their own blocks from the index information stored in their own blocks, without having to traverse the data stored in their own blocks. This is very convenient and time-saving, and improves the efficiency of merging.

[0063] In some embodiments of this disclosure, the object storage layer is further configured to load the index tree into each database when the index tree in the SSTable is updated.

[0064] For example, when the SSTable merge causes an index tree update, or when the SSTable performs a DDL operation causing an index tree update, the object storage layer can load the updated index tree into each database to replace the local index tree of the database, thereby completing the index tree warm-up.

[0065] Object storage is characterized by high bandwidth and high latency, so its read performance is often optimized by accumulating large I / O operations. However, normal query operations are latency-sensitive, so disk warm-up using a local SSD is often necessary. This involves preloading the data to be read onto the local disk (e.g., Solid State Disk, SSD) to reduce the impact of object storage latency on users. Downloading the requested data block to the local SSD before the query request arrives ensures that subsequent query requests hit the local SSD as much as possible, avoiding repeated requests to the object storage layer that lead to excessively high query latency. This process is called warm-up.

[0066] This embodiment completes the warm-up process during compaction and DDL operations. For compaction, newly generated index blocks are warmed up on the local SSDs of each database in the distributed cluster. For DDL operations, index blocks written by the primary replica are also synchronized to the local SSDs of each replica via clog replay. In this way, when a new SSTable is generated, or after a leader replica switch occurs, query services can be quickly provided to users using the index blocks warmed up on the local SSDs.

[0067] At least one embodiment of this specification provides a data management method. Please refer to Figure 3, which exemplarily illustrates a flowchart of the method, including steps S301 to S303.

[0068] In step S301, it is determined in turn whether the data in each data block of the ordered string table SSTable has incremental updates, and the data and index information in the data blocks with incremental updates are updated to obtain the update result.

[0069] In step S302, the merged result of the SSTable is generated based on the data within the update result of each data block with incremental updates and the data within each data block without incremental updates.

[0070] In step S303, an index tree is constructed within the merged result of the SSTable based on the index information within the update result of each data block with incremental updates and the index information within each data block without incremental updates.

[0071] For example, the step of constructing an index tree in the merged result of the SSTable based on the index information in the update result of each data block with incremental updates and the index information in each data block without incremental updates includes: loading the index information in the data blocks without incremental updates into memory; loading the index information in the update result of the data blocks with incremental updates into memory; writing the index information in memory to disk into the merged result of the SSTable, and constructing an index tree based on the index information written to disk.

[0072] For example, the step of writing the index information in memory to disk into the merged result of the SSTable and constructing an index tree based on the index information written to disk includes: in response to the index information in memory reaching the disk writing threshold, writing the index information in memory to disk into the merged result of the SSTable; in response to the index information of all data blocks in the merged result of the SSTable having been written to disk, constructing an index tree based on the index information written to disk.

[0073] For example, the method further includes loading the index tree within the merged result or DDL operation result of the SSTable into each database by the object storage layer.

[0074] At least one embodiment of this specification provides a data query method, which includes: determining the data block where the data targeted by the SQL request is located based on the index tree in the ordered string table SSTable, and reading the data targeted by the SQL request within the data block.

[0075] Further details regarding the above steps have been described in detail in the data system section of the above embodiments, and will not be repeated here.

[0076] Figure 4 is a schematic structural diagram of a device provided in an exemplary embodiment. Referring to Figure 4, at the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and may also include other hardware required for tasks. One or more embodiments of this specification can be implemented in software, for example, the processor 402 reads the corresponding computer program from the non-volatile memory 410 into the memory 408 and then runs it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0077] Please refer to Figure 5. The data management device can be applied to the device shown in Figure 4 to implement the technical solution of this specification. The data management device may include the following modules.

[0078] The update module 501 is used to sequentially determine whether the data in each data block of the ordered string table SSTable has incremental updates, and to update the data and index information in the data blocks with incremental updates to obtain the update result.

[0079] The merging module 502 is used to generate the merged result of the SSTable based on the data within the update result of each data block with incremental updates and the data within each data block without incremental updates.

[0080] Index module 503 is used to construct an index tree within the merged result of the SSTable based on the index information within the update result of each data block with incremental updates and the index information within each data block without incremental updates.

[0081] In one or more embodiments of this specification, the indexing module is used to: load index information in data blocks that do not have incremental updates into memory; load index information in the update results of data blocks that have incremental updates into memory; write the index information in memory to disk into the merged result of the SSTable, and construct an index tree based on the index information written to disk.

[0082] In one or more embodiments of this specification, when the index module is used to write the index information in memory to disk into the merged result of the SSTable and to construct an index tree based on the index information written to disk, it is used to: in response to the index information in memory reaching the disk write threshold, write the index information in memory to disk into the merged result of the SSTable; in response to the index information of all data blocks in the merged result of the SSTable having been written to disk, construct an index tree based on the index information written to disk.

[0083] In one or more embodiments of this specification, the apparatus further includes a preheating module for: loading the index tree within the merged result or DDL operation result of the SSTable into each database from the object storage layer.

[0084] The data query device can be applied to the device shown in Figure 4 to implement the technical solution of this specification. The data query device may include: a query module, used to determine the data block containing the data targeted by the SQL request based on the index tree within the ordered string table SSTable, and to read the data targeted by the SQL request within the data block.

[0085] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0086] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0087] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0088] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0089] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0090] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0091] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0092] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0093] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0094] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A data system, the data system comprising: The object storage layer stores at least one ordered string table SSTable. The SSTable stores multiple data blocks and an index tree. The data blocks store data and the index information of the data. The index tree includes multiple nodes in a tree structure. The leaf nodes of the multiple nodes correspond one-to-one with the multiple data blocks, and the leaf nodes store the index information of the corresponding data blocks. At least one database is communicatively connected to the object storage layer and is used to provide data services to external parties based on the data stored in the object storage layer.

2. The data system according to claim 1, wherein the data block comprises multiple data segments, the index information within the data block comprises multiple index rows, and the multiple data segments correspond one-to-one with the multiple index rows.

3. In the data system according to claim 1, the non-leaf nodes among the plurality of nodes store index information of the child nodes.

4. In the data system according to claim 1, the root node among the plurality of nodes is stored in the metadata of the SSTable.

5. The data system according to claim 1, wherein the object storage layer is used to merge the SSTable in the following manner: Determine in turn whether the data in each data block has incremental updates, and update the data and index information in the data blocks with incremental updates to obtain the update result; The merged result of the SSTable is generated based on the update results of each data block with incremental updates and each data block without incremental updates; An index tree is constructed within the merged result of the SSTable based on the index information within the update results of each data block with incremental updates and the index information within each data block without incremental updates.

6. The data system according to claim 5, wherein the object storage layer, when constructing an index tree within the merged result of the SSTable based on the index information within the update result of each data block with incremental updates and the index information within each data block without incremental updates, is used for: Load the index information within data blocks that do not have incremental updates into memory; Load the index information within the update results of the data blocks with incremental updates into memory; The index information in memory is written to disk into the merged result of the SSTable, and an index tree is built based on the index information written to disk.

7. The data system according to claim 6, wherein the object storage layer is used to persist index information in memory to disk into the merged result of the SSTable, and when constructing an index tree based on the persisted index information, it is used for: When the index information in memory reaches the disk write threshold, the index information in memory is written to disk into the merged result of the SSTable; In response to the fact that the index information of all data blocks in the merged result of the SSTable has been written to disk, an index tree is constructed based on the index information written to disk.

8. The data system according to claim 5, wherein the object storage layer is further configured to load the index tree into each database when the index tree in the SSTable is updated.

9. The data system according to claim 5, wherein the database is further configured to determine the data block where the data targeted by the SQL request is located based on the index tree in the SSTable, and read the data targeted by the SQL request within the data block.

10. A data management method, the method comprising: The algorithm sequentially determines whether the data in each data block of the ordered string table SSTable has been incrementally updated, and updates the data and index information in the data blocks that have been incrementally updated to obtain the update result. The merged result of the SSTable is generated based on the data within the update result of each data block with incremental updates and the data within each data block without incremental updates; An index tree is constructed within the merged result of the SSTable based on the index information within the update results of each data block with incremental updates and the index information within each data block without incremental updates.

11. The data management method according to claim 10, wherein constructing an index tree within the merged result of the SSTable based on the index information within the update result of each data block with incremental updates and the index information within each data block without incremental updates includes: Load the index information within data blocks that do not have incremental updates into memory; Load the index information within the update results of the data blocks with incremental updates into memory; The index information in memory is written to disk into the merged result of the SSTable, and an index tree is built based on the index information written to disk.

12. The data management method according to claim 11, wherein writing the index information in memory to disk into the merged result of the SSTable and constructing an index tree based on the written index information includes: When the index information in memory reaches the disk write threshold, the index information in memory is written to disk into the merged result of the SSTable; In response to the fact that the index information of all data blocks in the merged result of the SSTable has been written to disk, an index tree is constructed based on the index information written to disk.

13. The data management method according to claim 10, further comprising: The index tree within the merged results or DDL operation results of the SSTable is loaded into each database by the object storage layer.

14. A data query method, the method comprising: The data block containing the data targeted by the SQL request is determined based on the index tree within the ordered string table SSTable, and the data targeted by the SQL request is read from the data block.

15. A data management device, the device comprising: The update module is used to sequentially determine whether the data in each data block of the ordered string table SSTable has incremental updates, and update the data and index information in the data blocks with incremental updates to obtain the update result; a merging module configured to generate a merging result of the SSTable based on data in the update result of each data block with the incremental update and data in each data block without the incremental update; an indexing module configured to construct an index tree in the merging result of the SSTable based on index information in the update result of each data block with the incremental update and index information in each data block without the incremental update.

16. The data management apparatus of claim 15, further comprising a preheating module configured to: load the index tree in the merging result of the SSTable or the DDL operation result to each database by the object storage layer.

17. A data query apparatus, comprising: a query module configured to determine a data block where data targeted by a SQL request is located based on an index tree in a sorted string table (SSTable) and read the data targeted by the SQL request in the data block.

18. A computer program product comprising computer programs / instructions which, when executed by a processor, implement the steps of the method of any one of claims 10 to 14.

19. An electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the method of any one of claims 10 to 14 by running the executable instructions.

20. A computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the method of any one of claims 10 to 14.

Images