Log index generation method, log reading method, and database system

By reading and generating log index entries in the log space, the system burden problem caused by existing log index generation methods is solved, achieving efficient log index storage and access, and improving the performance and reliability of the database system.

WO2026232970A1PCT designated stage Publication Date: 2026-11-12CLOUD INTELLIGENCE ASSETS HOLDING (SINGAPORE) PTE LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/083223
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-09
Filing Date
2026-03-13
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing log index generation methods result in a large number of fine-grained I/O operations, increasing the system burden. Furthermore, they lack effective utilization of the relationships between log items, making it difficult to achieve efficient merging and batch writing operations, thus limiting the overall optimization space for log processing.

Method used

The target replay log entry is read from the log space where the replay log is stored, a log index entry is generated, and the next replay log entry is retrieved from the log space until at least two log index entries are generated and then stored persistently. Asynchronous threads are used to improve read efficiency and avoid negative impacts on normal write operations.

Benefits of technology

It achieves efficient caching, merging, and persistent storage of replay log entries in the log space, improves the efficiency of log index generation and access, and ensures the operational security and fault recovery capabilities of the database system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026083223_12112026_PF_FP_ABST
    Figure CN2026083223_12112026_PF_FP_ABST
Patent Text Reader

Abstract

A log index generation method, a log reading method, and a database system. The method comprises: reading a target redo log entry in a log space storing a redo log (102); generating a log index entry on the basis of log information of a redo log entry, and acquiring, in the log space, a next redo log entry associated with the redo log entry (104); using the next redo log entry as the redo log entry (106); until at least two log index entries are generated, and generating a log index on the basis of the at least two log index entries, and then persistently storing the log index (108).
Need to check novelty before this filing date? Find Prior Art

Description

Log index generation methods, log reading methods, and database systems Cross-reference to related applications

[0001] This disclosure claims priority to Chinese patent application No. 202510602360.7, filed on May 9, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of database technology, and in particular to a log index generation method, a log reading method, and a database system. Background Technology

[0003] With the continuous development and widespread application of database systems, distributed storage systems, and transaction processing systems, the log mechanism, as an important means to ensure data consistency and support fault recovery, directly affects the overall operation of the system. Traditional log management methods typically use sequential writing to record the redo log to disk to ensure that uncommitted transactions can be recovered through log replay in the event of a system failure.

[0004] Existing technologies propose constructing log index structures to accelerate the location and access of replay log entries. However, these methods often perform persistent writing immediately after each log index entry is generated, resulting in a large number of fine-grained I / O operations, increasing system load and degrading overall performance. Furthermore, existing index construction processes lack effective utilization of the relationships between log entries, making efficient merging and batch writing operations difficult, thus limiting the overall optimization potential of log processing. Therefore, a more effective log index generation method is urgently needed to address these issues. Summary of the Invention

[0005] In view of this, embodiments of this disclosure provide a log index generation method. One or more embodiments of this disclosure also relate to a log reading method, a database system, a log index generation system, a log index generation apparatus, a log reading apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the present disclosure, a log index generation method is provided, comprising: reading a target replay log item from a log space storing replay logs; generating a log index item based on the log information of the replay log item, and obtaining a next replay log item associated with the replay log item from the log space; using the next replay log item as the replay log item, and performing the step of generating a log index item based on the replay log item; until at least two log index items are generated, generating a log index based on the at least two log index items, and persistently storing the log index.

[0007] According to a second aspect of the present disclosure, a log reading method is provided, comprising: determining a rewrite log range corresponding to log information to be read; within the rewrite log range, determining segment information of an index item segment of the log to be read by querying a segment relationship file based on the log information to be read; querying an index storage file based on the segment information to obtain target index information associated with the log information to be read, and reading a target log in the log space based on the target index information.

[0008] According to a third aspect of the present disclosure, a log index generation system is provided, applied to a log index engine of a database system, the database system including a storage engine and a storage layer; the storage engine determines a log space for storing replay logs and reads a target replay log item from the log space; a log index item is generated based on the log information of the replay log item, and the next replay log item associated with the replay log item is obtained from the log space; the next replay log item is used as the replay log item, and the step of generating a log index item based on the replay log item is executed; until at least two log index items are generated, a log index is generated based on the at least two log index items, and the log index is stored in the storage layer for persistent storage.

[0009] According to a fourth aspect of the present disclosure, a database system is provided, including a storage engine, a log index engine, and a storage layer; the storage engine is configured to generate and store multiple replay log entries in the order of operations on data pages; the log index engine is configured to read multiple replay log entries from the storage engine, generate a log index according to a log generation method, and store the log index in the storage layer; the storage layer is configured to persistently store the log index generated by the log index engine.

[0010] According to a fifth aspect of the present disclosure, a log index generation apparatus is provided, comprising: a reading module configured to read a target replay log item from a log space storing replay logs; a generation module configured to generate a log index item based on log information of the replay log item, and to obtain a next replay log item associated with the replay log item from the log space; an execution module configured to use the next replay log item as the replay log item, and to execute the step of generating a log index item based on the replay log item; and a storage module configured to generate a log index based on the at least two log index items until at least two log index items are generated, and to persistently store the log index.

[0011] According to a sixth aspect of the present disclosure, a log reading apparatus is provided, comprising: a determining module configured to determine a rewrite log range corresponding to log information to be read; a querying module configured to determine segment information of an index item segment of the log to be read within the rewrite log range by querying a segment relationship file based on the log information to be read; and a reading module configured to query an index storage file based on the segment information to obtain target index information associated with the log information to be read, and to read a target log in the log space based on the target index information.

[0012] According to a seventh aspect of the present disclosure, a computing device is provided, comprising: a memory and a processor; the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the steps of the above-described log index generation method.

[0013] According to an eighth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the log index generation method described above.

[0014] According to a ninth aspect of the present disclosure, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the log index generation method described above.

[0015] This disclosure provides a log index generation method according to one embodiment, which involves reading a target replay log item from a log space storing replay logs; generating a log index item based on the log information of the replay log item; obtaining the next replay log item associated with the replay log item in the log space; using the next replay log item as the replay log item; and generating a log index item based on the replay log item, until at least two log index items are generated. In other words, for the log space storing replay log items, each replay log item is read one by one, and a corresponding log index item is generated. Following the storage order of the replay logs in the log space, one replay log item is read, a corresponding log index item is generated, and stored in memory. This read-generate operation is repeated until at least two log index items are generated. Then, a log index is generated based on these at least two log index items, and the log index is persistently stored. This achieves efficient caching, merging, persistence, and normalized storage of replay log items in the log space. Subsequently, the replay log items stored in the log space can be quickly accessed based on the generated log index. Attached Figure Description

[0016] Figure 1 is a flowchart of a log index generation method provided in an embodiment of this disclosure.

[0017] Figure 2 is a flowchart of a log index generation method provided in an embodiment of this disclosure.

[0018] Figure 3 is a database architecture diagram of a log index generation method provided in an embodiment of this disclosure.

[0019] Figure 4 is a schematic diagram of log index storage for a log index generation method provided in an embodiment of this disclosure.

[0020] Figure 5 is a log access diagram of a log index generation method provided in an embodiment of this disclosure.

[0021] Figure 6 is a pointer-based query diagram of a log index generation method provided in an embodiment of this disclosure.

[0022] Figure 7 is a flowchart of a log reading method provided in an embodiment of this disclosure.

[0023] Figure 8 is a schematic diagram of the structure of a log index generation system provided in an embodiment of this disclosure.

[0024] Figure 9 is a schematic diagram of the structure of a database system provided in an embodiment of this disclosure.

[0025] Figure 10 is a schematic diagram of the structure of a log index generation device provided in an embodiment of this disclosure.

[0026] Figure 11 is a schematic diagram of the structure of a log reading device provided in an embodiment of this disclosure.

[0027] Figure 12 is a structural block diagram of a computing device provided in an embodiment of this disclosure. Detailed Implementation

[0028] Numerous specific details are set forth in the following description to provide a full understanding of this disclosure. However, this disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this disclosure. Therefore, this disclosure is not limited to the specific implementations disclosed below.

[0029] The terminology used in one or more embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this disclosure. The singular forms “a,” “the,” and “the” as used in one or more embodiments of this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this disclosure refers to and includes any or all possible combinations of one or more associated listed items.

[0030] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this disclosure, and similarly, second may also be referred to as first. Depending on the context, the word “if” as used herein may be interpreted as “when”, “in response to a determination”, or “when…”.

[0031] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this disclosure are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0032] First, the terms and concepts involved in one or more embodiments of this disclosure will be explained.

[0033] Redo Log: This log records all modifications made to the database. In the event of a system crash or transaction failure, replaying the Redo Log allows the database to be restored to its most recent consistent state, ensuring data integrity and consistency.

[0034] Page: The physical structure and basic unit used in a database system to organize and manage data.

[0035] Physical replication: A method of replicating database nodes by using Redo Logs to track the update status of Pages.

[0036] Logindex: A way to organize redo logs, efficiently indexing the redo log of a specified page.

[0037] Primary: Read / write node, also called RW node, provides read and write services.

[0038] Secondary: A read-only node, also called a RO node, which follows the primary node's writes through physical replication and provides read-only services.

[0039] LSN: Log Sequence Number, is a unique number used to identify the position of a redo record in the redo log file. Each redo record has a unique LSN, which indicates the physical location or order of the record in the redo log. Using the LSN, specific log records can be accurately located and applied in the correct order to restore the database state, especially during crash recovery.

[0040] MTR (Mini-Transaction): A logically smallest unit of transaction, corresponding to a set of redo log records that need to be atomically committed.

[0041] This disclosure provides a log index generation method, and also relates to a log reading method, a database system, a log index generation system, a log index generation device, a log reading device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0042] Referring to Figure 1, Figure 1 shows a flowchart of a log index generation method according to an embodiment of the present disclosure, which specifically includes the following steps:

[0043] Step 102: Read the target replay log entry from the log space where the replay log is stored.

[0044] The replay log, also known as the redo log, records all modifications made to the database. In the event of a system crash or transaction failure, replaying the redo log allows the database to be restored to its most recent consistent state, ensuring data integrity and consistency.

[0045] Log index generation methods can be applied to database systems, which include log indexing engines, storage engines, and shared storage (storage layer). The log space corresponds to the storage engine, where replay logs are stored. When it's necessary to build log index entries based on replay logs and persistently store these entries as log indexes, the target replay log entry for indexing can be read from the log space corresponding to the storage engine.

[0046] It should be noted that both the storage engine and the log indexing engine are integrated into the compute nodes of the database system. These compute nodes can be read-write nodes (Primary, i.e., master nodes) or read-only nodes (Secondary, i.e., slave nodes). Both master and slave nodes contain the log indexing engine and the storage engine. In the master node, the log indexing engine reads the target replay log from the storage engine and generates the corresponding log index entry, which is temporarily stored in memory. Once the log index entry in memory meets the conditions for persistent storage, it is written to disk and stored in shared storage, thus achieving persistent storage of the log index entry.

[0047] For example, the purpose of reading the target replay log in the log space is to generate a log index entry for the target replay log, and then persist the log index entry. This allows for subsequent database system fault recovery based on the persistently stored log index entry in shared storage, ensuring the operational security of the database system. The persistently stored log index entry in shared storage can also be used to access the replay log. The replay log is read based on the log index entry and applied to the corresponding data page, thereby achieving data persistence after transaction commit, data synchronization of read-only nodes, and on-demand replay of the replay log by read-only nodes. The database system can adopt a PolarDB cluster architecture based on shared storage. The read / write nodes in PolarDB are the master nodes. The replay log can be stored in the storage engine of the master node. The master node also includes a log index engine, used to read the replay log from the storage engine and generate the corresponding log index entry.

[0048] Step 104: Generate a log index item based on the log information of the replay log item, and obtain the next replay log item associated with the replay log item in the log space.

[0049] A log index entry is a log index corresponding to a replay log entry. It records the replay log's offset information, data page information, and log sequence number. Log index entries can be stored using a B+Tree index structure. The next replay log entry associated with a replay log entry in the log space can be a replay log entry read from the log space based on the time sequence of its generation. The replay log entry and the next replay log entry correspond to the same data page.

[0050] For example, in the log space, replay logs are arranged in chronological order of their generation, or stored together with replay logs corresponding to the same data page. After the target replay log item is read from the log space corresponding to the storage engine in the master node of the database system, and a log index item corresponding to the target replay log item is generated, the next replay log item corresponding to the target replay log item can be selected in the log space, and a next log index item can be generated for the next replay log item. The next log index item is temporarily stored in memory, waiting to be written to disk.

[0051] Step 106: Use the next replay log entry as the replay log entry, and execute the step of generating a log index entry based on the replay log entry.

[0052] Step 108: Generate at least two log index entries, generate a log index based on the at least two log index entries, and persist the log index.

[0053] A log index is a log index obtained by integrating at least two log index entries. After the log index is persisted, replay logs can be accessed based on the persisted log index.

[0054] The log indexing engine reads the target replay log from the storage engine and generates a target log index corresponding to the target replay log, temporarily storing the corresponding target log index in memory. Then, it reads the next replay log from the storage engine, continues to generate the next log index entry for the next replay log, and temporarily stores it in memory, and so on, until the log index entries accumulated in memory exceed the size of a rewrite log file, or exceed a preset threshold, or when a log file switch is sent, at least two log index entries accumulated in memory are uniformly persisted.

[0055] In practical applications, the log index corresponding to the replay log is a metadata structure organized by data page dimension, establishing a mapping relationship between the page number and the corresponding replay log. For example, the log index file stored using a B+Tree structure is generated asynchronously and persisted to shared storage. It should be noted that the log index of the replay log is a metadata structure organized by data page dimension.

[0056] It should be noted that the log space stores replay log entries corresponding to multiple versions of data pages. When data deletion, data addition, and data modification operations are performed on the target data page sequentially, each data operation generates a corresponding replay log entry, which is stored in the log space. When generating log index entries subsequently, multiple versions of log index entries can also be generated. Storing the corresponding multi-version log indexes in shared storage allows the compute nodes in the database system to recognize and utilize the multi-version log indexes in shared storage. By accessing and rewriting logs based on the log indexes, multi-version data page file access can be achieved.

[0057] In the log space, following the storage order of replay logs, each replay log entry is read, a corresponding log index entry is generated, and stored in memory. This read-generate operation is repeated until at least two log entry index entries are generated. Based on these at least two index entries, a log index is generated and persistently stored. This achieves efficient caching, merging, persistence, and normalization of replay log entries in the log space. Subsequently, the replay log entries stored in the log space can be quickly accessed based on the generated log index.

[0058] To improve the efficiency of reading and processing target replay logs, an asynchronous thread can be added to read the target replay logs from the log space. In one optional embodiment of this disclosure, reading target replay log entries from the log space storing replay logs includes:

[0059] Determine the log space corresponding to the database system, and determine the replay log entries generated by the operation on the data pages in the database system in the log space, wherein the replay log entries are associated with multiple versions of the data pages;

[0060] The target replay log entry is determined from the replay log entries stored in the log space using an asynchronous thread.

[0061] The database system can adopt a PolarDB cluster architecture based on shared storage. The read and write nodes in PolarDB are master nodes. Replay logs can be stored in the storage engine of the master node. The master node also includes a log index engine, which is used to read replay logs in the storage engine and generate log index entries corresponding to the replay logs.

[0062] During operations on data pages in a database system, a replay log of the database system can be generated. The replay log can be generated in batches, sequentially, or in parallel during these operations; no specific limitation is made here.

[0063] Replay log entries are associated with multiple versions of data pages. In a database system, when operations are performed on data pages, corresponding replay log entries (Redo) are generated for each operation according to the chronological order of the operation time. One replay log entry corresponds to one version of the data page. Different versions of data pages are generated by continuously operating on the data pages iteratively, and multiple versions of data pages are recorded in the form of a series of replay log entries.

[0064] In practical applications, the log space can be a redo cache that stores replay logs. When reading a target replay log item, an asynchronous thread can be used to read it directly from the replay cache.

[0065] For example, in a database system, when a transaction is committed, replay log entries are generated based on multiple data page operations accumulated in memory, and these replay log entries are appended to the log space in batches. To facilitate the management of replay logs in the log space, a log index can be generated based on the replay log entries in the log space. An asynchronous thread reads the target replay log entry from the log space, and subsequently, a log index entry for the target replay log entry can be generated based on information such as the transaction ID, operation type, and data page address of the read target replay log entry.

[0066] In this embodiment, asynchronous reading of the target replay log item in the database system's log space improves the efficiency of replay log item reading. When generating the log index item corresponding to the target replay log item, an asynchronous generation method can also be used based on an asynchronous thread. This ensures that the write path of the replay log item remains unchanged while reading the target replay log item, avoiding negative impacts on normal replay log item write operations during index generation.

[0067] Since replay log entries are generated sequentially in a streaming manner, maintaining a global log index is not feasible because it is impossible to determine how much storage space a log index entry corresponding to a data page will require. This would lead to complex space allocation and reclamation strategies, as well as significant space and I / O amplification. In one optional embodiment of this disclosure, the target replay log entry is determined from the replay log entries stored in the log space using an asynchronous thread, including:

[0068] In the log space, the target replay log file is determined, and the replay log entries in the replay log text are divided into at least one log entry interval with an associated arrangement order according to the data page identifier corresponding to the replay log entries.

[0069] The target replay log entry is determined in chronological order from the replay log entries contained in the at least one log entry interval.

[0070] A target replay log file can be a batch of replay logs obtained by dividing the replay logs in the log space according to the generation time dimension. The target replay log file can be represented as a "batch," meaning a replay log partition. A target replay log file contains multiple replay log entries, which may correspond to at least one data page. Therefore, the replay log entries in the target replay log file can be divided into at least one log entry interval corresponding to different data pages at the data page dimension. That is, the replay log entries contained in the target replay log file are summarized according to the data page identifier, and replay log entries with the same data page identifier are grouped into one log entry interval. The order of the at least one log entry interval can be sorted according to the size of the data page identifier.

[0071] In practical applications, determining the target replay log file within the log space can be viewed as partitioning and sorting the replay log entries in the log space. The size of the target replay log file corresponds to the size of the redo file; one redo file can correspond to one target replay log file, forming a replay log partition, i.e., a batch. The size of a replay log partition can correspond to the size of one redo file, or it can be the size of reserved memory space. The actual size of a replay log partition is limited by the amount of memory available for sorting and the acceptable latency of the read-only node.

[0072] In practical implementation, a Memory Redo Hash is configured in the read-only nodes of the database system. This prevents the read-only nodes from accessing the latest generated log index entries in the read-write nodes. Therefore, the generation of log index entries can be significantly delayed, and the read-only node access time for log index entries can be reduced to 500MB-1GB. Considering the memory usage for sorting and the size of a single I / O operation when generating log index entries, a 100MB replay log partition size can be used in practical applications. That is, the replay log entries in the log space are divided according to a 100MB replay log partition size.

[0073] For replay log entries in the target replay log file, replay log entries can be aggregated according to data page identifiers. The aggregated replay log entries are then used as log entry ranges. After generating log index entries for each replay log entry within a log entry range, the log index entries are temporarily stored in memory as aggregates of log entry ranges. Persistence storage is then performed after the log index corresponding to the target replay log file is generated, storing the log index in shared storage. The log index corresponding to a range can use a header to indicate the data page corresponding to the range and the location for storing a set of redo bodies. The header can contain Space ID, Page Number, Range Length, and Range Offset; the redo body can contain information such as type, starting LSN, and body length. When the cached range content in a batch exceeds a threshold size, or when a redo file switch occurs, it will be written to disk and stored in shared storage.

[0074] In this embodiment of the disclosure, by setting the target replay log file and log entry range, complex space allocation and reclamation strategies and significant space and IO amplification are avoided.

[0075] To ensure the continuity and order of log item intervals, the log item intervals can be linked together first and last in the order they are arranged to form an index item interval chain. In one optional embodiment of this disclosure, generating a log index based on the at least two log index items includes: determining an interval log index item corresponding to each log item interval from the at least two log index items; obtaining an index item interval chain associated with the interval log index item corresponding to each log item interval by connecting each log item interval according to the order they are arranged; and generating the log index based on the at least two log indexes contained in the index item interval chain, provided that the index item interval chain meets the storage conditions.

[0076] An index entry for a log entry range can be the preceding log entry for each log entry range in the target replay log file. Linking each log entry range in the target replay log file with its preceding log entry yields an index entry range chain. This chain is formed by sequentially connecting the first and last log entry ranges in the target replay log file according to their arrangement. The connections between log entry ranges within the index entry range chain can be represented by pointers, and range pointer information is generated based on these connections to indicate the sequential order of the log entry ranges. The index entry range chain satisfies a storage condition, either by reaching a preset storage space threshold or by a Redo file switch.

[0077] For example, the log index is stored according to log item intervals, where the interval log index item corresponding to a log item interval refers to the preceding log item interval. By concatenating the first and last log item intervals, an index item interval chain containing interval connection relationships can be obtained. When the storage space occupied by the generated index item interval chain reaches a preset storage space threshold, the index log corresponding to the index item interval chain can be persistently stored.

[0078] In this embodiment of the disclosure, an index item interval chain containing interval connection relationships can be obtained by connecting the first and last log item intervals. When storing the log index, the structured storage of the log index items is realized, which facilitates subsequent log access for rewriting the log index.

[0079] In one optional embodiment of this disclosure, persistent storage of the log index includes:

[0080] Identify at least two log index entry segments associated with the log index, and persist the log index according to the segment order of the at least two log index entry segments.

[0081] The log index entry segment indicates the storage format or aggregation format of the log index when it is stored in the database's shared storage. In shared storage, the log index is stored in the database's storage layer as log index entry segments. Each log index entry segment contains log index entries corresponding to the same data page. In other words, the log index contains multiple generated log index entries corresponding to replay logs. Since each replay log has its corresponding data page, when performing disk persistence on the log index, it can be aggregated according to the data page dimension to obtain the log index entry segment before being stored in the shared storage.

[0082] For example, the log index contains multiple log index entries, each corresponding to a data page. When the log index corresponds to data page 1, data page 2, and data page 3, during the log index write-to-disk process, a set of log index entries corresponding to data page 1, data page 2, and data page 3 can be found among the multiple log index entries in the log index, and each set of log index entries is considered a log index entry segment. Based on the size of the data page, the three log index entry segments are stored concatenated in shared storage.

[0083] In this embodiment of the disclosure, the log index is persistently stored according to the segment order of at least two log index item segments, thereby achieving formatted and standardized storage of the log index, which facilitates subsequent access to the log index.

[0084] To improve the efficiency of reading log index entries in persistent storage, a segment relationship file can be maintained while persisting at least two log index entry segments. This file records the storage order and pointer-level dependencies of at least two log index entry segments in the index storage file. In one optional embodiment of this disclosure, persistently storing the log index according to the segment order of the at least two log index item segments includes: determining an index storage file corresponding to the log space; determining a log index item segment to be stored among the at least two log index item segments; storing the segment association relationship to a segment relationship file if a segment association relationship is established between the log index item segment to be stored and the preceding log index item segment; and storing the log index item segment to be stored in the index storage file; determining a next log index item segment among the at least two log index item segments, and using the next log index item segment as the log index item segment to be stored, and performing the step of storing the log index item segment to be stored in the index storage file if a segment association relationship is established between the log index item segment to be stored and the preceding log index item segment; until the at least two log index item segments are stored in the index storage file, and persistently storing the index storage file and the segment relationship file.

[0085] The index store file is used to temporarily store in memory the log index entries contained in at least two log index entry segments. The segment relationship file is used to temporarily store in memory the association relationships between at least two log index entry segments; that is, the segment association relationship between the log index entry segment to be stored and its predecessor log index entry segment. This segment association relationship is a pointer relationship between the two segments. Storing these pointer relationships in the segment relationship file, the segment relationship file can be represented as `ib_parsemeta`. Each log index entry segment, concatenated end-to-end, can be stored in the index store file, which can be represented as `ib_parsedata`.

[0086] In practical applications, after determining at least two log index entry segments (Range), each Range can be stored sequentially in the in-memory index storage file. The at least two log index entry segments (Range) are stored in the index storage file in a concatenated manner. Before each Range is written to the index storage file, the position of the last Range on the corresponding data page is queried from the memory structure of the segment relationship file, and the identifier field is updated. After each Range is written to the index storage file, it needs to be registered in the memory structure of the segment relationship file, forming a linked list structure.

[0087] In practical applications, ib_parsemeta may be frequently modified during the write process from the Range to ib_parsedata. Thanks to the read-only node's tolerance for the lag in log index generation, each ib_parsemeta file is actually only flushed to disk once at the end. Only after ib_parsemeta is written to disk is the Logindex truly accessible; that is, the segment relationship file is persistently stored. Therefore, a lag of one Redo file size in Logindex will actually be maintained.

[0088] For example, each log index entry segment sequentially stores the log index entries corresponding to the replay logs. Each log index entry segment is joined according to the aforementioned arrangement to obtain an index entry interval chain associated with the interval log index entries corresponding to each log index entry segment. The log index entries in the index entry interval chain are the log index entries divided and stored according to the log index entry segments. The index entry interval chain corresponds to the generated log index. When the index entry interval chain meets the storage conditions, it indicates that during the caching of the index entry interval chain in memory, the space occupancy corresponding to the index entry interval has reached a preset space occupancy threshold, or the replay log of a Redo file has completed the construction of the log index entries, or a Redo file switch has occurred. At this time, the log index corresponding to the index entry interval chain can be persistently stored in the shared storage of the database system.

[0089] In this embodiment of the disclosure, persistent storage of the log index is achieved by storing and recording log index item segments with adjacent arrangement relationships according to the association relationship, as well as structured and standardized storage of the log index, so that the stored log index has high availability.

[0090] In this embodiment of the disclosure, by persistently storing the index storage file and the segment relationship file, the storage order and pointer-level dependency relationship of at least two log index item segments in the index storage file are recorded, thereby improving the reading efficiency of the log index in the persistent storage space.

[0091] In one optional embodiment of this disclosure, after persistently storing the index storage file, the method further includes: when a read-only node associated with a read-write node reads rewrite logs from the index storage file, determining the rewrite log range based on the log information to be read; within the rewrite log range, determining the segment information of the log index item segment to be read by querying the segment relationship file based on the log information to be read; querying the index storage file based on the segment information to obtain target index information associated with the log information to be read, and reading the target log in the log space based on the target index information.

[0092] Both read-write nodes and read-only nodes are computing nodes in the database system. Typically, a read-only node initiates a read request to read the rewrite log from the index storage file stored in shared storage. The log information to be read is the relevant information of the rewrite log that the read-only node determines to be read before performing the rewrite log read. This log information may include the log range of the rewrite log to be loaded, which contains data page version information, checkpoint information, etc.

[0093] The segment information of the log index segment to be read is the segment identifier and segment position corresponding to the segment information to be read. Once the segment identifier and segment position are determined, the index storage file can be queried based on the segment identifier and segment position to obtain the target index information. The target log can then be read based on the target index information, and subsequently applied to the corresponding data pages.

[0094] In this embodiment of the disclosure, the target index information of the associated log information to be read is obtained by querying the index storage file based on the segment information. Subsequently, the target log is read in the log space based on the target index information, so as to achieve accurate and efficient reading of the target log.

[0095] In one optional embodiment of this disclosure, after reading the target log in the log space based on the target index information, the method further includes: determining the target data page associated with the log information to be read, and processing the target data page based on the target log.

[0096] In a database system, read-only nodes can access replay logs by reading log indexes in shared storage. After reading the replay log corresponding to the log information to be read, the replay log can be applied to the target data page, and the target data page can be processed based on the target log.

[0097] In practical applications, processing the target data page based on the replay log corresponding to the log information to be read can be used to solve data consistency problems, i.e., to ensure data consistency. For example, when a transaction is committed, the replay log has been persisted, but the target data page may still be in memory (such as the buffer pool). If a failure occurs at this time, the data page needs to be updated to disk via the replay log upon restart. Read-only nodes can update the data pages in their local buffer pool by replaying the replay log, maintaining consistency with read-write nodes and ensuring data consistency.

[0098] For example, processing target data pages based on target logs can also be used for fault recovery and ensuring data integrity. Replay logs record all non-persistent data changes; during fault recovery, data pages need to be reconstructed by replaying the replay log records. If replay logs are not applied to data pages, the data pages may remain in an old version, leading to inconsistent query results. When the database starts, the process typically replays incomplete transactions using the replay logs to update the data pages in the Buffer Pool to the latest state. During master-slave replication, slave nodes (read-only nodes) synchronize data with the master node by applying replay logs.

[0099] In this embodiment of the disclosure, by processing the target data page based on the target log, the target log is applied to the target data page, thereby ensuring data consistency between master and slave nodes and ensuring data integrity during fault recovery.

[0100] The following description, in conjunction with Figure 2, uses the application of the log index generation method provided in this disclosure in a database system as an example to further illustrate the log index generation method. Figure 2 shows a flowchart of the processing steps of a log index generation method according to an embodiment of this disclosure, specifically including the following steps.

[0101] Step 202: Determine the log space corresponding to the database system, and use an asynchronous thread to identify the target replay log entry from the replay log entries stored in the log space.

[0102] As shown in Figure 3, taking PolarDB as an example, the database system contains master nodes and slave nodes. A master node can be a read-write node, represented as Primary; a slave node can be a read-only node, represented as Secondary. Both master and slave nodes are compute nodes. Master nodes are generally used for writing data pages, while slave nodes are generally used for writing data pages. The master node contains a server engine, an InnoDB storage engine, and a multi-version engine. Correspondingly, slave nodes also contain a server engine, an InnoDB storage engine, and a multi-version engine.

[0103] In the master node, the storage engine includes transactions, B+trees, a buffer pool, and redo logs. The multi-version engine includes a flush pool, sequence redo hashes, and page redo hashes. Sequence redo hashes and page redo hashes are used to merge redo logs in memory by page and create a logindex for the redo logs based on the page. The flush pool caches dirty pages awaiting write-back and maintains valid versions.

[0104] In the slave node, the storage engine includes transactions, B+trees, a buffer pool, and redo apply. The multi-version engine includes memory redo hashes and persistent redo hashes for Logindex-based log access. For slave nodes, the persistent redo hash maintained in memory is used to index all relevant redo logs page-by-page. This allows the secondary node to retrieve a specific version of the page file, obtain the best-fit, persisted page file, and collect the redo logs that need to be applied for accurate page recovery. This PolarDB system architecture allows for master-slave decoupling during the subsequent generation and use of Logindex.

[0105] In practical applications, asynchronous threads can be used to read the target replay log entries that need to be indexed from the rewrite log cache (log space), i.e., the redo log. After building the Logindex from the read target replay log, it is organized in memory according to pages.

[0106] Step 204: After generating log index entries for the read target replay log entries, organize them into multiple log index entry segments in memory according to data pages.

[0107] After reading the redo and generating the corresponding logindex using an asynchronous thread, the logindex of the same page can be organized in memory according to the page. The logindex of the same page can be organized into a log index item segment, namely, a range. By continuously reading the redo and generating the logindex, multiple ranges can be continuously obtained and accumulated in memory.

[0108] In practical applications, PolarDB has made extensive multi-threaded and lock-free modifications to its log generation logic, resulting in a relatively efficient Redo generation process. However, Redo write-to-disk is a critical path for transaction commits and dirty data flushing, and its performance significantly impacts the entire database. Furthermore, the MTR-based Redo log generation process results in discrete Redo logs for individual pages. Maintaining Logindex along the critical path of Redo generation would lead to significant performance impacts and metadata bloat. Therefore, this embodiment employs asynchronous Logindex generation. This means keeping the main Redo write path unchanged, using Redo Cache capabilities to cache the last small segment of Redo in memory, adding an asynchronous thread to read Redo from the Redo Cache, parse and generate logindex, and persistently storing the accumulated logindex.

[0109] In practical implementation, during the Logindex generation process, it's crucial to ensure that the generation speed of the page-based Logindex can keep pace with the already highly efficient Redo Log generation, while avoiding excessive CPU and I / O consumption. Since Redo is generated sequentially in a streaming manner, maintaining a global Logindex is impractical because the exact space required for a single page's index cannot be determined, leading to complex space allocation and reclamation strategies and significant space and I / O amplification. Therefore, this embodiment employs a partitioned sorting approach, generating a Logindex for each page within a specific Redo Log segment. An asynchronous thread sequentially reads the Redo Log and, according to...<Space ID,Page ID,LSN> The data is sorted in memory. When the accumulated content exceeds the size of a redo file or reaches the reserved memory limit, it is flushed to disk, generating a segmented Logindex. The final result, viewed globally, forms a partitioned Logindex.

[0110] The size of a partition is actually limited by the amount of memory available for sorting and the acceptable latency of the read-only node. Too small a segment leads to poor Logindex merging performance and low Logindex query efficiency, while too large a segment results in excessive memory consumption, delayed Logindex generation, and a backlog of single-time I / O operations. Based on the Memory Redo Hash in the slave node, the most recently generated Logindex is not accessed. Therefore, Logindex generation can be significantly delayed; according to the online configuration, the allowable latency for slave node access to the Logindex is at least 500MB-1GB. Considering the memory consumption for sorting and the size of a single I / O operation when generating the Logindex, a partition (batch) size of 100MB LSN was used in practice.

[0111] Step 206: Accumulate log index entry segments. If the accumulated log index entry segments meet the storage conditions, store the accumulated log index entry segments to the index storage file, and update the segment relationship file according to the write results.

[0112] In memory, log index entries are accumulated in the range. When the cached range content in a batch exceeds the threshold size, or when a redo file switch occurs, or when a redo file size is reached, the index storage file (ib_parsedata) is written to disk. During this process, all ranges in the batch are concatenated sequentially, and I / O is performed in an append-only manner, which ensures efficient I / O and thus guarantees the overall generation speed of Logidnex.

[0113] PolarDB's default redo file size is 1GB, and file allocation, purging, and backup are all performed at the file level. The lifecycle of a logindex is exactly the same as its corresponding redo file; therefore, the ib_parsedata file is also implemented one-to-one bound to its corresponding redo file, meaning that each ib_parsedata file records the logindex of a record in a redo file.

[0114] In addition, a corresponding ib_parsemeta file (segment relationship file) is added to each ib_parsedata file to index the Ranges within all files. The ib_parsemeta file is a file hash implementation from Page number to Range, with a default size of 50MB, which can cover almost all the Pages required in various scenarios. If the number of Pages exceeds this, it will expand backward to handle hash collisions. Each file may contain multiple Ranges for the same Page; only the position of the last Range is recorded in the ib_pasemeta file, in the corresponding field. The preceding Range is indexed by the forward pointer of the last Range.

[0115] As shown in Figure 4, taking Batch 1 of a 100MB rewrite log as an example, it includes the ranges for data page 3, data page 6, and data page 100. When storing the ranges, they are stored in an index storage file concatenated end-to-end, and index pointers are created in the segment relationship file. Subsequent batches, such as Batch 2, are concatenated with the preceding batches.

[0116] Step 208: Determine the range of rewrite logs to be read for the data page to be read, and determine the rewrite file associated with the data page to be read based on the rewrite log range.

[0117] It should be noted that there is no necessary execution order between step 208 and the preceding steps 202-206 in terms of execution time. At any time after the Logindex is generated and written to disk, log access can be achieved by executing steps 208-212.

[0118] Slave nodes can read the Logindex from shared storage at any time after it is written to disk, and then read the corresponding rewrite logs based on the Logindex. When accessing the logs, the range of rewrite logs to be read must first be determined, for example: [Min(Page_LSN,Chkp_LSN),Memory_lwm]. Here, Page_LSN is read directly from the Page, Chkp_LSN is the current checkpoint position on the master node obtained from control information exchanged with the master node, and Memory_lwm is the smallest LSN position retained in the Memory Redo Hash. All Redo files containing this Page are filtered within this range; a Bloom filter can help narrow down the range.

[0119] Step 210: Determine the segment relationship file to be read based on the data to be read and the rewritten file, and determine the segment position of the target segment in the rewritten file by querying the segment relationship file to be read.

[0120] The required ib_parsemeta files are read sequentially to obtain the last range position of the required page in the file. To avoid read I / O amplification, the contents of ib_parsemeta can be cached in memory at 16KB MetaBlock granularity, and it is determined whether to reload from the file based on the current LSN.

[0121] Step 212: Determine the target log index entry in the log index based on the segment location, and determine the target rewrite log based on the target log index entry.

[0122] Based on the Range Offset, read the Logindex content at the corresponding position, and follow the forward pointer of the Range to find all available Ranges and obtain all Logindexes. According to the position marked by the Logindex, retrieve the required Log Records from the corresponding Redo Logs and apply them to the corresponding Pages.

[0123] In practical applications, the process of a slave node accessing the rewrite log via Logindex is illustrated in Figure 5. As shown in Figure 5, the slave node first determines the range of Redo Logs to be loaded, which defines the log interval to be loaded from persistent storage. Bloom filters and other techniques are used to quickly filter out Redo files containing modification records of the target page, avoiding a full scan of all log files. The ib_parsemeta file is read to obtain the position of the last Range of the target page in the log file and determine the offset information of that Range. Then, the Logindex is read, and based on the Logindex, the Redo Logs to be read are determined. The slave node can read the rewrite logs (Redo Logs) based on the Logindex and shared storage, and apply the read rewrite logs. In specific implementations, the Logindex can be queried in conjunction with the ib_parsemeta file to access the replay logs. As shown in Figure 6, there are pointer relationships between Range1, Range2, Range3, and other Ranges, i.e., a forward pointer approach is used, making each subsequent Range point to the previous Range. Therefore, when Logindex needs to be accessed, the Range search method shown in Figure 6 can be used to search for all Ranges of this Page within the file by following the forward pointer from the metadata of a Range.

[0124] This embodiment proposes an efficient implementation scheme for Logindex cache merging and generation. Based on PolarDB's shared storage architecture, it implements a Multi-Version Engine on the compute nodes, fully utilizing the resources of the compute nodes to supplement the advantages and capabilities of multi-versioning while retaining the strengths of PolarDB. The Multi-Version Engine supports accessing Pages from any LSN range, from the persistent version in storage to the latest version. This feature significantly supports multi-database node architectures and meets diverse needs for Page versions.

[0125] Referring to Figure 7, Figure 7 shows a flowchart of a log reading method provided according to an embodiment of the present disclosure, specifically including the following steps 702 to 706.

[0126] Step 702: Determine the rewrite log range corresponding to the log information to be read.

[0127] Step 704: Within the scope of the rewrite log, determine the segment information of the index item segment of the log to be read by querying the segment relationship file based on the log information to be read.

[0128] Step 706: Based on the segment information, query the index storage file to obtain the target index information associated with the log information to be read, and read the target log in the log space based on the target index information.

[0129] In practical applications, both read-write nodes and read-only nodes are computing nodes in the database system. Generally, read-only nodes initiate read requests to read the rewrite logs from the index storage files stored in shared storage. The log information to be read is the relevant information of the rewrite logs that the read-only node determines to be read before performing the rewrite log reading. The log information to be read may include the log range of the rewrite logs to be loaded, which contains data page version information, checkpoint information, etc.

[0130] The segment information of the log index segment to be read is the segment identifier and segment position corresponding to the segment information to be read. Once the segment identifier and segment position are determined, the index storage file can be queried based on the segment identifier and segment position to obtain the target index information. Furthermore, the target log can be read based on the target index information, achieving fast target log reading. Read-only nodes can apply the target log to the corresponding data pages, achieving data synchronization with read-write nodes and ensuring data synchronization efficiency and accuracy.

[0131] Corresponding to the above method embodiments, this disclosure also provides a log index generation system embodiment. Figure 8 shows a schematic diagram of the structure of a log index generation system provided in one embodiment of this disclosure. As shown in Figure 8, the log index generation system 800 is applied to the log index engine 812 of a database system 810. The database system includes a storage engine 814 and a storage layer 816. The storage engine determines a log space for storing replay logs and reads a target replay log item from the log space. A log index item is generated based on the log information of the replay log item, and the next replay log item associated with the replay log item is obtained from the log space. The next replay log item is used as the replay log item, and the step of generating a log index item based on the replay log item is executed. This continues until at least two log index items are generated. A log index is generated based on the at least two log index items, and the log index is stored in the storage layer 816 for persistent storage.

[0132] In practical applications, both the storage engine and the log indexing engine are integrated into the compute nodes of the database system. These compute nodes can be read-write nodes (Primary, i.e., master nodes) or read-only nodes (Secondary, i.e., slave nodes). Both master and slave nodes contain the log indexing engine and the storage engine. In the master node, the log indexing engine reads the target replay log from the storage engine and generates the corresponding log index entry, which is temporarily stored in memory. Once the log index entry in memory meets the conditions for persistent storage, it is written to disk and stored in the shared storage (storage layer), thus achieving persistent storage of the log index entry.

[0133] For example, the purpose of the log indexing engine reading the target replay log in the log space is to generate a log index entry for the target replay log, and then persist the log index entry. This allows for subsequent fault recovery of the database system based on the persistently stored log index entry in shared storage, ensuring the operational security of the database system. The persistently stored log index entry in shared storage can also be used to access the replay log. The replay log is read based on the log index entry and applied to the corresponding data page, thereby achieving data persistence after transaction commit, data synchronization of read-only nodes, and on-demand replay of the replay log by read-only nodes. The database system can adopt a PolarDB cluster architecture based on shared storage. The read-write nodes in PolarDB are the master nodes. The replay log can be stored in the storage engine of the master node. The master node also includes a log indexing engine, which is used to read the replay log in the storage engine and generate the corresponding log index entry for the replay log.

[0134] The log indexing engine reads the target replay log from the storage engine and generates a target log index corresponding to the target replay log, temporarily storing the corresponding target log index in memory. Then, it reads the next replay log from the storage engine, continues to generate the next log index entry for the next replay log, and temporarily stores it in memory, and so on, until the log index entries accumulated in memory exceed the size of a rewrite log file, or exceed a preset threshold, or when a log file switch is sent, at least two log index entries accumulated in memory are uniformly persisted. That is, after the log index is generated, it is persistently stored in the storage layer.

[0135] In the log space, following the storage order of replay logs, each replay log entry is read, a corresponding log index entry is generated, and stored in memory. This read-generate operation is repeated until at least two log entry index entries are generated. Based on these at least two index entries, a log index is generated and persistently stored. This achieves efficient caching, merging, persistence, and normalization of replay log entries in the log space. Subsequently, the replay log entries stored in the log space can be quickly accessed based on the generated log index.

[0136] The above is an illustrative scheme of a log index generation system according to this embodiment. It should be noted that the technical solution of this log index generation system and the technical solution of the log index generation method described above belong to the same concept. For details not described in detail in the technical solution of the log index generation system, please refer to the description of the technical solution of the log index generation method described above.

[0137] Corresponding to the above method embodiments, this disclosure also provides a database system embodiment. Figure 9 shows a schematic diagram of the structure of a database system provided in one embodiment of this disclosure. As shown in Figure 9, the database system 900 includes a storage engine 910, a log index engine 920, and a storage layer 930. The storage engine 910 is used to generate and store multiple replay log entries according to the operation order of data pages. The log index engine 920 is used to read multiple replay log entries from the storage engine 910, generate a log index according to the log generation method, and store the log index in the storage layer 930. The storage layer 930 is used to persistently store the log index generated by the log index engine 920.

[0138] In practical applications, both the storage engine and the log indexing engine are integrated into the compute nodes of the database system. These compute nodes can be read-write nodes (Primary, i.e., master nodes) or read-only nodes (Secondary, i.e., slave nodes). Both master and slave nodes contain the log indexing engine and the storage engine. In the master node, the log indexing engine reads the target replay log from the storage engine and generates the corresponding log index entry, which is temporarily stored in memory. Once the log index entry in memory meets the conditions for persistent storage, it is written to disk and stored in the shared storage (storage layer), thus achieving persistent storage of the log index entry.

[0139] A Multi Version Engine module (log indexing engine) is introduced at the database storage engine layer. This module's integration sits between the storage engine and the storage layer within the database compute node, maintaining the high performance of a lightweight distributed storage layer. This architecture enables compute nodes to recognize and utilize multiple versions of page files in shared storage. The Multi Version Engine supports accessing data pages across any LSN range, from the persistent version on storage to the latest version. This feature significantly supports multi-database node architectures, meeting diverse needs for data page versions.

[0140] Corresponding to the above method embodiments, this disclosure also provides an embodiment of a log index generation device. Figure 10 shows a schematic diagram of the structure of a log index generation device provided in one embodiment of this disclosure. As shown in Figure 10, the device includes a reading module 1002, a generation module 1004, an execution module 1006, and a storage module 1008.

[0141] The read module 1002 is configured to read the target replay log item from the log space where the replay log is stored.

[0142] The generation module 1004 is configured to generate a log index item based on the log information of the replay log item, and to obtain the next replay log item associated with the replay log item in the log space.

[0143] The execution module 1006 is configured to use the next replay log entry as the replay log entry and execute the step of generating a log index entry based on the replay log entry.

[0144] Storage module 1008 is configured to generate a log index based on at least two log index entries and persistently store the log index until at least two log index entries are generated.

[0145] In an optional embodiment, the reading module 1002 is further configured to: determine the log space corresponding to the database system, and determine in the log space replay log entries generated by operations on data pages in the database system, wherein the replay log entries are associated with multiple versions of data pages; and determine the target replay log entry from the replay log entries stored in the log space using an asynchronous thread.

[0146] In an optional embodiment, the reading module 1002 is further configured to: determine a target replay log file in the log space, and divide the replay log entries in the replay log text into at least one log entry interval with an associated arrangement order according to the data page identifier corresponding to the replay log entries; determine the target replay log entry in chronological order among the replay log entries contained in the at least one log entry interval.

[0147] In an optional embodiment, the generation module 1004 is further configured to: determine the interval log index item corresponding to each log item interval from the at least two log index items; obtain an index item interval chain associated with the interval log index item corresponding to each log item interval by performing interval connection on each log item interval according to the arrangement order; and generate the log index based on the at least two log indexes contained in the index item interval chain if the index item interval chain satisfies the storage conditions.

[0148] In an optional embodiment, the storage module 1008 is further configured to: determine at least two log index entry segments associated with the log index, and persistently store the log index according to the segment order of the at least two log index entry segments.

[0149] In an optional embodiment, the storage module 1008 is further configured to: determine an index storage file corresponding to the log space; determine a log index item segment to be stored among the at least two log index item segments, and, if a segment association relationship is established between the log index item segment to be stored and the preceding log index item segment, store the segment association relationship in a segment relationship file and store the log index item segment to be stored in the index storage file; determine a next log index item segment among the at least two log index item segments, and use the next log index item segment as the log index item segment to be stored, and perform the step of storing the log index item segment to be stored in the index storage file if a segment association relationship is established between the log index item segment to be stored and the preceding log index item segment; until the at least two log index item segments are stored in the index storage file, persist the index storage file and the segment relationship file.

[0150] In an optional embodiment, the storage module 1008 is further configured to: determine the rewrite log range based on the log information to be read when a read-only node associated with a read-write node performs rewrite log reading on the index storage file; within the rewrite log range, determine the segment information of the log index item segment to be read by querying the segment relationship file based on the log information to be read; query the index storage file based on the segment information to obtain target index information associated with the log information to be read, and read the target log in the log space based on the target index information.

[0151] In an optional embodiment, the storage module 1008 is further configured to: determine a target data page associated with the log information to be read, and process the target data page based on the target log.

[0152] One embodiment of this disclosure provides a log index generation apparatus that reads a target replay log item from a log space storing replay logs; generates a log index item based on the log information of the replay log item; retrieves the next replay log item associated with the replay log item from the log space; uses the next replay log item as the replay log item; and generates a log index item based on the replay log item, until at least two log index items are generated. In other words, for the log space storing replay log items, each replay log item is read one by one, and a corresponding log index item is generated. Following the storage order of the replay logs in the log space, one replay log item is read, a corresponding log index item is generated, and stored in memory. This read-generate operation is repeated until at least two log index items are generated. Then, a log index is generated based on these at least two log index items, and the log index is persistently stored. This achieves efficient caching, merging, persistence, and normalized storage of replay log items in the log space. Subsequently, the replay log items stored in the log space can be quickly accessed based on the generated log index.

[0153] The above is an illustrative scheme of a log index generation device according to this embodiment. It should be noted that the technical solution of this log index generation device and the technical solution of the log index generation method described above belong to the same concept. For details not described in detail in the technical solution of the log index generation device, please refer to the description of the technical solution of the log index generation method described above.

[0154] Corresponding to the above method embodiments, this disclosure also provides a log reading device embodiment. Figure 11 shows a schematic diagram of the structure of a log reading device provided in one embodiment of this disclosure. As shown in Figure 11, the device includes a determining module 1102, a querying module 1104, and a reading module 1106.

[0155] The determination module 1102 is configured to determine the rewrite log range corresponding to the log information to be read.

[0156] The query module 1104 is configured to determine the segment information of the index item segment of the log to be read by querying the segment relationship file based on the log information to be read within the scope of the rewrite log.

[0157] The reading module 1106 is configured to query the index storage file based on the segment information, obtain the target index information associated with the log information to be read, and read the target log in the log space based on the target index information.

[0158] The log information to be read refers to the relevant information of the rewrite log that the read-only node determines to be read before performing rewrite log reading. This information may include the log range of the rewrite log to be loaded, containing data page version information, checkpoint information, etc. The segment information of the index item segment to be read consists of the segment identifier and segment position corresponding to the segment information to be read. Once the segment identifier and segment position are determined, the index storage file can be queried based on the segment identifier and segment position to obtain the target index information. Furthermore, the target log can be read based on the target index information, achieving fast target log reading. The read-only node can apply the target log to the corresponding data page, achieving data synchronization with the read-write node and ensuring data synchronization efficiency and accuracy.

[0159] The above is an illustrative scheme of a log reading device according to this embodiment. It should be noted that the technical solution of this log reading device and the technical solution of the log reading method described above belong to the same concept. For details not described in detail in the technical solution of the log reading device, please refer to the description of the technical solution of the log reading method described above.

[0160] Figure 12 shows a structural block diagram of a computing device 1200 according to an embodiment of the present disclosure. The components of the computing device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and a database 1250 is used to store data.

[0161] The computing device 1200 also includes an access device 1240, which enables the computing device 1200 to communicate via one or more networks 1260. Examples of such networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1240 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0162] In one embodiment of this disclosure, the aforementioned components of the computing device 1200, as well as other components not shown in FIG. 12, may also be connected to each other, for example, via a bus. It should be understood that the computing device block diagram shown in FIG. 12 is merely for illustrative purposes and is not intended to limit the scope of this disclosure. Those skilled in the art can add or replace other components as needed.

[0163] The computing device 1200 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1200 can also be a mobile or stationary server.

[0164] The processor 1220 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above method.

[0165] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the above method.

[0166] An embodiment of this disclosure also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described method.

[0167] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the method described above belong to the same concept, and all details not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the method described above.

[0168] An embodiment of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0169] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the computer program product can be referred to the description of the technical solution of the above method.

[0170] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0171] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0172] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this disclosure are not limited to the described order of actions, because according to the embodiments of this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this disclosure.

[0173] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0174] The preferred embodiments disclosed above are merely illustrative of this disclosure. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments of this disclosure. These embodiments are selected and specifically described in this disclosure to better explain the principles and practical applications of the embodiments of this disclosure, thereby enabling those skilled in the art to better understand and utilize this disclosure. This disclosure is limited only by the claims and their full scope and equivalents.

Claims

1. A method for generating a log index, comprising: reading a target replay log item in a log space storing replay logs; generating a log index item based on log information of the replay log item, and obtaining a next replay log item associated with the replay log item in the log space; taking the next replay log item as the replay log item, and performing the step of generating a log index item based on the replay log item; until at least two log index items are generated, generating a log index based on the at least two log index items, and persistently storing the log index. 2.The method of claim 1, wherein the step of reading a target replay log item in a log space storing replay logs comprises: determining the log space corresponding to a database system, and determining replay log items recording operations on data pages in the database system in the log space, the replay log items being associated with multi-version data pages; determining the target replay log item in the replay log items stored in the log space by using an asynchronous thread. 3.The method of claim 2, wherein the step of determining the target replay log item in the replay log items stored in the log space by using an asynchronous thread comprises: determining a target replay log file in the log space, and dividing replay log items in the replay log file into at least one log item interval in an associated arrangement order according to data page identifiers corresponding to the replay log items; determining the target replay log item in the replay log items included in the at least one log item interval in a time order. 4.The method of claim 3, wherein the step of generating a log index based on the at least two log index items comprises: determining an interval log index item corresponding to each log item interval in the at least two log index items; obtaining an index item interval chain associated with the interval log index item corresponding to each log item interval by interval connecting each log item interval in the arrangement order; generating the log index based on the at least two log index items included in the index item interval chain in a case where the index item interval chain meets a storage condition. 5.The method of claim 1, wherein the step of persistently storing the log index comprises: determining at least two log index item sections associated with the log index, and persistently storing the log index according to a section order of the at least two log index item sections. 6.The method of claim 5, wherein the step of persistently storing the log index according to a section order of the at least two log index item section comprises: determining an index storage file corresponding to the log space; determining a to-be-stored log index item section in the at least two log index item sections, storing a section association relationship between the to-be-stored log index item section and a previous log index item section to a section relationship file in a case where the section association relationship is established, and storing the to-be-stored log index item section to the index storage file. Determine the next log index entry segment from the at least two log index entry segments, and use the next log index entry segment as the log index entry segment to be stored. Then, perform the step of storing the log index entry segment to be stored into the index storage file, provided that a segment association relationship has been established between the log index entry segment to be stored and the previous log index entry segment. Until the at least two log index entry segments are stored in the index storage file, the index storage file and the segment relationship file are persistently stored.

7. The log index generation method according to claim 6, further comprising, after persistently storing the index storage file: When a read-only node associated with a read-write node reads the rewrite log for the index storage file, the scope of the rewrite log is determined based on the log information to be read. Within the scope of the rewrite log, the segment information of the index item segment of the log to be read is determined by querying the segment relationship file based on the log information to be read; Based on the segment information, the index storage file is queried to obtain the target index information associated with the log information to be read, and the target log is read from the log space based on the target index information.

8. The log index generation method according to claim 7, further comprising, after reading the target log in the log space based on the target index information: Identify the target data page associated with the log information to be read, and process the target data page based on the target log.

9. A log reading method, comprising: Determine the rewrite log range corresponding to the log information to be read; Within the scope of the rewrite log, the segment information of the index item segment of the log to be read is determined by querying the segment relationship file based on the log information to be read; Based on the segment information, query the index storage file to obtain the target index information associated with the log information to be read, and read the target log in the log space based on the target index information.

10. A log index generation system, applied to the log index engine of a database system, the database system comprising a storage engine and a storage layer; The storage engine determines the log space for storing replay logs and reads the target replay log item from the log space; A log index item is generated based on the log information of the replay log item, and the next replay log item associated with the replay log item is obtained in the log space; The next replay log entry is used as the replay log entry, and the step of generating a log index entry based on the replay log entry is executed; The process continues until at least two log index entries are generated. A log index is then generated based on the at least two log index entries and stored in the storage layer for persistent storage.

11. A database system, comprising a storage engine, a log indexing engine, and a storage layer; The storage engine is used to generate and store multiple replay log entries in the order of operations on the data pages; The log index engine is configured to read multiple replay log items in the storage engine, generate a log index according to the log index generation method in any one of claims 1-8, and store the log index in the storage layer. The storage layer is configured to persistently store the log index generated by the log index engine. 12.A log index generation apparatus, comprising: a reading module configured to read a target replay log item in a log space storing replay logs; a generating module configured to generate a log index item based on log information of the replay log item, and acquire a next replay log item associated with the replay log item in the log space; an executing module configured to take the next replay log item as the replay log item, and execute the step of generating a log index item based on the replay log item; a storing module configured to generate at least two log index items until the at least two log index items are generated, generate a log index based on the at least two log index items, and persistently store the log index. 13.A computing device, comprising: a memory and a processor; the memory is configured to store a computer program or instructions, and the processor is configured to execute the computer program or instructions, which, when executed by the processor, implement the steps of the method in any one of claims 1-9. 14.A computer-readable storage medium storing a computer program or instructions, which, when executed by a processor, implement the steps of the method in any one of claims 1-9. 15.A computer program product comprising a computer program or instructions, which, when executed by a processor, implement the steps of the method in any one of claims 1-9.