Metadata querying method, device, system, storage medium, and program product

Abstract

Embodiments of the present disclosure provide a metadata querying method, a device, a system, a storage medium, and a program product. In the embodiments of the present disclosure, on the basis of the frequency at which target data or target metadata is accessed, the number of replicas of at least part of metadata required for accessing the target data is dynamically expanded to form a metadata state in which primary rows and replica rows coexist, and the primary rows and the replica rows tend to be eventually distributed in at least two data partitions of a metadata table, so that a storage system can effectively cope with sudden traffic, thereby ensuring the stability of access of hot spot data, improving the access performance of the storage system, and further helping the storage system meet an SLA. In addition, the at least two data partitions are located on at least two storage nodes, thereby facilitating improving the concurrent access capability of the storage system. When access traffic of replica rows on a certain storage node is high, the access traffic can be shunted to replica rows distributed on other storage nodes, thereby facilitating improving the service performance and reliability of the distributed storage system.

Description

Metadata query methods, devices, systems, storage media, and program products

[0001] This disclosure claims priority to Chinese Patent Application No. 202510061714.1, filed with the China National Intellectual Property Administration on January 14, 2025, entitled “Metadata Query Method, Device, System, Storage Medium and Program Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of cloud computing technology, and in particular to a metadata query method, device, system, storage medium, and program product. Background Technology

[0003] Some storage systems store large amounts of data such as photos, messages, and videos. The metadata of this data, also known as a data fingerprint (FP), is a unique identifier (ID) describing the data. This metadata is stored in a database, which organizes and manages it using tables. Each row in a table records the metadata of a single piece of data.

[0004] In practical applications, in order to improve the access performance of data tables, databases usually divide a data table into multiple data partitions according to the partition key in the data table, and use load balancing technology to schedule these partitions to different storage nodes to provide metadata services to the outside world.

[0005] Among them, the access performance of individual data is limited by the capabilities of the partition where the metadata is located. In particular, for some hot data with high access frequency, its QPS (Queries-Per-Second) may reach tens of thousands or even higher. Metadata will become a bottleneck for the access performance of hot data, which will seriously affect the storage system's achievement of SLA (Service Level Agreement). Summary of the Invention

[0006] This disclosure provides a metadata query method, device, system, storage medium, and program product to improve the access performance of a storage system and help the storage system achieve its Service Level Agreement (SLA).

[0007] This disclosure provides a metadata query method for querying target metadata from a metadata table in a database. The metadata table includes a main row containing the target metadata. The method includes: monitoring the access volume of target data or target metadata, wherein the target data is the data described by the target metadata; when the access volume meets the replica creation conditions, creating at least one replica row of the main row in the metadata table, and querying at least some metadata required to access the target data from the at least one replica row; wherein the replica row is a copy of at least some metadata stored in the main row, and the main row and the at least one replica row are distributed in at least two data partitions of the metadata table.

[0008] This disclosure also provides a control device, including a memory and a processor. The memory is used to store a computer program, and the processor is coupled to the memory to execute the computer program to implement the steps in the various methods provided in this disclosure.

[0009] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the processor to perform the steps in the methods described above.

[0010] This disclosure also provides a computer program product, which includes a computer program / instructions that, when executed by a processor, enable the processor to perform the steps described in the method embodiments above.

[0011] This disclosure also provides a storage system, including: a management device, a database, and at least one storage node; the database includes a metadata table, the metadata table includes a main row containing target metadata, and the at least one storage node is used to store target data corresponding to the target metadata; the management device is used to execute the steps in the metadata query method to query the target metadata from the metadata table.

[0012] In this embodiment, based on the access status of the target data or target metadata, the number of replicas of at least some of the metadata required to access the target data is dynamically expanded, forming a metadata state where primary and replica rows coexist. Ultimately, the primary and replica rows are distributed across at least two data partitions of the metadata table. This enables the storage system to effectively handle sudden traffic surges, ensure the stability of access to hot data, improve the access performance of the storage system, and thus help the storage system achieve its SLA. Furthermore, having at least two data partitions located on at least two storage nodes improves the concurrent access capability of the storage system. When the access volume of a replica row on a certain storage node is high, the access volume can be distributed to replica rows on other storage nodes, which helps improve the service performance and reliability of the distributed storage system. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0014] Figure 1 is a schematic diagram of the structure of the metadata table partition provided in an embodiment of this disclosure;

[0015] Figure 2 is a schematic diagram of the structure of a distributed storage system provided in an embodiment of this disclosure;

[0016] Figure 3 is a schematic diagram of a lifecycle state provided in an embodiment of this disclosure;

[0017] Figure 4 is a schematic diagram of a process for creating a copy row provided in another exemplary embodiment of this disclosure;

[0018] Figure 5 is a schematic diagram of another copy line waste recycling process provided by yet another exemplary embodiment of this disclosure;

[0019] Figure 6 is a flowchart illustrating a metadata query method provided in another exemplary embodiment of this disclosure;

[0020] Figure 7 is a schematic diagram of the structure of a control device provided in another exemplary embodiment of this disclosure. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0022] It should be noted that, in the cases involving user information in the embodiments of this disclosure, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this disclosure (including but not limited to language models or large models) comply with relevant laws and standards.

[0023] In a storage system, various types of data, such as images, videos, and text, can be stored. The metadata of this data, which is a unique identifier describing the data, can be stored in a database. This metadata is organized and managed as data tables, and these tables are called metadata tables. The database engine is a module within the database responsible for processing and managing the metadata stored in the database. It can perform query and update operations on the metadata in the database. The database engine uses partitioning technology to divide the metadata table into multiple data partitions, which can be simply called partitions. These partitions are distributed across different storage nodes according to a certain strategy to provide metadata services to the outside world.

[0024] Figure 1 illustrates the structure of the metadata table partitioning. The metadata table comprises multiple rows and columns, with each row representing a single metadata record. Each metadata record includes a primary key column and several regular columns. Based on the primary key column, the data table is divided into three partitions: Partition A, Partition B, and Partition C. Each partition stores several metadata records. The metadata corresponding to Partition A, Partition B, and Partition C is distributed across different storage nodes, which improves the concurrent access capability of the storage system and thus enhances its overall service capabilities.

[0025] While distributing different partitions across different storage nodes helps reduce the load on individual partitions, for any single piece of metadata in a partition, if access requests are concentrated on a single piece of metadata in a particular partition, and the access volume is large, the access performance is limited by the capacity of the partition where the metadata resides, leading to access errors. This is especially true for some hot data with high access frequency, which may reach tens of thousands or even higher. The limitation of the metadata on the capacity of its respective partition will become a bottleneck for the access performance of hot data, and thus seriously affect the storage system's achievement of SLA.

[0026] For example, in the e-commerce field, the storage system can be implemented as a product information database. The product information database stores a large amount of product information and organizes and manages this product information through the metadata of the product information. The metadata of the product information records the storage address, the most recent access time, the number of accesses, etc. If a large number of users request access to a certain product information in a short period of time, the product information will become hot data. In order to retrieve the product information from the product information database and return it to the user, it is necessary to frequently query the metadata of the product information. The metadata of the product information may become the performance bottleneck of the entire product information database and affect the system performance.

[0027] For example, a storage system can function as a cloud storage and photo album service system. It provides users with cloud storage services and is responsible for storing users' photos, videos, and other data. In addition, it offers users one-stop data storage and analysis capabilities, allowing them to quickly and efficiently build the cloud storage and photo album services they need. Cloud storage and photo album service systems manage a large amount of users' photos, videos, and other data. This data includes a significant amount of frequently accessed data, where the QPS (queries per second) for a single frequently accessed data point can reach tens of thousands or even higher. The metadata of this frequently accessed data can become a bottleneck for access performance.

[0028] To address the aforementioned issues, this disclosure provides a metadata query method. In this embodiment, the data that a user needs to access is referred to as target data, and the metadata of the target data is referred to as target metadata. The target data and its metadata may vary depending on the application scenario and are not limited thereto. Based on the access status of the target data or target metadata, the number of replicas of at least some of the metadata required to access the target data is dynamically expanded, forming a metadata state where primary and replica rows coexist. Ultimately, the primary and replica rows tend to be distributed across at least two data partitions in the metadata table, thereby enabling the storage system to effectively handle sudden traffic surges, ensure the stability of access to hot data, improve the access performance of the storage system, and thus contribute to the storage system's achievement of its Service Level Agreement (SLA). Furthermore, having at least two data partitions located on at least two storage nodes is beneficial for improving the concurrent access capability of the storage system. When the access volume of a replica row on a certain storage node is high, the access volume can be distributed to replica rows distributed on other storage nodes, which is beneficial for improving the service performance and reliability of the distributed storage system.

[0029] Figure 2 is a schematic diagram of a storage system provided in an embodiment of this disclosure. This storage system can be a distributed storage system or a non-distributed storage system, and there is no limitation in this regard. Furthermore, this disclosure does not limit the product implementation form of the storage system 20, for example, object storage systems, block storage systems, cloud storage systems, cloud storage and photo album service systems, etc. Any storage system that uses a database for storing and managing metadata is applicable to this disclosure embodiment, wherein the database used to store metadata can be implemented as various relational or non-relational database products. As shown in Figure 2, the storage system 20 includes: a control node 22, a database 23, and at least one storage node 24. The storage node 24 is responsible for storing various types of data, the control node 22 is the node responsible for scheduling in the storage system, and the database 23 provides metadata services based on a database engine (not shown in the figure). The control node 22 receives user access requests to the storage node 24 provided by the client 21. In this embodiment, from the perspective of different access objects, access requests are divided into data access requests and data metadata access requests. Data access requests also request access to their metadata. In one example, the control node 22 queries the metadata of the data based on the access request, such as querying the metadata table of database 23 from the database engine to obtain the identification information of the corresponding storage node 24. Here, the access request can be regarded as an access request for the data's metadata. Then, the corresponding storage node 24 can be scheduled to provide data storage and read / write services to the user. Here, the access request can be regarded as an access request for the data itself, and the corresponding access result is returned upon completion of the access. The access request can be a read request or a write request; correspondingly, the access result can be a read result or a write result. A write result can be a successful write or a failed write; a read result can be data that was read successfully or failed to read.

[0030] In this embodiment, during access to target data or target metadata on storage node 24, the access volume of the target data or target metadata can be dynamically monitored. Access volume can be monitored by the number of access requests, reflecting the query traffic handled by the main bank; therefore, access volume can also be referred to as access load. The target data can be any data stored on storage node 24, and the target metadata refers to data used to describe the target data. For example, when monitoring the access volume of data on storage node 24, data with access popularity meeting the requirements can be identified as target data. For example, data with high access popularity can be used as target data. Besides popular data, target data can also be data of a specific type, data with a specific identifier, or data of a specific user, etc., depending on the specific requirements. Correspondingly, the target metadata can be metadata of any data stored on storage node 24. For example, when monitoring the access volume of metadata on storage node 24, metadata with access popularity meeting the requirements can be identified as target metadata. For example, metadata with high access popularity can be used as target data. Besides popular metadata, target metadata can also be metadata of a specific type, or metadata with a specific identifier, depending on the specific requirements. Since the access volume of target data or target metadata is similar or the same, when monitoring access volume, the access object monitored can be target data or target metadata, without any limitation.

[0031] In this embodiment, database 23 organizes and manages metadata in the form of data tables, which are referred to as metadata tables. The database engine of database 23, based on partitioning technology, divides the metadata tables in database 23 into multiple partitions. These partitions are distributed and stored on different storage devices to provide metadata services. Specifically, the database engine can divide the metadata tables into multiple partitions according to primary key partitioning rules and schedule these partitions to different storage devices for storage. Regarding primary key partitioning rules, this embodiment does not limit the specific rules. For example, it can be based on the range of primary key values, grouping data rows with primary key values ​​within a certain range into the same partition; or it can be based on primary key values ​​that meet a specific pattern (e.g., odd numbers, or values ​​spaced at intervals of 3) into the same partition, and so on. Furthermore, it should be noted that the storage device where the partitions are located can be storage node 24, or it can be another storage device different from storage node 24; this is not limited. In subsequent embodiments, taking the storage of the metadata table partitions on storage node 24 as an example, a storage node 24 can store at least one data partition from the metadata table. In other words, a storage node 24 stores one or more data partitions. Different data partitions in the metadata table are stored separately on different storage nodes 24. Each partition includes at least one data row, and each data row records the metadata corresponding to a piece of data on the storage node 24.

[0032] In this embodiment, as shown in ① of Figure 2, the control node 22 is responsible for receiving access requests from the client 21. Further, as shown in ② and ③ of Figure 2, when the control node 22 receives an access request for target data of the storage node 24, it can query the target metadata corresponding to the target data from the metadata table of the database 23 according to the access request. The target metadata describes the target data and is its unique identifier. The target metadata is located in the target partition of the metadata table, and the target partition includes the data row containing the target metadata. For ease of description, in this embodiment, based on the order in which data rows for the target metadata are created in the metadata table, the first data row created for the target metadata in the metadata table is called the primary row, and the data rows created after the primary row and dependent on the primary row are called replica rows. The primary row stores at least the target metadata and supports read and write operations on the target metadata stored therein, for example, read and write operations on the target metadata on the primary row. Taking a read operation on the primary row as an example, as shown in ④ of Figure 2, the target storage node storing the target data can be determined based on the target metadata. As shown in ⑤ of Figure 2, the access request is forwarded to the target storage node. As shown in Figure 2, ⑥, the target storage node performs an access operation on the target data based on the access request and returns the access result to the control node 22. As shown in Figure 2, ⑦, the control node 22 returns the access result to the client 21 upon receiving it.

[0033] To address the single point of failure issue in target metadata, during dynamic access to target data or target metadata, the access volume of the target data or target metadata is monitored. The access volume of the target data reflects the access volume to the primary row containing the target metadata, and this access volume also reflects the query load faced by the primary row. This allows for assessment of whether the access volume of the primary row has exceeded its capacity, and further determines whether to add replica rows to help distribute the access load. A replica row is a copy of at least a portion of the metadata stored in the primary row containing the target metadata, responsible for sharing the primary row's access load, such as handling read operations on the target metadata. In the metadata table shown in Figure 2, primary row a of partition 1 is used as an example of the primary row containing the target metadata. In actual applications, the primary row may not necessarily be located in partition 1, but could be located in any partition. In this embodiment, replica creation conditions are pre-set. By determining whether the access volume of the target data meets the replica creation conditions, it is determined whether a replica row needs to be created for the primary row. If the access volume of the target data meets the replica creation conditions, at least one replica row of the primary row is created in the metadata table. As shown in Figure 2, at least one replica row of the primary row a is created in the metadata table. This at least one replica row includes replica row a1 and replica row a2. It should be understood that the number of replica rows and the partition where each replica row is located are merely examples and do not constitute a limitation on this embodiment. Furthermore, this embodiment does not limit the conditions for replica creation. For example, the replica creation condition could be that the access volume of the target data reaches a preset threshold. Another example is that the replica creation condition could be a surge in access volume of the target data during a certain period. Additionally, it should be noted that the replica row creation process is dynamic. Each time it is determined that the access volume of the target data meets the replica creation condition, one replica row can be created, or multiple replica rows can be created; there is no limitation on this.

[0034] Each replica row is a copy of at least a portion of the metadata stored in the main row containing the target metadata. That is, a replica row can be a copy of all metadata stored in the main row containing the target metadata, or it can be a copy of only a portion of the metadata stored in the main row containing the target metadata. This depends on the query requirements of the upper-layer application (different application scenarios have different query requirements), and there are no restrictions on this. The main row includes multiple data columns. From the perspective of data columns, a replica row can include some columns from the main row, or it can include all columns from the main row; some columns correspond to some metadata, and all columns correspond to all metadata.

[0035] It's important to note that regardless of whether the copy row is a copy of all or part of the metadata in the main row, the metadata required in the copy row may differ for different query needs or application scenarios. However, the copy row for the same query need or application scenario will contain the same metadata. For example, if the target data is a song file, the metadata in the main row includes, but is not limited to: song name, artist name, album name, year, track number, and duration. In one application scenario, if the query requires the song name and artist name, the copy row can contain the song name, artist name, and album name. In another application scenario, if the query requires the year of the song, the copy row can include the song name and year. For example, if the target data is a photo, the metadata in the main row includes, but is not limited to: photo identifier, photo type, shooting time, shooting location, shooting angle, people in the photo, number of visits, number of downloads, and last access time. In one application scenario, if you need to query the people in the photo, the copy row can include the photo identifier and the people in the photo. In another application scenario, if you need to query the shooting information of the photo, the copy row can include the photo identifier, photo type, shooting time, shooting location, shooting angle, and the people in the photo. In yet another application scenario, if you need to query the access information of the photo, the copy row can include the photo identifier, number of visits, number of downloads, and last access time. In yet another application scenario, if you need to query all kinds of information about the photo, the copy row can include all the metadata in the main row, that is, the photo identifier, photo type, shooting time, shooting location, shooting angle, people in the photo, number of visits, number of downloads, and last access time.

[0036] When a replica row is created, it is created in a data partition of the metadata table. Whether the replica row resides in the same data partition as the primary row or any existing replica row is not limited and can be determined by the database engine. Furthermore, during database operation, the database engine can dynamically merge data partitions based on factors such as access load or partition size. This can also lead to situations where the primary row and replica rows are in the same data partition, or multiple replica rows are in the same data partition. For example, if two replica rows A and B are located in different data partitions, when either or both partitions are small, they may be merged, resulting in replica rows A and B being in the same data partition. If the primary row and replica rows are in the same data partition, or multiple replica rows are in the same data partition, the database engine can trigger a partition split operation automatically or based on application requirements to distribute the primary row and at least one created replica row across different data partitions. For example, the primary row and at least one replica row should be distributed across at least two data partitions of the metadata table, thus resolving the bottleneck problem of single-row metadata. As mentioned above, the distribution of primary and replica rows is dynamic, ultimately tending to distribute the primary row and at least one replica row across at least two data partitions of the metadata table. If there is only one replica row, the ultimate tendency is to distribute the primary row and replica row across two different data partitions; if there are multiple replica rows, the ultimate tendency is to distribute the primary row and multiple replica rows across at least two data partitions. For example, the primary row and multiple replica rows may be located in different data partitions; the primary row and some replica rows may be located in the same data partition, while other replica rows may be located in one or more data partitions; or the primary row may be located in one data partition, and multiple replica rows may be located in one or more data partitions. As shown in the metadata table in Figure 2, primary row a and replica rows a1 and a2 are located in different data partitions of the metadata table; and replica rows a1 and a2 are distributed in different data partitions, with replica row a1 located in partition 2 and replica row a2 located in partition 3.

[0037] Once replica rows are created, at least a portion of the metadata required to access the target data can be queried from at least one replica row. Since at least one replica row is distributed across at least two data partitions of the metadata table, and these partitions reside on at least two storage nodes, this improves the concurrent access capability of the storage system. Even if one storage node fails, the replica rows distributed across other storage nodes can still be accessed normally, thus enhancing the service performance and reliability of the distributed storage system and ultimately improving its overall service capability.

[0038] In this embodiment, based on the access status of the target data or target metadata, the number of replicas of at least some of the metadata required to access the target data is dynamically expanded, forming a metadata state where primary and replica rows coexist. Ultimately, the primary and replica rows are distributed across at least two data partitions of the metadata table. This enables the storage system to effectively handle sudden traffic surges, ensure the stability of access to hot data, improve the access performance of the storage system, and thus help the storage system achieve its SLA. Furthermore, having at least two data partitions located on at least two storage nodes improves the concurrent access capability of the storage system. When the access volume of a replica row on a certain storage node is high, the access volume can be distributed to replica rows on other storage nodes, which helps improve the service performance and reliability of the distributed storage system.

[0039] In an optional embodiment, lifecycle management is performed on replica rows to maintain consistency. As shown in Figure 3, the lifecycle states of any replica row include initial state, normal state, and recycling state. The initial state refers to the state when the replica row is first written to the data partition. A replica row in the initial state does not store any valid metadata, meaning it does not yet have metadata query capabilities. When a replica row in the initial state is associated with the primary row containing the target metadata, it is updated from the initial state to the normal state. The normal state means that the replica row in the initial state is associated with the primary row containing the target metadata and has metadata query capabilities, enabling it to provide metadata services for the target data. If a replica row is determined to be a replica row to be recycled, or if the primary row associated with the replica row has completed or is currently performing garbage collection, it means that the replica row cannot provide or cannot correctly provide metadata query capabilities. In this case, the replica row can be marked as a recycling state. A replica row to be recycled refers to a replica row that needs to undergo garbage collection. A replica row marked as recycling does not have metadata query capabilities.

[0040] As can be seen from the above, the entire lifecycle of a replica row may involve creation, updating, and recycling, with each stage corresponding to a different lifecycle state. The creation, updating, and recycling processes of replica rows will be described in detail in subsequent embodiments.

[0041] Figure 4 is a schematic diagram of a process for creating a replica row according to another exemplary embodiment of this disclosure. In this embodiment, the process of creating the replica row is described in conjunction with the metadata table in Figure 2. In Figure 2, the main row 'a' serves as the main row containing the target metadata, which refers to the metadata used to describe the target data. Furthermore, the first to third columns of the metadata table in Figure 2 are used as primary key columns, but the method is not limited to this.

[0042] In an optional embodiment, when the access volume of the target data or target metadata meets the replica creation condition, at least one replica row is created in the metadata table. During this process, in response to access requests for the target data or target metadata, as shown in S4031 of Figure 4, if there is no replica row of the primary bank, the access volume of the primary bank is queried. The access volume of the primary bank refers to the query traffic handled by the primary bank. If the access volume of the primary bank meets the first load condition, it means that the primary bank has reached or is about to reach the performance bottleneck. Then, a replica row of the primary bank is created in the metadata table, and the replica row shares the query traffic to solve the performance bottleneck problem faced by the primary bank. If the primary bank already has a corresponding replica row, the access volume of the primary bank can be handled by both the primary bank and the replica row. Then, the access volume of the existing replica row is further queried. If the access volume of the existing replica row meets the second load condition, it means that both the primary bank and the existing replica row have reached or are about to reach the performance bottleneck. Then, replica rows are created in the metadata table, and more replica rows share the query traffic to solve the performance bottleneck problem faced by the primary bank and the currently existing replica rows. The access volume of the primary bank can be determined based on the access volume of the target data or target metadata. Taking the target data as an example, the higher the access volume of the target data, the higher the access volume of the corresponding primary bank; conversely, the lower the access volume of the target data, the lower the access volume of the corresponding primary bank. Furthermore, the first load condition and the second load condition can be the same or different, without limitation. The first load condition can be that the access volume of the primary bank exceeds a set first load threshold, and the second load condition can be that the access volume of the replica bank exceeds a set second load threshold, but is not limited to these.

[0043] In one optional embodiment, during the creation of replica rows in the metadata table, a replica row primary key is generated based on the database's primary key partitioning rules and the primary key of the primary row containing the target metadata. The replica row primary key indicates the creation of a replica row within a data partition. Whether the primary row or existing replica rows are stored in the data partition is not limited and can be determined by the database engine based on load. Considering that primary keys do not possess clustering properties, different replica rows and the primary row are generally distributed across different data partitions. Due to the merging and splitting of data partitions, it is also possible for replica rows and the primary row, or multiple replica rows, to reside in the same data partition. For detailed explanation, please refer to the aforementioned embodiments. The primary key partitioning rule refers to defining a column of the primary key as the sharding key. When partitioning, partitions can be created according to the sharding key; data with the same sharding key is partitioned into the same partition. Of course, data within the same partition can have different sharding keys. When generating the replica row primary key based on the database's primary key partitioning rules and the primary key of the primary row, it is ensured that the replica row primary key is different from the primary key of the primary row, and that the replica row primary key is not the same as the primary keys of other replica rows of the same primary row. Optionally, the generation of the replica row primary key is also related to the application requirements, application scenarios, and application types at the upper layer. Different scenarios, requirements, and types will generate different replica row primary keys.

[0044] Furthermore, in this optional embodiment, based on the generated replica row primary key, a row is written from the corresponding data partition of the metadata table as a replica row, and this replica row is set to its initial state. The replica row in the initial state is an empty row, that is, it is not filled with any valid data. Here, the data partition to be written refers to the data partition where neither the primary row nor the replica row containing the target metadata exists. There may be multiple such data partitions, and the specific data partition in which the replica row is written can be determined by the database engine according to the partitioning rules, and there is no limitation on this.

[0045] Optionally, when writing a row as a replica row from the corresponding data partition of the metadata table based on the replica row's primary key and setting the replica row to its initial state, the process includes: sending the replica row's primary key along with the creation instruction to the database engine, so that the database engine can write a row as a replica row from the corresponding data partition of the metadata table based on the replica row's primary key and set the replica row's state to its initial state. For the database engine, upon receiving the creation instruction, it determines the target data partition according to the primary key partitioning rules; and writes an empty row (a data row without valid data) into the target data partition; this empty row is then used as the replica row. Furthermore, the creation instruction also includes an indication to set the replica row's state to its initial state and a value for the initial state. Based on this indication, a column in the replica row is used as a status column, and this status column is set to the initial state value, thus achieving the purpose of setting the replica row's state to its initial state. Optionally, as shown in S4032 of Figure 4, after completing the operation of writing the replica row, it can be determined whether the replica row was written successfully. If the replica row was written successfully, subsequent operations can continue.

[0046] Optionally, as shown in S401-S402 of Figure 4, before writing the replica row, it is possible to first query whether a replica row corresponding to the primary key of the replica row already exists in the metadata table. Optionally, when querying the replica row, the control node sends a query instruction containing the identifier of the replica row to the database engine. The replica row identifier is a unique identifier for the replica row stored in the primary row, and it corresponds to the primary key of the replica row. The primary key of the corresponding replica row can be uniquely determined based on the replica row identifier. Upon receiving the query instruction, the database engine determines the corresponding primary key of the replica row based on the replica row identifier and queries the corresponding data partition to see if a replica row corresponding to that primary key exists. If it is determined that a replica row corresponding to the primary key of the replica row does not yet exist in the metadata table, S4031, i.e., the write operation of the replica row, is executed. If the replica row is successfully written, as shown in S404 of Figure 4, the identifier of the replica row is updated in the auxiliary column of the primary row to associate the replica row with the primary row.

[0047] As shown in S4033 of Figure 4, if it is determined that a replica row corresponding to the replica row primary key already exists in the metadata table, then the system queries whether the replica row is in a normal state. Optionally, querying whether the replica row is in a normal state includes: sending a first query instruction to the database engine, the first query instruction including the identifier of the replica row to be queried; upon receiving the first query instruction, the database engine, based on the correspondence between the replica row identifier and the replica row primary key, converts the replica row identifier in the first query instruction into the corresponding replica row primary key, and queries the status of the replica row in the corresponding data partition based on the replica row primary key.

[0048] If the query result is yes, meaning that the existing replica row is found to be in a normal state, it means that the replica row already exists and has metadata query capabilities, so there is no need to create it again. In this case, a second query instruction carrying the replica row identifier can be sent directly to the database engine, so that the database engine can determine the corresponding replica row primary key based on the replica row identifier in the second query instruction, and query the metadata of the replica row in the corresponding data partition of the metadata table based on the replica row primary key.

[0049] If the query result is negative, meaning the existing replica row is found to be in a state other than normal (e.g., initial or recycled), as shown in S404 of Figure 4, the replica row's identifier is updated in the auxiliary column of the primary row to associate the replica row with the primary row. As shown in Figure 2, the primary row in the metadata table includes auxiliary columns. Figure 2 uses the last two columns of the primary row as an example, but it is not limited to this. The auxiliary columns of the primary row record the identifiers of the replica rows associated with that primary row. The identifier of a replica row corresponds to its primary key. The replica row identifier can be the primary key of the replica row, or it can be information mapped to the primary key through calculation; there is no limitation on this.

[0050] As shown in Figure 4, after S4032 or S4033, S404 is executed to update the identifier of the replica row to the auxiliary column of the primary row. That is, an update operation needs to be performed on the primary row containing the target metadata. Various update operations involved in this embodiment can employ conditional updates (CAS, Compare-And-Set). Conditional updates refer to a mechanism that supports concurrent access in the database, used to ensure the integrity and consistency of data in concurrent update operations. That is, the database engine will only execute the corresponding update operation if the update condition check passes; different update operations can have their own corresponding update conditions. Based on this, before performing an update operation on the primary row, the database engine will verify whether the initiator of the operation has permission to update the primary row. Whether the initiator has permission to update can be determined by querying the primary row's sequence number from the database engine when the initiator performs the replica row creation operation through the management node, and using the queried primary row's sequence number as the sequence number currently held by the initiator of the update instruction. When an update command is initiated against a primary row, the command carries the currently held sequence number and the latest sequence number to be updated after the operation is completed. These sequence numbers can increment according to a set step size or be randomly generated; there is no limitation on this. Then, the sequence number currently held by the initiator in the update command is compared with the sequence number of the primary row maintained locally by the database engine for permission verification. If the corresponding update conditions are met, the initiator of the update command is allowed to perform the update operation on the primary row. After the update operation is completed, the latest sequence number to be updated carried by the initiator in the update command is used as the latest sequence number of the primary row maintained locally by the database engine. In other words, the sequence number of the primary row changes dynamically with the update of the primary row, thus clearly identifying the current version of the metadata stored on the primary row and avoiding overwrite conflicts. For example, when there are multiple update operations, permission verification for the update operations is performed using the dynamically updated sequence number of the primary row, thereby avoiding overwrite conflicts caused by concurrent update operations.

[0051] Based on this, during the process of updating the replica row's identifier to the auxiliary column of the primary row containing the target metadata, a first update instruction is sent to the database engine. This allows the database engine to update the replica row's identifier to the auxiliary column of the primary row if the primary row's sequence number meets the first update condition. During this process, the replica row's identifier and the queried primary row's corresponding first sequence number can be sent to the database engine along with the first update instruction. The first sequence number is the primary row's sequence number currently held by the initiator of the first update instruction, used to identify whether the initiator has permission to update the primary row. Upon receiving the first update instruction, the database engine determines whether the primary row's first update condition is met based on the first sequence number in the instruction. Meeting the primary row's first update condition means that the primary row's first sequence number included in the first update instruction is the same as the primary row's second sequence number maintained locally by the database engine. The second sequence number is the latest primary row sequence number maintained locally by the database engine. If the first update condition is met, the replica row's identifier is updated to the primary row's auxiliary column.

[0052] Optionally, after adding the identifier of the replica row to the auxiliary column of the primary row, as shown in S405 of Figure 4, it is determined whether the identifier of the replica row has been added successfully. If the addition is successful, further, as shown in S406 of Figure 4, the replica row is updated from the initial state to the normal state.

[0053] Updating a replica row from its initial state to its normal state requires an update operation. This update can be to the replica row's state and / or stored metadata. Before performing the update operation, the database engine verifies whether the initiator of the update operation has the necessary permissions. This permission can be determined by querying the replica row's sequence number from the database engine upon successful creation. This sequence number is then used as the current sequence number of the replica row held by the initiator of the second update instruction for permission verification. Specifically, when sending the second update instruction to update the replica row, the instruction carries the currently held replica row sequence number and the latest sequence number of the replica row to be updated after the operation. These sequence numbers can increment according to a set step size or be randomly generated; there is no limitation on this. Furthermore, the current sequence number of the replica row held by the initiator is compared with the sequence numbers of the replica rows maintained locally by the database engine for permission verification. If the second update conditions are met, the initiator of the second update instruction is allowed to update the replica row's state. After the operation is completed, the latest sequence number of the replica row to be updated, carried by the initiator in the second update instruction, is used as the latest sequence number of the replica row maintained locally by the database engine. In other words, the sequence number of the replica row changes dynamically as the replica row is updated, thus clearly identifying the current version of the metadata stored on the replica row, ensuring that the current version is the latest version, and maintaining data consistency. Furthermore, it avoids overwrite conflicts. For example, when there are multiple update operations, permission verification for the update operations is performed using the dynamically updated replica row sequence number, thereby avoiding overwrite conflicts caused by concurrent update operations.

[0054] Based on this, the control node sends a second update instruction to the database engine, so that the database engine can update the replica row from its initial state to a normal state if the current state and sequence number of the replica row meet the second update condition. Specifically, the current state and the corresponding third sequence number of the replica row are sent to the database engine along with the second update instruction, so that the database engine can update the replica row from its initial state to a normal state if the current state and third sequence number of the replica row meet the second update condition. At the same time, since the replica row has already been associated with the primary row, it has also stored at least some of the metadata required by the primary row to access the target data, thus completing the replica row creation operation. Here, the third sequence number is the sequence number of the replica row currently held by the initiator of the second update instruction, used to identify whether the initiator has permission to update the replica row. The current state and third sequence number of the replica row meeting the second update condition means that the current state of the replica row belongs to the initial state or the recycling state defined by the second update condition, and the third sequence number is the same as the fourth sequence number in the second update condition. The fourth sequence number is the latest sequence number of the replica row maintained locally by the database engine.

[0055] Optionally, after performing the operation of updating the replica row from its initial state to its normal state, it can be determined whether the state of the replica row has been successfully updated.

[0056] The update operation for replica rows also includes updating the metadata stored in the replica rows. For example, if the metadata of the primary row changes, an update operation can be performed on at least one replica row of the primary row. Optionally, when it is necessary to update the metadata of a replica row, the second update instruction also includes: target update data. This allows the database engine to update the metadata stored in the replica row using the target update data, provided that the current state of the replica row and the third sequence number meet the second update conditions corresponding to the replica row.

[0057] In this embodiment, since the replica row in a normal state is associated with the primary row where the target metadata resides and has metadata query capabilities, at least some of the metadata required to access the target data can be queried from at least one replica row. Optionally, during the query process, the control node responds to the access request for the target data, determines the target replica row based on the access volume of at least one replica row, and then queries at least some of the metadata required to access the target data from the determined target replica row.

[0058] In this embodiment, the method for determining the target replica row is not limited. For example, the replica row with less access volume among at least one replica row can be preferentially selected as the target replica row. Alternatively, based on the region of the storage node where at least one replica row is stored, replica rows on storage nodes closer to the user's geographical location can be preferentially selected, thereby reducing network latency.

[0059] Furthermore, the current status of the target replica row and at least some of its stored metadata are queried. The query method for the replica row's status and metadata can be found in the aforementioned embodiments and will not be repeated here. If the target replica row is found to be in a normal state, then the at least some metadata stored in the target replica row is used as the at least some metadata required to access the target data. If the target replica row is found to be in another state, such as an initial state or a recycled state, then the at least some metadata required to access the target data is queried again from the primary row. Optionally, during the process of re-querying the at least some metadata required to access the target data from the primary row, the at least some metadata required to access the target data can be queried directly from the primary row; alternatively, a new replica row can be determined from the auxiliary columns of the primary row as the target available replica row, and the at least some metadata required to access the target data can be queried from that target available replica row. The method for querying the target data based on the target available replica row can be found in the query method of the aforementioned embodiments and will not be repeated here.

[0060] If the control node finds at least some of the metadata required to access the target data, it determines the target storage node where the target data resides based on that metadata. Then, the control node can provide an access request for the target data to the target storage node, which can then perform the access operation based on the request and return the result to the user.

[0061] The above embodiments describe how to perform creation, update, and query operations on replica rows. This disclosure also provides a garbage collection mechanism for replica rows. Based on this mechanism, storage resources occupied by replica rows can be released when they are no longer needed, thus saving storage resources. The garbage collection process is described below with reference to Figure 5.

[0062] In one optional embodiment, when the access volume of the target data meets the replica reclamation conditions, garbage collection is performed on at least some replica rows corresponding to the main row where the target metadata is located in the metadata table, so as to release the storage resources of the storage node.

[0063] In an optional embodiment, during the garbage collection process of at least a portion of the replica rows in the metadata table, if the access volume meets the replica collection condition, at least a portion of the existing replica rows are determined as replica rows to be collected. This embodiment does not limit the specific implementation of the replica collection condition. For example, it could be when the access volume of the target data is low (e.g., the access volume is less than a set access volume threshold, or the access volume drops in a short period of time). Another example is when the main row containing the target metadata becomes invalid.

[0064] Further, as shown in S501 of Figure 5, the replica rows of the main row in the metadata table are queried. The query method can refer to the above embodiment by sending a query command to the database engine, which will not be repeated here. Further, as shown in S502 of Figure 5, it is determined whether there are any identified replica rows to be recycled in the replica rows of the main row. As shown in S5032, if a replica row to be recycled is found in the metadata table, the current status of the replica row to be recycled is updated to the recycling status. Being marked as recycling status indicates that the replica row to be recycled does not have the ability to query metadata. There are two possible reasons why a replica row to be recycled is marked as recycling status. One is that the replica row to be recycled needs to be garbage collected. If metadata query is needed at this time, it can be queried from the main row. The other is that the main row associated with the replica row has been garbage collected or is being garbage collected. In this case, the recycling status specifically means that the replica row to be recycled is associated with the main row that has been or is being garbage collected, indicating that it needs to be queried from the main row's metadata.

[0065] Optionally, updating the current state of the replica row to be recycled to a recycled state includes: sending a fourth update instruction to the database engine, so that the database engine updates the replica row from its current state to a recycled state if the current state and the replica row's sequence number meet the fourth update condition. The current state can be an initial state or a normal state. Specifically, the current state of the replica row and its corresponding fifth sequence number are included in the fourth update instruction and sent to the database engine, so that the database engine updates the replica row from its current state to a recycled state if the current state and the fifth sequence number meet the fourth update condition. Being in a recycled state indicates that the replica row to be recycled does not have metadata query capabilities. The fifth sequence number is the sequence number of the replica row currently held by the initiator of the fourth update instruction, used to identify whether the initiator has permission to update the replica row. The current state and the fifth sequence number meeting the fourth update condition means that the current state of the replica row belongs to the initial state or normal state defined by the fourth update condition, and the fifth sequence number is the same as the sixth sequence number in the fourth update condition. The sixth sequence number is the latest sequence number of the replica row maintained locally by the database engine.

[0066] Optionally, after updating the copy line to be recycled to the recycled status, S504 in Figure 5 can be executed to determine whether the status of the copy line to be recycled has been successfully updated.

[0067] As shown in S505 of Figure 5, when a replica row to be recycled is marked as recycled, the identifier of the replica row to be recycled is deleted from the auxiliary column of the main row to remove the association between the replica row to be recycled and the main row. In an optional embodiment, the process of removing the association includes: sending a third update instruction to the database engine, so that the database engine deletes the identifier of the replica row to be recycled from the auxiliary column of the main row when the sequence number of the main row and the value of the auxiliary column meet the third update condition, thereby removing the association between the replica row to be recycled and the main row.

[0068] In an optional embodiment, during the process of deleting the identifier of the replica row to be recycled from the auxiliary column of the primary row, the value of the auxiliary column and the seventh sequence number corresponding to the primary row can be sent to the database engine in the third update instruction. The seventh sequence number is the sequence number of the primary row currently held by the initiator of the third update instruction, used to identify whether the initiator has permission to update the primary row. Upon receiving the third update instruction, the database engine determines whether the third update condition corresponding to the primary row is met based on the seventh sequence number in the third update instruction. Meeting the third update condition of the primary row means that: the seventh sequence number of the primary row included in the third update instruction is the same as the eighth sequence number of the primary row maintained locally by the database engine, and the eighth sequence number is the latest sequence number of the primary row maintained locally by the database engine; and the value of the auxiliary column included in the third update instruction is the same as the value of the auxiliary column of the primary row maintained locally by the database engine. If the third update condition is met, the identifier of the replica row to be recycled is deleted from the auxiliary column of the primary row.

[0069] Optionally, as shown in S506 of Figure 5, after deleting the identifier of the copy row to be recycled from the auxiliary column of the main row, it is determined whether the copy row to be recycled has been successfully deleted from the auxiliary column of the main row, that is, whether the copy row to be recycled has been decoupled from the main row.

[0070] Furthermore, as shown in S507 of Figure 5, when the replica row to be reclaimed is decoupled from the primary row, the replica row to be reclaimed can be deleted from the metadata table to complete the garbage collection process for the replica. Here, "deleting the replica row to be reclaimed from the metadata table" can be simply referred to as physical GC (Garbage Collection).

[0071] Optionally, in extreme cases, the replica row creation operation may be retried, resulting in some pending requests. If the replica row is deleted directly, these pending requests may eventually succeed, creating a replica row in a normal state, leading to dangling references. One solution is to create a replica row to be reclaimed and mark it as reclaimable, as shown in S5031, to address the dangling reference problem. Another solution is to delay the physical garbage collection (GC) process while a replica row to be reclaimed exists, which can mitigate the issue of recreating a replica row immediately after deletion.

[0072] In one optional embodiment, when deleting a replica row to be reclaimed from the metadata table, the process includes: if the replica row is marked as reclaimable, performing physical garbage collection (GC) on the replica row to be reclaimed based on the database engine's lifecycle management mechanism for data rows. The lifecycle management mechanism calculates the duration and frequency of unaccessed target data rows on storage nodes. If target data has not been accessed for a certain period or the access frequency is 0, the lifecycle of the replica row to be reclaimed is considered to have reached the end, and then physical GC is performed on the replica row marked as reclaimable, i.e., it is deleted from the metadata table.

[0073] In another optional implementation, when deleting a replica row to be recycled, the process includes: sending the identifier of the replica row to be recycled to the database engine in a delete command, so that the database engine can determine the corresponding replica row primary key based on the replica row identifier; and deleting the replica row to be recycled from the corresponding data partition of the metadata table based on the replica row primary key. Further optionally, if it is desired to delay physical garbage collection (GC), a delay time can be set as needed after the replica row to be recycled is marked as recycled. When the delay time is reached, a delete command carrying the replica row identifier is sent to the database engine, so that the database engine can immediately execute the delete operation upon receiving the delete command and delete the replica row to be recycled. The delay time can be set at the hourly, daily, or monthly level, and is not limited thereto.

[0074] Optionally, when querying the metadata table to see if a replica row to be recycled exists, it's possible that although the identifier of the replica row to be recycled is still in the auxiliary column of the main row, the replica row to be recycled has already been deleted by a wild request. In this case, the replica row to be recycled does not exist in the metadata table. As shown in S5031 of Figure 5, if no replica row to be recycled is found in the metadata table, the replica row to be recycled is written into the metadata table according to the primary key of the replica row corresponding to the replica row to be recycled, and the replica row to be recycled is in a recycling status. Further, if the replica row to be recycled is in a recycling status, the identifier of the replica row to be recycled is deleted from the auxiliary column of the main row to remove the association between the replica row to be recycled and the main row. Optionally, it is determined whether the identifier of the replica row to be recycled has been successfully deleted from the auxiliary column of the main row. If the identifier of the replica row to be recycled has been successfully deleted from the auxiliary column of the main row, the replica row to be recycled is deleted from the metadata table to complete the garbage collection processing of the replica row to be recycled.

[0075] In one optional embodiment, garbage collection can be performed on the primary row. During the garbage collection process of the primary row, "new references" may occur. A new reference refers to the primary row updating the identifier of the replica row in the auxiliary column of the primary row during garbage collection to associate the replica row with the primary row. If the primary row is physically garbage collected first, the update of the primary row's sequence number cannot be completed during the new reference process because the primary row has already been garbage collected. If the new reference occurs before the primary row's physical garbage collection, since the primary row's sequence number changes dynamically with the updates to the primary row, the occurrence of a new reference will cause the primary row's sequence number to change, resulting in a mismatch between the current primary row sequence number carried when the delete command is issued and the changed primary row sequence number, thus preventing the completion of the primary row's physical garbage collection.

[0076] Based on this, in response to a garbage collection operation targeting the primary bank, garbage collection is first performed on at least one replica bank of the primary bank. During this process, the identifiers of at least one replica bank currently associated with the primary bank are queried from the primary bank's auxiliary columns. Based on the identifiers of at least one replica bank, garbage collection is performed on at least one replica bank. The implementation method for garbage collection of replica banks can be referred to the above embodiments, and will not be repeated here. After completing the garbage collection of at least one replica bank, garbage collection is performed on the primary bank. Specifically, if there is only one replica bank, garbage collection is performed on that replica bank before garbage collection is performed on the primary bank; if there are multiple replica banks, garbage collection is performed on all multiple replica banks before garbage collection is performed on the primary bank.

[0077] In one optional embodiment, when performing garbage collection on the primary row, the process includes: sending the currently held primary row sequence number along with a delete instruction to the database engine, so that the database engine can delete the primary row from the target data partition of the metadata table based on the primary row sequence number. For the database engine, upon receiving the delete instruction, it performs an authorization check between the currently held sequence number in the delete instruction and the latest sequence number of the primary row maintained locally by the database engine. If the deletion conditions are met, the initiator of the delete instruction can be allowed to perform the delete operation on the primary row. Meeting the deletion conditions means that the current sequence number of the primary row included in the delete instruction is the same as the latest sequence number of the primary row maintained locally by the database engine.

[0078] Figure 6 is a flowchart illustrating a metadata query method provided in another exemplary embodiment of this disclosure. This method is used to query target metadata from a metadata table in a database. The metadata table includes the main row containing the target metadata. As shown in Figure 6, the method includes:

[0079] S601: Monitor the access volume of target data or target metadata. Target data is the data described by the target metadata.

[0080] S602: When the access volume meets the conditions for replica creation, create at least one replica row of the primary row in the metadata table, and query at least some of the metadata required to access the target data from the at least one replica row; wherein, the replica row is a copy of at least some of the metadata stored in the primary row, and the primary row and at least one replica row are distributed in at least two data partitions of the metadata table.

[0081] In an optional embodiment, when the access volume meets the replica creation condition, creating at least one replica row of the primary row in the metadata table includes: responding to an access request for the target data, if the access volume of the primary row meets a first load condition when no replica row exists, creating a replica row in the metadata table; if a replica row already exists, and the access volume of the existing replica row meets a second load condition, continuing to create replica rows in the metadata table.

[0082] In one optional embodiment, creating a replica row in the metadata table includes: generating a replica row primary key based on the primary key partitioning rules of the database and the primary key of the primary row, wherein the replica row primary key indicates the creation of a replica row in a data partition; writing a row as a replica row from the corresponding data partition of the metadata table based on the replica row primary key, and setting the replica row to an initial state; updating the identifier of the replica row to an auxiliary column of the primary row, wherein the auxiliary column is used to record the identifiers of replica rows associated with the primary row, and the identifiers of the replica rows correspond to their primary keys; and updating the replica row from the initial state to a normal state to complete the creation operation of the replica row, wherein the normal state indicates that the replica row is associated with the primary row and has metadata query capabilities.

[0083] In one optional embodiment, writing a row as a replica row from the corresponding data partition of the metadata table based on the replica row primary key and setting the replica row to an initial state includes: sending the replica row primary key in a creation instruction to the database engine, so that the database engine can write a row as a replica row from the corresponding data partition of the metadata table based on the replica row primary key and set the state of the replica row to an initial state.

[0084] In one optional embodiment, updating the identifier of the replica row to the auxiliary column of the primary row includes: sending a first update instruction to the database engine, so that the database engine updates the identifier of the replica row to the auxiliary column of the primary row when the sequence number of the primary row meets the first update condition.

[0085] In one optional embodiment, updating the replica row from an initial state to a normal state includes: sending a second update instruction to the database engine, so that the database engine updates the replica row from an initial state to a normal state if the current state of the replica row and the sequence number of the replica row satisfy a second update condition.

[0086] In an optional embodiment, before writing a row as a replica row from the corresponding data partition of the metadata table according to the replica row primary key, the method further includes: querying whether a replica row corresponding to the replica row primary key already exists in the metadata table, and determining that a replica row corresponding to the replica row primary key does not yet exist in the metadata table.

[0087] In an optional embodiment, the method further includes: if it is determined that a replica row corresponding to the primary key of the replica row already exists in the metadata table, then querying whether the replica row is in a normal state; if the query result is negative, updating the identifier of the replica row to the auxiliary column of the primary row to associate the replica row with the primary row, and updating the status of the replica row to a normal state to complete the recovery operation of the replica row.

[0088] In an optional embodiment, the method further includes: performing garbage collection on at least a portion of the replica rows in the metadata table when the access volume meets the replica reclamation conditions.

[0089] In an optional embodiment, when the access volume meets the replica reclamation condition, garbage collection is performed on at least a portion of the replica rows in the metadata table, including: when the access volume meets the replica reclamation condition, determining at least a portion of the existing replica rows as replica rows to be reclaimed; when the replica rows to be reclaimed exist in the metadata table, updating the status of the replica rows to be reclaimed to a reclamation status, the reclamation status indicating that the replica rows to be reclaimed do not have metadata query capabilities; deleting the identifier of the replica rows to be reclaimed from the auxiliary column of the main row to remove the association between the replica rows to be reclaimed and the main row; and deleting the replica rows to be reclaimed from the metadata table to complete the garbage collection process for the replica rows to be reclaimed.

[0090] In an optional embodiment, the method further includes: if the copy row to be recycled does not exist in the metadata table, writing the copy row to be recycled into the metadata table according to the primary key of the copy row corresponding to the copy row to be recycled, and the copy row to be recycled being recycled in a recycling state; deleting the identifier of the copy row to be recycled from the auxiliary column of the main row to remove the association between the copy row to be recycled and the main row; and deleting the copy row to be recycled from the metadata table to complete the garbage collection process of the copy row to be recycled.

[0091] In one optional embodiment, deleting the identifier of the replica row to be recycled from the auxiliary column of the main row to decouple the replica row to be recycled from the main row includes: sending a third update instruction to the database engine, so that the database engine deletes the identifier of the replica row to be recycled from the auxiliary column of the main row when the sequence number of the main row and the value of the auxiliary column meet the third update condition, thereby decoupled the replica row to be recycled from the main row.

[0092] In one optional embodiment, querying at least some metadata required to access the target data from the at least one replica row includes: responding to an access request for the target data and determining a target replica row based on the access volume of the at least one replica row; querying the current status and stored metadata of the target replica row; if the target replica row is in a normal state, using the stored metadata in the target replica row as the at least some metadata required to access the target data; if the target replica row is in another state, re-querying the at least some metadata required to access the target data from the primary row.

[0093] In an optional embodiment, the method further includes: responding to a garbage collection operation for the primary row by querying the identifier of at least one replica row currently associated with the primary row from an auxiliary column of the primary row; performing garbage collection on the at least one replica row based on the identifier of the at least one replica row; and performing garbage collection on the primary row after completing the garbage collection on the at least one replica row.

[0094] The detailed implementation methods and beneficial effects of each step in this embodiment have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0095] It should be noted that the execution subject of each step in the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 601 to 602 can be device A; or the execution subject of step 601 can be device A, and the execution subject of step 602 can be device B; and so on.

[0096] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 601, 602, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0097] Figure 7 is a schematic diagram of a control device provided in another exemplary embodiment of this disclosure. As shown in Figure 7, the control device includes a memory 74 and a processor 75.

[0098] Memory 74 is used to store computer programs and can be configured to store various other data to support operation on the control device. Examples of this data include instructions for any application or method used to operate on the control device, contact data, phone book data, messages, pictures, videos, etc.

[0099] The memory 74 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0100] Processor 75, coupled to memory 74, is configured to execute a computer program in memory 74 for: monitoring access volume to target data or target metadata, the target data being data described by the target metadata; and, if the access volume satisfies a replica creation condition, creating at least one replica row of the primary row in the metadata table, and querying at least some metadata required to access the target data from the at least one replica row; wherein the replica row is a copy of at least some metadata stored in the primary row, and the primary row and the at least one replica row are distributed across at least two data partitions of the metadata table.

[0101] In an optional embodiment, when the access volume meets the replica creation condition, the processor 75 creates at least one replica row of the primary row in the metadata table, specifically: in response to an access request for the target data, if no replica row exists, and the access volume of the primary row meets a first load condition, a replica row is created in the metadata table; if a replica row already exists, and the access volume of the existing replica row meets a second load condition, a replica row is created in the metadata table.

[0102] In an optional embodiment, when the processor 75 creates a replica row in the metadata table, it specifically performs the following steps: First, it generates a replica row primary key based on the primary key partitioning rules of the database and the primary key of the primary row, wherein the replica row primary key indicates the creation of a replica row in a data partition. Second, it writes a row from the corresponding data partition of the metadata table as a replica row based on the replica row primary key, and sets the replica row to an initial state. Third, it updates the identifier of the replica row to an auxiliary column of the primary row, wherein the auxiliary column records the identifiers of replica rows associated with the primary row, and the identifiers of the replica rows correspond to their primary keys. Fourth, it updates the replica row from the initial state to a normal state to complete the creation operation of the replica row, wherein the normal state indicates that the replica row is associated with the primary row and has metadata query capabilities.

[0103] In an optional embodiment, when the processor 75 writes a row as a replica row from the corresponding data partition of the metadata table based on the replica row primary key and sets the replica row to an initial state, it is specifically configured to: send the replica row primary key in a creation instruction to the database engine, so that the database engine can write a row as a replica row from the corresponding data partition of the metadata table based on the replica row primary key and set the state of the replica row to an initial state.

[0104] In an optional embodiment, when the processor 75 updates the identifier of the replica row to the auxiliary column of the primary row, it is specifically configured to: send a first update instruction to the database engine, so that the database engine updates the identifier of the replica row to the auxiliary column of the primary row when the sequence number of the primary row meets the first update condition.

[0105] In an optional embodiment, when the processor 75 updates the replica row from an initial state to a normal state, it is specifically configured to: send a second update instruction to the database engine, so that the database engine updates the replica row from an initial state to a normal state if the current state of the replica row and the sequence number of the replica row satisfy the second update condition.

[0106] In an optional embodiment, before writing a row as a replica row from the corresponding data partition of the metadata table according to the replica row primary key, the processor 75 is further configured to: query whether a replica row corresponding to the replica row primary key already exists in the metadata table, and determine that a replica row corresponding to the replica row primary key does not yet exist in the metadata table.

[0107] In an optional embodiment, the processor 75 is further configured to: if it is determined that a replica row corresponding to the primary key of the replica row already exists in the metadata table, query whether the replica row is in a normal state; if the query result is negative, update the identifier of the replica row to the auxiliary column of the primary row to associate the replica row with the primary row, and update the status of the replica row to a normal state to complete the recovery operation of the replica row.

[0108] In an alternative embodiment, the processor 75 is further configured to: perform garbage collection on at least a portion of the replica rows in the metadata table if the access volume satisfies the replica reclamation condition.

[0109] In an optional embodiment, when the processor 75 performs garbage collection on at least a portion of the replica rows in the metadata table when the access volume meets the replica collection conditions, it specifically performs the following steps: when the access volume meets the replica collection conditions, it determines at least a portion of the existing replica rows as replica rows to be collected; when the replica rows to be collected exist in the metadata table, it updates the status of the replica rows to be collected to a collection status, the collection status indicating that the replica rows to be collected do not have metadata query capabilities; it deletes the identifier of the replica rows to be collected from the auxiliary column of the main row to remove the association between the replica rows to be collected and the main row; and it deletes the replica rows to be collected from the metadata table to complete the garbage collection process for the replica rows to be collected.

[0110] In an optional embodiment, the processor 75 is further configured to: if the replica row to be recycled does not exist in the metadata table, write the replica row to be recycled into the metadata table according to the replica row primary key corresponding to the replica row to be recycled, and the replica row to be recycled is in a recycling status; delete the identifier of the replica row to be recycled from the auxiliary column of the main row to remove the association between the replica row to be recycled and the main row; and delete the replica row to be recycled from the metadata table to complete the garbage collection process of the replica row to be recycled.

[0111] In an optional embodiment, when the processor 75 deletes the identifier of the replica row to be recycled from the auxiliary column of the main row to remove the association between the replica row to be recycled and the main row, it is specifically configured to: send a third update instruction to the database engine, so that the database engine deletes the identifier of the replica row to be recycled from the auxiliary column of the main row to remove the association between the replica row to be recycled and the main row when the sequence number of the main row and the value of the auxiliary column meet the third update condition.

[0112] In an optional embodiment, when the processor 75 queries at least some of the metadata required to access the target data from the at least one replica row, it is specifically configured to: respond to the access request for the target data and determine the target replica row based on the access volume of the at least one replica row; query the current status of the target replica row and the at least some of the stored metadata; if the target replica row is in a normal state, use the at least some of the metadata stored in the target replica row as the at least some of the metadata required to access the target data; if the target replica row is in another state, query the at least some of the metadata required to access the target data again from the primary row.

[0113] In an optional embodiment, the processor 75 is further configured to: in response to a garbage collection operation for the primary row, query from an auxiliary column of the primary row the identifier of at least one replica row currently associated with the primary row; perform garbage collection on the at least one replica row based on the identifier of the at least one replica row; and perform garbage collection on the primary row after completing the garbage collection on the at least one replica row.

[0114] Furthermore, as shown in Figure 7, the control device also includes other components such as a communication component 76, a display 77, a power supply component 78, and an audio component 79. Figure 7 only schematically shows some components and does not imply that the control device only includes the components shown in Figure 7. Additionally, the components within the dashed boxes in Figure 7 are optional, not mandatory, and their specific inclusion depends on the product form of the control device. The control device in this embodiment can be implemented as a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device, or as a server-side device such as a conventional server, cloud server, or server array. If the control device in this embodiment is implemented as a terminal device such as a desktop computer, laptop computer, or smartphone, it may include the components within the dashed boxes in Figure 7; if the control device in this embodiment is implemented as a server-side device such as a conventional server, cloud server, or server array, it may not include the components within the dashed boxes in Figure 7.

[0115] Accordingly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the methods described above.

[0116] This disclosure also provides a computer program product, which includes a computer program / instructions that, when executed by a processor, enable the processor to perform the steps described in the method embodiments above.

[0117] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0118] The aforementioned communication components are configured to facilitate wired or wireless communication between the device containing the communication components and other devices. The device containing the communication components can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication components receive broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication components also include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0119] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0120] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0121] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0122] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program code.

[0123] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0124] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0125] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0126] In a typical configuration, a computing device includes one or more processors (Central Processing Units, CPUs), input / output interfaces, network interfaces, and memory.

[0127] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0128] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0129] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0130] The above are merely embodiments of this disclosure and are not intended to limit the scope of this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of the claims of this disclosure.

Claims

1. A metadata query method, wherein, The method is used to query target metadata from a metadata table in a database, the metadata table including the main row where the target metadata is located, and includes: Monitor the access volume of target data or target metadata, where the target data is the data described by the target metadata; If the access volume meets the replica creation conditions, at least one replica row of the primary row is created in the metadata table, and at least some of the metadata required to access the target data is queried from the at least one replica row. The replica row is a copy of at least a portion of the metadata stored in the primary row, and the primary row and the at least one replica row are distributed in at least two data partitions of the metadata table, and the at least two data partitions are located on at least two storage nodes.

2. The method according to claim 1, wherein, If the access volume meets the replica creation conditions, at least one replica row of the primary row is created in the metadata table, including: In response to an access request for the target data or target metadata, if the access volume of the primary row meets the first load condition and there is no replica row, a replica row is created in the metadata table. If a replica row already exists, and the access volume of the existing replica row meets the second load condition, continue to create replica rows in the metadata table.

3. The method according to claim 1 or 2, wherein, Create a replica row in the metadata table, including: Based on the primary key partitioning rules of the database and the primary key of the primary row, a replica row primary key is generated, which indicates the creation of a replica row in a data partition; Based on the primary key of the replica row, write a row from the corresponding data partition of the metadata table as a replica row, and set the replica row to its initial state; Update the identifier of the replica row to the auxiliary column of the primary row. The auxiliary column is used to record the identifier of the replica row associated with the primary row, and the identifier of the replica row has a corresponding relationship with its primary key. The replica row is updated from its initial state to a normal state to complete the creation operation of the replica row. The normal state indicates that the replica row has been associated with the primary row and has the ability to query metadata.

4. The method according to claim 3, wherein, Based on the primary key of the replica row, write a row as a replica row from the corresponding data partition of the metadata table, and set the replica row to its initial state, including: The primary key of the replica row is carried in the creation instruction and sent to the database engine, so that the database engine can write a row as a replica row from the corresponding data partition of the metadata table based on the primary key of the replica row and set the replica row to the initial state.

5. The method according to claim 3 or 4, wherein, Updating the identifier of the replica row to the auxiliary column of the primary row includes: A first update instruction is sent to the database engine, so that the database engine updates the identifier of the replica row to the auxiliary column of the main row if the sequence number of the main row meets the first update condition.

6. The method according to any one of claims 3-5, wherein, Updating the replica row from its initial state to its normal state includes: A second update instruction is sent to the database engine, so that the database engine updates the replica row from its initial state to a normal state if the current state of the replica row and the sequence number of the replica row meet the second update condition.

7. The method according to any one of claims 3-6, wherein, Before writing a row as a replica row from the corresponding data partition of the metadata table based on the replica row primary key, the process also includes: Query whether a replica row corresponding to the primary key of the replica row already exists in the metadata table, and determine that a replica row corresponding to the primary key of the replica row does not yet exist in the metadata table.

8. The method according to claim 7, wherein, Also includes: If it is determined that a replica row corresponding to the primary key of the replica row already exists in the metadata table, then query whether the replica row is in a normal state; If the query result is negative, the identifier of the replica row is updated in the auxiliary column of the main row to associate the replica row with the main row, and the status of the replica row is updated to normal to complete the recovery operation of the replica row.

9. The method according to any one of claims 1-8, wherein, Also includes: If the access volume meets the replica reclamation conditions, at least some replica rows in the metadata table are garbage collected.

10. The method according to claim 9, wherein, If the access volume meets the replica reclamation conditions, garbage collection is performed on at least a portion of the replica rows in the metadata table, including: If the access volume meets the replica reclamation conditions, at least a portion of the existing replica rows will be identified as replica rows to be reclamped. If the copy row to be recycled exists in the metadata table, the status of the copy row to be recycled is updated to recycled status, which indicates that the copy row to be recycled does not have the ability to query metadata. Remove the identifier of the copy row to be recycled from the auxiliary column of the main row to decouple the copy row to be recycled from the main row; The row of the copy to be recycled is deleted from the metadata table to complete the garbage collection process for the copy to be recycled.

11. The method according to claim 10, wherein, Also includes: If the copy row to be recycled does not exist in the metadata table, the copy row to be recycled is written into the metadata table according to the primary key of the copy row to be recycled, and the copy row to be recycled is in the recycling status. Remove the identifier of the copy row to be recycled from the auxiliary column of the main row to decouple the copy row to be recycled from the main row; The row of the copy to be recycled is deleted from the metadata table to complete the garbage collection process for the copy to be recycled.

12. The method according to claim 10 or 11, wherein, Removing the identifier of the copy row to be recycled from the auxiliary column of the main row to decouple the copy row from the main row includes: A third update instruction is sent to the database engine, so that the database engine, if the sequence number of the main row and the value of the auxiliary column meet the third update condition, deletes the identifier of the replica row to be recycled from the auxiliary column of the main row, thereby severing the association between the replica row to be recycled and the main row.

13. The method according to any one of claims 1-12, wherein, Querying at least some of the metadata required to access the target data from the at least one copy row includes: In response to the access request for the target data, the target replica row is determined from the at least one replica row based on the query load of the replica row; Query the current status of the target replica row and at least some of the stored metadata; If the target replica is in a normal state, at least some of the metadata stored in the target replica row will be used as at least some of the metadata required to query and access the target data; If the target copy is in another state, at least some of the metadata required to access the target data should be retrieved from the main row again.

14. The method according to any one of claims 1-13, wherein, Also includes: In response to a garbage collection operation targeting the primary row, query the identifier of at least one replica row currently associated with the primary row from the auxiliary columns of the primary row; Based on the identifier of the at least one replica row, perform garbage collection on the at least one replica row; After the garbage collection process of the at least one replica row is completed, the garbage collection process of the primary row is performed.

15. The method according to any one of claims 3-6, wherein, Before updating the replica row from its initial state to its normal state, the following steps are also included: At least a portion of the metadata stored in the primary row is synchronized to the replica row, wherein the synchronization operation is performed by the control node and is triggered when the replica row is in its initial state and has been associated with the primary row through an auxiliary column.

16. The method according to claim 10 or 11, wherein, Before deleting the copy row to be recycled from the metadata table, the method also includes: A delayed deletion strategy is implemented on the replica rows to be reclaimed. The delayed deletion strategy includes: after marking the replica rows to be reclaimed as reclaimed, waiting for a preset delay time before performing the physical deletion operation. The preset delay time is used to avoid wild reference problems caused by network latency or retry requests.

17. A control device, wherein, include: A memory and a processor, the memory for storing a computer program, the processor coupled to the memory for executing the computer program to implement the steps of the method of any one of claims 1-16.

18. A storage system, wherein, include: The system comprises a control device, a database, and at least one storage node; the database includes a metadata table, the metadata table includes a main row containing the target metadata, and the at least one storage node is used to store the target data corresponding to the target metadata; the control device is used to perform the steps of the method according to any one of claims 1-16 to query the target metadata from the metadata table.

19. A computer-readable storage medium storing a computer program / instructions, wherein, When the computer program / instructions are executed by the processor, the processor is enabled to perform the steps of the method according to any one of claims 1-16.

20. A computer program product, wherein, include: A computer program / instruction that, when executed by a processor, causes the processor to perform the steps of the method according to any one of claims 1-16.

Images