Data processing

Abstract

Embodiments of the present disclosure relate to a data processing method, a computing device, a storage medium, and a computer program product. The data processing method comprises: in response to a data writing request for a target data structure, determining data to be written carried in the data writing request; when it is determined that the data to be written comprises first data, allocating a first data storage node outside the target data structure to the first data when the target data structure is lock-free, wherein the data volume of the first data is greater than a preset data volume threshold; when the target data structure is lock-free, writing the first data into the first data storage node, wherein the first data storage node is associated with the target data structure.

Description

Data processing Technical Field

[0001] This disclosure relates to the field of database technology, and in particular to data processing. Background Technology

[0002] In practical applications, large field types are widely used in the database field. They are typically used to store large amounts of data, such as log text, images, and binary streams. However, because large field data occupies a large amount of space, writing large field data to the database places high demands on database performance, and write performance is relatively low. Therefore, an effective technical solution is urgently needed to address these issues. Summary of the Invention

[0003] In view of the above, embodiments of this disclosure provide a data processing method. One or more embodiments of this disclosure also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product.

[0004] According to a first aspect of the present disclosure, a data processing method is provided, comprising: in response to a data write request for a target data structure, determining data to be written carried in the data write request; if it is determined that the data to be written includes first data, allocating a first data storage node outside the target data structure to the first data when the target data structure is not locked, wherein the data volume of the first data is greater than a preset data volume threshold; and writing the first data to the first data storage node when the target data structure is not locked, wherein the first data storage node is associated with the target data structure.

[0005] According to a second aspect of the present disclosure, a data processing apparatus is provided, comprising: a determining module configured to determine, in response to a data write request for a target data structure, data to be written carried in the data write request; an allocation module configured to, when it is determined that the data to be written contains first data, allocate a first data storage node outside the target data structure to the first data when the target data structure is not locked, wherein the amount of the first data is greater than a preset data amount threshold; and a writing module configured to, when the target data structure is not locked, write the first data to the first data storage node, wherein the first data storage node is associated with the target data structure.

[0006] According to a third aspect of the present disclosure, a computing device is provided, comprising: a memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, wherein the computer programs / instructions, when executed by the processor, implement the steps of the above-described data processing method.

[0007] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0008] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0009] One embodiment of this disclosure provides a data processing method, comprising: in response to a data write request for a target data structure, determining data to be written carried in the data write request; if it is determined that the data to be written includes first data, allocating a first data storage node outside the target data structure to the first data when the target data structure is not locked, wherein the data volume of the first data is greater than a preset data volume threshold; and writing the first data to the first data storage node when the target data structure is not locked, wherein the first data storage node is associated with the target data structure.

[0010] In the above method, after responding to a data write request for the target data structure, the data to be written carried by the data write request can be determined. If the data to be written contains the first data, since the data volume of the first data is greater than the preset data volume threshold, that is, the first data is a large field data, the first data cannot be written into the target data structure. In the absence of a lock on the target data structure, a first data storage node outside the target data structure can be allocated for the first data. And in the absence of a lock on the target data structure, the first data is written to the external first data storage node. The writing of the first data is performed without locking the first data storage node and without locking the target data structure, thus ensuring the writing performance of the first data. Attached Figure Description

[0011] Figure 1 is a schematic diagram of an application scenario of a data processing method provided in an embodiment of this disclosure.

[0012] Figure 2 is a flowchart of a data processing method provided in an embodiment of this disclosure.

[0013] Figure 3 is a schematic diagram of the target data structure in a data processing method provided in an embodiment of this disclosure.

[0014] Figure 4 is a schematic diagram of allocating a first data storage node in a data processing method provided in an embodiment of this disclosure.

[0015] Figure 5 is a schematic diagram of writing non-large field data in a data processing method provided in an embodiment of this disclosure.

[0016] Figure 6 is a schematic diagram of external page pointer modification in a data processing method provided by an embodiment of this disclosure.

[0017] Figure 7 is a schematic diagram of data deletion in a data processing method provided in an embodiment of this disclosure.

[0018] Figure 8 is a flowchart of a data processing method provided in an embodiment of this disclosure.

[0019] Figure 9 is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this disclosure.

[0020] Figure 10 is a structural block diagram of a computing device provided in an embodiment of this disclosure. Detailed Implementation

[0021] Numerous specific details are set forth in the following description to provide a full understanding of this disclosure. However, this disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this disclosure. Therefore, this disclosure is not limited to the specific implementations disclosed below.

[0022] The terminology used in one or more embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this disclosure. The singular forms “a,” “the,” and “the” as used in one or more embodiments of this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this disclosure refers to and includes any or all possible combinations of one or more associated listed items.

[0023] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this disclosure, and similarly, second may also be referred to as first. Depending on the context, the word “if” as used herein may be interpreted as “when”, “in response to a determination”, or “when…”.

[0024] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this disclosure are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0025] First, the terms and concepts involved in one or more embodiments of this disclosure will be explained.

[0026] Large data fields: Data storage fields that occupy a large amount of space in the database, including both character objects and binary objects. Examples include images, audio files, video files, documents, and other types of multimedia content.

[0027] Mutex lock: Also known as a write lock, if a write lock exists on a leaf node, that leaf node cannot be accessed. It is used to prevent multiple threads from accessing shared resources at the same time. Other threads need to wait for the current thread to release the write lock before they can access it.

[0028] Shared lock: also known as read lock, is a synchronization mechanism used to solve concurrent access control problems, allowing multiple threads holding read locks to access shared resources simultaneously.

[0029] B-tree: A self-balancing tree-like data structure where all leaf nodes are at the same level, and each node can have multiple child nodes. It is a widely used data structure in database management systems to improve the speed of data retrieval.

[0030] Primary key index: A special type of index in a database, created based on the primary key of a table, typically using a B-tree index. A primary key is one or a set of fields used to uniquely identify each row in a table.

[0031] Page: The smallest unit of data organization in a database.

[0032] In practical applications, large field types are widely used in databases. Users can use large fields to store large amounts of data, such as log text, images, and binary streams, achieving persistence by writing them to the database. Concurrent writes of large fields pose a significant challenge to database performance because large fields themselves occupy a large amount of space and cannot be directly stored in fixed-size pages of the index. Furthermore, the long write time consumes more I / O bandwidth for log and data writes. During multi-threaded concurrent writes, it is usually necessary to hold mutex locks on the index and outer pages. Long-running operations on large fields consume even more lock holding time, reducing concurrent write performance. Even with higher I / O bandwidth, user write performance cannot be improved. Therefore, an effective technical solution is urgently needed to address these issues.

[0033] This disclosure provides a data processing method, and also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0034] Referring to Figure 1, Figure 1 illustrates an application scenario of a data processing method provided according to an embodiment of the present disclosure. The data processing method specifically includes the following steps.

[0035] In response to a data write request for a target data structure, the data to be written carried in the data write request is determined; if the data to be written contains first data, a first data storage node outside the target data structure is allocated to the first data if the target data structure is not locked, wherein the data volume of the first data is greater than a preset data volume threshold; if the target data structure is not locked, the first data is written to the first data storage node, wherein the first data storage node is associated with the target data structure.

[0036] Specifically, Figure 1 includes an end-side device 102 and a database 104, which uses the target data structure for data indexing and data storage.

[0037] In specific implementation, the user can send a data write request to the database 104 through the terminal device 102. The database 104 can determine the data to be written carried in the data write request. If it is determined that the data to be written contains first data with a data volume greater than a preset data volume threshold, the database 104 allocates a first data storage node outside the target data structure for the first data if the target data structure does not hold any lock. If the target data structure does not hold any lock, the first data is written to the first data storage node outside the target data structure, thereby realizing the writing of the first data.

[0038] The edge device 102 may include a browser, an app (application), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The edge device can be developed based on a software development kit (SDK) provided by the server, such as a real-time communication (RTC) SDK. The edge device can be deployed in an electronic device and depends on the device's operation or certain apps within the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0039] Referring to Figure 2, Figure 2 shows a flowchart of a data processing method provided according to an embodiment of the present disclosure, which specifically includes the following steps.

[0040] Step 202: In response to a data write request for a target data structure, determine the data to be written carried in the data write request.

[0041] The target data structure can be understood as the data structure used for data indexing and data storage in the database. In practical applications, it can be a primary key index, which can be understood as an index built on the primary key in the database. In the case of a relational database, the primary key index can be a B-tree structure. The data write request can be understood as a data write request to the database corresponding to the target data structure. The data to be written can be understood as the data that needs to be written to the database.

[0042] Specifically, the database can respond to a user's data write request sent through a client for a target data structure and determine the data that needs to be written to the database, carried in the data write request.

[0043] Step 204: If it is determined that the data to be written contains the first data, and the target data structure does not have a lock, allocate a first data storage node outside the target data structure to the first data, wherein the amount of the first data is greater than a preset data amount threshold.

[0044] The target data structure does not have locks, which can be understood as the target data structure not having global read locks and write locks. The first data can be understood as data whose volume is greater than a preset data volume threshold. The first data can be data of the first data type, such as large field data. The first data storage node can be understood as an external page outside the target data structure, used to store the first data in the database.

[0045] Specifically, if it is determined that the data to be written contains large field data, an external page outside the target data structure can be allocated for the large field data, provided that the target data structure does not hold a global read lock or write lock.

[0046] In practical applications, refer to Figure 3, which illustrates a schematic diagram of the target data structure in a data processing method according to an embodiment of this disclosure. As shown in Figure 3, the target data structure can be a B-tree (i.e., a primary key index). The target data structure includes multiple leaf nodes, each of which can be used to store data. Each leaf node can contain row records and column data. Row records can be used to store data index information, and column data can be used to store data. Outside the target data structure, there is an external page list, which consists of multiple external pages used to store large field data. Specifically, in a relational database organized using a B-tree index, each leaf node of the B-tree stores a fixed amount of data. If a row of data cannot fit in a single leaf node, large field data that occupies a large amount of space will be stored separately on a data page outside the primary key index. Therefore, the large field data and the primary key index stored in the B-tree are separated. The large field data is stored separately in an external page outside the primary key index structure, and a reference to this external page is stored in the primary key index.

[0047] In specific implementation, allocating a first data storage node outside the target data structure to the first data includes: allocating a first data storage node outside the target data structure to the first data based on the data volume of the first data, wherein the storage space of the first data storage node is greater than or equal to the data volume.

[0048] Specifically, the number of external pages that need to be allocated can be calculated based on the amount of data in the first data, and external pages outside the target data structure can be allocated to the first data.

[0049] In practical applications, referring to Figure 4, which illustrates the allocation of a first data storage node in a data processing method according to an embodiment of this disclosure, the required number of external pages (i.e., the first data storage node) can be calculated for each large field column (i.e., the amount of first data) in the written row record without holding any locks on the target data structure. Disk space for these external pages can then be allocated to the first data as the first data storage node. Furthermore, to improve space allocation efficiency, an asynchronous allocation thread can be used to allocate the first data storage node. That is, the required external pages can be allocated using an asynchronous allocation thread. The thread writing the large field data in the foreground can allocate the required external page disk space from the file system in batches while holding a file lock. For each large field data, the allocated external pages are chained together into a page linked list, and the content of each large field data is copied one by one to the external page linked list, thereby writing the content of the large field data that needs to be stored on the external pages.

[0050] In summary, since a global file lock is held only when allocating disk space from the file system, most time-consuming operations on large fields of data are lock-free, such as page initialization and data copying. This supports parallel write operations on multiple large fields of data.

[0051] Furthermore, after writing the first data to the first data storage node, the method further includes: determining reference information corresponding to the first data storage node; and associating the first data storage node and the target data structure according to the reference information.

[0052] The reference information of the first data storage node can be understood as information used to index from the target data structure to the first data storage node. For example, it can be the location information of the first data storage node, or it can be the link information that can be linked to the first data storage node.

[0053] Specifically, the location information of the first data storage node can be determined, and this location information can be used to associate the primary key index information of the first data storage node and the target data structure, so as to query the data content of the first data storage node through the primary key index.

[0054] In summary, by utilizing location information to manage the primary key index information of the first data storage node and the target data structure, the location of the first data storage node can be determined through the primary key index, and the data content in the first data storage node can be further queried.

[0055] In specific implementation, associating the first data storage node and the target data structure according to the reference information includes: storing the reference information in the target data structure, and realizing the association between the first data storage node and the target data structure according to the reference information stored in the target data structure.

[0056] Specifically, the location information of the first data storage node can be stored in the primary key index to realize the association between the first data storage node and the target data structure.

[0057] Step 206: If there is no lock on the target data structure, write the first data to the first data storage node, wherein the first data storage node is associated with the target data structure.

[0058] The target data structure does not have a lock, which can be understood as the target data structure not having a read lock or a write lock.

[0059] Furthermore, after determining that the data to be written contains the first data, the method further includes: if it is determined that the data to be written also contains the second data, determining the second data storage node corresponding to the second data in the target data structure according to the data index information of the second data, wherein the data volume of the second data is less than the preset data volume threshold; and writing the second data into the second data storage node.

[0060] Here, the second data can be understood as data with a volume smaller than a preset data volume threshold. The second data can be data of a second data type; that is, the first data type and the second data type are different. Therefore, the second data can be, for example, non-large field data. The second data storage node can be understood as a leaf node in the target data structure. Therefore, the second data storage node can include column data in the row records whose length is smaller than the preset data volume threshold.

[0061] Specifically, if it is determined that the data to be written includes first data and second data, the first data is allocated to the first data storage node as described above. For the second data, the second data storage node in the target data structure to be determined can be determined based on the data index information of the second data and the primary key recorded in the row record included in the second data storage node, and the second data is written to the second data storage node.

[0062] In summary, when it is determined that the data to be written contains both large and non-large fields, different processing methods can be applied to the large and non-large fields to achieve the writing of data for both fields.

[0063] In specific implementation, determining the second data storage node corresponding to the second data in the target data structure based on the data index information of the second data includes: determining the second data storage node corresponding to the second data in the target data structure based on the data index information of the second data when a read lock exists in the target data structure; writing the second data to the second data storage node includes: writing the second data to the second data storage node when a write lock exists in the second data storage node.

[0064] The data index information of the second data can be used to determine which leaf node in the target data structure the second data is stored in. The read lock on the target data structure can be understood as a global read lock on the target data structure.

[0065] Specifically, if the target data structure holds a global read lock, the second data storage node where the second data is about to be written can be determined in the target data structure based on the data index information of the second data, and a write lock can be added to the second data storage node. If the second data storage node holds a write lock, the second data can be written to the second data storage node.

[0066] In practical applications, refer to Figure 5, which illustrates a schematic diagram of non-large field data writing in a data processing method according to an embodiment of this disclosure. As shown in Figure 5, when the target data structure holds a global read lock, starting from the root node of the target data structure, the leaf node (i.e., the second data storage node) where the data to be written is about to be written is searched. Then, the write lock of the second data storage node is acquired. The column data of the non-large field data stored in the primary key index is modified first, and the space required for the column data of the large field data is reserved. Through the marking of the row record by the transaction lock system, other threads cannot modify the current row record. Based on this, the global read lock of the target data structure and the write lock of the second data storage node can be released. Since only the write lock of the second data storage node exists at this stage, and the global write lock of the target data structure is not held, parallel write operations of multiple large fields can be supported.

[0067] Furthermore, after writing the second data to the second data storage node when a write lock exists on the second data storage node, the method further includes: releasing the read lock on the target data structure and the write lock on the second data storage node.

[0068] Specifically, after writing non-large field data to the second data storage node, the read lock of the target data structure and the write lock of the second data storage node can be released, which facilitates subsequent processing of the data in the target data structure and the second data storage node.

[0069] In practical applications, storing the reference information into the target data structure includes: storing the reference information into the target data structure when a read lock exists in the target data structure.

[0070] Specifically, when the target data structure holds a read lock and the second data storage node holds a write lock, the reference information of the first data storage node is stored in the target data structure.

[0071] In practical applications, referring to Figure 6, which illustrates the modification of external page pointers in a data processing method according to an embodiment of this disclosure, when the data to be written includes both large-field data and non-large-field data, as shown in Figure 5 above, before the content of the large-field data is written to the external page, the row records in the target data structure only modify the content of the primary key index, reserving the space pointing to the external page in each large-field data, without actually writing it to the actual external page location. Based on this, when the target data structure holds a global read lock on the primary key index and a write lock on the corresponding leaf node (i.e., the second data storage node), the location of the external page linked list corresponding to the already written large-field data (i.e., the reference information of the first data storage node) is stored in the primary key index. Since there is no global write lock on the primary key index at this time, parallel write operations of multiple large fields can be supported.

[0072] In practical applications, the target data structure includes multiple leaf nodes; determining the second data storage node corresponding to the second data in the target data structure based on the data index information of the second data includes: determining the second data storage node corresponding to the second data among the multiple leaf nodes contained in the target data structure based on the data index information of the second data and the index information of the target data structure.

[0073] The index information of the target data structure can be understood as the data index information stored in the row records of the leaf nodes in the target data structure.

[0074] Specifically, based on the data index information of the second data and the data index information of the row records stored in the leaf nodes of the target data structure, the second data storage node for writing the second data can be determined from among the multiple leaf nodes contained in the target data structure. This enables subsequent writing of the second data.

[0075] In addition, in one embodiment of this disclosure, the method further includes: in response to a data deletion request for the first data, deleting the first data stored in the first data storage node using a data deletion thread to obtain a deleted first data storage node; and providing the deleted first data storage node to a data writing thread using an asynchronous allocation thread for data writing.

[0076] Among them, the data deletion thread can be understood as the thread that deletes data, the asynchronous allocation thread can be understood as the thread that allocates external pages, and the data writing thread can be understood as the thread that writes the first and second data.

[0077] Specifically, when a data deletion request for large field data is received, the data deletion thread can be used to clean up the first data storage node, that is, to delete the first data stored in the first data storage node, obtain the deleted first data storage node, and use the asynchronous allocation thread to provide the deleted first data storage node to the data writing thread for subsequent data writing.

[0078] In practical applications, refer to Figure 7, which illustrates a data deletion process in a data processing method according to an embodiment of this disclosure. For data update and deletion operations, old external pages need to be cleaned up. To reduce the overhead of foreground write tasks, a background asynchronous allocation thread can be used for cleanup. The external pages of large fields marked for deletion can be prefetched from the disk into memory. Then, while holding a file lock, the external page space is cleaned up and returned to the asynchronous allocation thread for reuse during the next large field data write. Since the data deletion task is not on the data write thread and the time-consuming disk read operation is performed without holding a global write lock, the parallel writing efficiency of large fields is improved.

[0079] In addition, after determining the data to be written carried in the data write request, the method further includes: if the data to be written is determined to be second data, determining the second data storage node corresponding to the second data in the target data structure according to the data index information of the second data, wherein the data volume of the second data is less than the preset data volume threshold; and writing the second data into the second data storage node.

[0080] Specifically, when the data to be written is not a large field, the second data storage node can be directly determined from the target data structure, and the non-large field data can be written to the second data storage node.

[0081] In the above method, after responding to a data write request for the target data structure, the data to be written carried by the data write request can be determined. If the data to be written contains the first data, since the data volume of the first data is greater than the preset data volume threshold, that is, the first data is a large field data, the first data cannot be written into the target data structure. The first data can be allocated to a first data storage node outside the target data structure, and the first data can be written to the external first data storage node when there is no lock on the target data structure. The first data is written without locking the first data storage node and without locking the target data structure, thus ensuring the writing performance of the first data.

[0082] The following description, in conjunction with Figure 8, uses the application of the data processing method provided in this disclosure for writing large-field data and non-large-field data as examples to further illustrate the data processing method. Figure 8 shows a flowchart of the processing procedure of a data processing method provided in an embodiment of this disclosure, specifically including the following steps.

[0083] Step 802: In response to a data write request for a target data structure, determine the data to be written carried in the data write request.

[0084] Step 804: If it is determined that the data to be written contains both large field data and non-large field data, determine the second data storage node corresponding to the non-large field data in the target data structure according to the data index information of the non-large field data, and write the non-large field data into the second data storage node.

[0085] Specifically, in the first stage, based on the index information of the non-large field data, the position of the row record in the target data structure (i.e., the primary key index) can be located, and the content of the non-large field data can be modified. Holding a global read lock on the primary key index, starting from the root node of the target data structure, the search proceeds to the second data storage node where the large field data will be written. Then, a write lock is acquired on the second data storage node. The column data of the non-large field data stored in the primary key index is modified first, while reserving space for the column data of the large field data. Finally, the global read lock on the primary key index and the write lock on the second data storage node are released.

[0086] Step 806: Allocate a first data storage node outside the target data structure for the large field data, and write the large field data into the first data storage node.

[0087] Specifically, the second stage involves offline allocation and copying of the external pages (i.e., the first data storage node) for large field data. Without holding any locks on the primary key index (global locks and leaf node locks), the number of external pages required for each large field data column in the written row records is calculated. To improve space allocation efficiency, an asynchronous allocation thread is used to allocate the required external pages. The foreground thread writing the large field data only needs to allocate the required disk space for the external pages from the file system in batches while holding the file lock. For each large field data, the requested external pages are chained together into a page linked list, and the content of each large field data is copied one by one to the external page linked list. The content of the large field data that needs to be stored on the external pages is then written.

[0088] Step 808: Write the reference information of the first data storage node into the target data structure to realize the association between the first data storage node and the target data structure.

[0089] Specifically, the third stage involves modifying the external page pointers for large fields in the primary key index. In the first stage, because the content of the large fields was not written to the external pages, the row records only modified the non-large fields of the primary key index, reserving the space for pointing to external pages in each large field without actually writing to the actual external page location. Therefore, the global read lock on the primary key index and the write lock on the corresponding leaf node (i.e., the second data storage node) are held again. The positions of the external page linked lists corresponding to the already written large fields (i.e., the reference information of the first data storage node) are stored in the primary key index.

[0090] Step 810: In response to the data deletion request for large field data, use the background data deletion thread to delete the large field data stored in the first data storage node and obtain the deleted first data storage node; use the asynchronous allocation thread to provide the deleted first data storage node to the data writing thread for the next data writing.

[0091] In this case, the data deletion request can mark only the large field data that needs to be deleted.

[0092] Specifically, a background asynchronous thread can be used for cleanup. First, the external pages of expired large field data are prefetched from the disk into memory. Then, the file lock is acquired, the external page space is cleaned up, and returned to the asynchronous allocation thread for reuse when allocating space for the next large field data write.

[0093] In the above method, after responding to a data writing request for a target data structure, the data to be written carried in the data writing request can be determined. When it is determined that the data to be written contains the first data, since the amount of the first data is greater than a preset data volume threshold, that is to say, the first data is large field data. At this time, the first data cannot be written into the target data structure. A first data storage node outside the target data structure can be allocated for the first data, and when there is no lock on the target data structure, the first data is written into the external first data storage node. The first data is written without locking the first data storage node and without locking the target data structure, ensuring the writing performance of the first data.

[0094] Corresponding to the above method embodiment, the present disclosure also provides a data processing apparatus embodiment. FIG. 9 shows a schematic structural diagram of a data processing apparatus provided by an embodiment of the present disclosure. As shown in FIG. 9, the apparatus includes: a determining module 902, configured to determine the data to be written carried in the data writing request in response to a data writing request for a target data structure; an allocating module 904, configured to allocate a first data storage node outside the target data structure for the first data when it is determined that the data to be written contains the first data, where the amount of the first data is greater than a preset data volume threshold; a writing module 906, configured to write the first data into the first data storage node when there is no lock on the target data structure, where the first data storage node is associated with the target data structure.

[0095] In an optional embodiment, the allocating module 904 is further configured to: allocate a first data storage node outside the target data structure for the first data according to the amount of the first data, where the storage space of the first data storage node is greater than or equal to the data volume.

[0096] In an optional embodiment, the writing module 906 is further configured to: determine the reference information corresponding to the first data storage node; and associate the first data storage node with the target data structure according to the reference information.

[0097] In an optional embodiment, the writing module 906 is further configured to: store the reference information into the target data structure, and implement the association between the first data storage node and the target data structure according to the reference information stored in the target data structure.

[0098] In an optional embodiment, the writing module 906 is further configured to: when it is determined that the data to be written also contains second data, determine the second data storage node corresponding to the second data in the target data structure according to the data index information of the second data, wherein the data volume of the second data is less than the preset data volume threshold; and write the second data into the second data storage node.

[0099] In an optional embodiment, the writing module 906 is further configured to: determine the second data storage node corresponding to the second data in the target data structure according to the data index information of the second data when a read lock exists in the target data structure; and write the second data to the second data storage node when a write lock exists in the second data storage node.

[0100] In an optional embodiment, the write module 906 is further configured to: release the read lock of the target data structure and the write lock of the second data storage node.

[0101] In an optional embodiment, the writing module 906 is further configured to: store the reference information into the target data structure when a read lock exists in the target data structure.

[0102] In an optional embodiment, the target data structure includes multiple leaf nodes; the writing module 906 is further configured to: determine the second data storage node corresponding to the second data from among the multiple leaf nodes contained in the target data structure, based on the data index information of the second data and the index information of the target data structure.

[0103] In an optional embodiment, the apparatus further includes a deletion module configured to: in response to a data deletion request for the first data, use a data deletion thread to delete the first data stored in the first data storage node to obtain a deleted first data storage node; and use an asynchronous allocation thread to provide the deleted first data storage node to a data writing thread for data writing.

[0104] In an optional embodiment, the writing module 906 is further configured to: when it is determined that the data to be written is second data, determine the second data storage node corresponding to the second data in the target data structure according to the data index information of the second data, wherein the data volume of the second data is less than the preset data volume threshold; and write the second data to the second data storage node.

[0105] In the aforementioned device, in response to a data write request for a target data structure, the data to be written carried by the data write request can be determined. If the data to be written contains first data, and since the amount of the first data is greater than a preset data amount threshold, that is, the first data is large field data, the first data cannot be written into the target data structure. The first data can be allocated to a first data storage node outside the target data structure, and the first data can be written to the external first data storage node when there is no lock on the target data structure. The first data is written without locking the first data storage node and without locking the target data structure, thus ensuring the writing performance of the first data.

[0106] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0107] Figure 10 shows a structural block diagram of a computing device 1000 according to an embodiment of the present disclosure. The components of the computing device 1000 include, but are not limited to, a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and a database 1070 is used to store data.

[0108] The computing device 1000 also includes an access device 1040, which enables the computing device 1000 to communicate via one or more networks 1060. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1040 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0109] In one embodiment of this application, the aforementioned components of the computing device 1000, as well as other components not shown in FIG. 10, may be interconnected, for example, via a bus. It should be understood that the computing device structural block diagram shown in FIG. 10 is merely for illustrative purposes and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0110] The computing device 1000 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1000 can also be a mobile or stationary server.

[0111] The processor 1020 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0112] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the data processing method embodiments.

[0113] An embodiment of this disclosure also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0114] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the data processing method embodiments.

[0115] An embodiment of this disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0116] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0117] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0119] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this disclosure are not limited to the described order of actions, because according to the embodiments of this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this disclosure.

[0120] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0121] The preferred embodiments disclosed above are merely illustrative of this disclosure. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments of this disclosure. These embodiments are selected and specifically described in this disclosure to better explain the principles and practical applications of the embodiments of this disclosure, thereby enabling those skilled in the art to better understand and utilize this disclosure. This disclosure is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: In response to a data write request for a target data structure, determine the data to be written carried in the data write request; If it is determined that the data to be written contains first data, and the target data structure does not have a lock, a first data storage node outside the target data structure is allocated to the first data, wherein the amount of the first data is greater than a preset data amount threshold. If no lock exists in the target data structure, the first data is written to the first data storage node, wherein the first data storage node is associated with the target data structure.

2. The method according to claim 1, wherein allocating a first data storage node outside the target data structure to the first data comprises: Based on the amount of the first data, a first data storage node other than the target data structure is allocated to the first data, wherein the storage space of the first data storage node is greater than or equal to the amount of data.

3. The method according to claim 1, further comprising, after writing the first data to the first data storage node: Determine the reference information corresponding to the first data storage node; Based on the reference information, the first data storage node and the target data structure are associated.

4. The method according to claim 3, wherein associating the first data storage node and the target data structure according to the reference information includes: The reference information is stored in the target data structure, and the association between the first data storage node and the target data structure is realized based on the reference information stored in the target data structure.

5. The method according to any one of claims 1-4, further comprising, after determining that the data to be written contains the first data: If it is determined that the data to be written also contains second data, the second data storage node corresponding to the second data is determined in the target data structure according to the data index information of the second data, wherein the data volume of the second data is less than the preset data volume threshold; Write the second data to the second data storage node.

6. The method according to claim 5, wherein determining the second data storage node corresponding to the second data in the target data structure based on the data index information of the second data comprises: If a read lock exists in the target data structure, the second data storage node corresponding to the second data is determined in the target data structure based on the data index information of the second data. The step of writing the second data to the second data storage node includes: If a write lock exists on the second data storage node, the second data is written to the second data storage node.

7. The method according to claim 6, further comprising, after writing the second data to the second data storage node when a write lock exists on the second data storage node: Release the read lock on the target data structure and the write lock on the second data storage node.

8. The method according to claim 4, wherein storing the reference information into the target data structure comprises: If a read lock exists in the target data structure, the reference information is stored in the target data structure.

9. The method according to claim 5, wherein the target data structure includes multiple leaf nodes; The step of determining the second data storage node corresponding to the second data in the target data structure based on the data index information of the second data includes: Based on the data index information of the second data and the index information of the target data structure, the second data storage node corresponding to the second data is determined from among the multiple leaf nodes contained in the target data structure.

10. The method according to any one of claims 1-4, further comprising: In response to the data deletion request for the first data, the first data stored in the first data storage node is deleted using a data deletion thread to obtain the first data storage node after deletion; An asynchronous allocation thread is used to provide the deleted first data storage node to the data writing thread for data writing.

11. The method according to any one of claims 1-4, wherein after determining the data to be written carried in the data write request, it further comprises: If the data to be written is determined to be second data, the second data storage node corresponding to the second data is determined in the target data structure according to the data index information of the second data, wherein the data volume of the second data is less than the preset data volume threshold. Write the second data to the second data storage node.

12. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which perform the following operations when executed by the processor: In response to a data write request for a target data structure, determine the data to be written carried in the data write request; If it is determined that the data to be written contains first data, and the target data structure does not have a lock, a first data storage node outside the target data structure is allocated to the first data, wherein the amount of the first data is greater than a preset data amount threshold. If no lock exists in the target data structure, the first data is written to the first data storage node, wherein the first data storage node is associated with the target data structure.

13. The computing device of claim 12, wherein allocating a first data storage node other than the target data structure to the first data comprises: Based on the amount of the first data, a first data storage node other than the target data structure is allocated to the first data, wherein the storage space of the first data storage node is greater than or equal to the amount of data.

14. The computing device according to claim 12, wherein after writing the first data to the first data storage node, the operation further includes: Determine the reference information corresponding to the first data storage node; Based on the reference information, the first data storage node and the target data structure are associated.

15. The computing device of claim 14, wherein associating the first data storage node and the target data structure according to the reference information comprises: The reference information is stored in the target data structure, and the association between the first data storage node and the target data structure is realized based on the reference information stored in the target data structure.

16. The computing device according to any one of claims 12-15, wherein after determining that the data to be written contains the first data, the operation further comprises: If it is determined that the data to be written also contains second data, the second data storage node corresponding to the second data is determined in the target data structure according to the data index information of the second data, wherein the data volume of the second data is less than the preset data volume threshold; Write the second data to the second data storage node.

17. The computing device according to claim 16, wherein determining the second data storage node corresponding to the second data in the target data structure based on the data index information of the second data comprises: If a read lock exists in the target data structure, the second data storage node corresponding to the second data is determined in the target data structure based on the data index information of the second data. The step of writing the second data to the second data storage node includes: If a write lock exists on the second data storage node, the second data is written to the second data storage node.

18. The computing device according to claim 17, wherein after writing the second data to the second data storage node when a write lock exists on the second data storage node, the operation further includes: Release the read lock on the target data structure and the write lock on the second data storage node.

19. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

20. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

Images