Data processing method, system and apparatus

Abstract

Provided in embodiments of the present disclosure are a data processing method, system and apparatus. The method comprises: receiving a plurality of data processing requests, wherein the plurality of data processing requests carry virtual file identifiers and data processing content corresponding to the data processing requests; on the basis of the virtual file identifiers, determining virtual files corresponding to the data processing requests and target file sets corresponding to the virtual files, wherein each target file set is a set of physical files in a target data storage system, at least two virtual files among the plurality of virtual files correspond to a same target file set, and one target file set corresponds to one file handle; and using the file handles of the target file sets to open the target file sets, and executing the plurality of data processing requests on the basis of the data processing content, the virtual files, and the target file sets corresponding to the virtual files. When a large number of files are processed, one (or a small number of) files are actually written into a file system, thereby reducing resource occupation and supporting more concurrent processing.

Description

Data processing methods, systems and devices

[0001] This disclosure claims priority to Chinese Patent Application No. 202411554935.4, filed with the China Patent Office on November 1, 2024, entitled “Data Processing Method, System and Apparatus”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of file system technology, and in particular to a data processing method, system, and apparatus. Background Technology

[0003] Distributed databases often shard tables into regions, with different shards distributed across different machines to enable multi-machine collaborative services. Even a single machine may have multiple shards. Because a large amount of concurrent file processing (including concurrent writes and reads, each write or read requires opening the file) can lead to insufficient file handle resources, HBase (an open-source, distributed storage system) uses HDFS (Hadoop Distributed File System) for log writing. Each time HDFS opens a file, it consumes several TCP (Transmission Control Protocol) threads.

[0004] To address this issue, a single database instance often writes multiple sharded WAL (Write-Ahead Logging, a transaction log in databases; in database management systems, WAL is a logging mechanism used to ensure data consistency and durability. When the database executes a transaction, all changes are first recorded in a WAL log file before updating the main data store) to a single file. This reduces the consumption of file handle resources. However, during error recovery, the log file needs to be split according to the shards before data recovery can proceed. This not only increases data recovery time but also introduces additional complexity to the database itself. Therefore, this method is not suitable for solving the problem of wasted file handle resources during concurrent file writes. Summary of the Invention

[0005] In view of this, the present disclosure provides two data processing methods. One or more embodiments of the present disclosure also relate to a data processing system, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to solve the technical defects in the prior art where concurrent processing of multiple files causes insufficient file handle resources.

[0006] According to a first aspect of the present disclosure, a data processing method is provided, comprising:

[0007] Receive multiple data processing requests, wherein each of the multiple data processing requests carries a corresponding virtual file identifier and data processing content;

[0008] Based on the virtual file identifier, the virtual file corresponding to the data processing request and the target file set corresponding to the virtual file are determined. The target file set is a set of physical files in the target data storage system, and at least two virtual files in the multiple virtual files correspond to the same target file set. One target file set corresponds to one file handle.

[0009] The target file set is opened using the file handle of the target file set, and the multiple data processing requests are executed according to the data processing content, the virtual file, and the target file set corresponding to the virtual file.

[0010] According to a second aspect of the present disclosure, a data processing method is provided, applied to a log file scenario, including:

[0011] Receive multiple log data processing requests, wherein each of the multiple data processing requests carries a corresponding virtual log file identifier and log data processing content;

[0012] Based on the virtual log file identifier, the virtual log file corresponding to the log data processing request and the target log file set corresponding to the virtual log file are determined. The target log file set is a set of physical log files in the target data storage system, and at least two virtual log files correspond to the same target log file set. One target log file set corresponds to one file handle.

[0013] The target log file set is opened using the file handle of the target log file set, and the multiple log data processing requests are executed according to the log data processing content, the virtual log file, and the target log file set corresponding to the virtual log file.

[0014] According to a third aspect of the present disclosure, a data storage system is provided, comprising:

[0015] The receiving unit is used to receive multiple data processing requests, wherein each of the multiple data processing requests carries a corresponding virtual file identifier and data processing content;

[0016] The determining unit is configured to determine, based on the virtual file identifier, the virtual file corresponding to the data processing request and the target file set corresponding to the virtual file, wherein the target file set is a set of physical files in the target data storage system, and at least two virtual files in the multiple virtual files correspond to the same target file set, and one target file set corresponds to one file handle;

[0017] An execution unit is configured to open the target file set using the file handles of the target file set, and execute the multiple data processing requests according to the data processing content, the virtual file, and the target file set corresponding to the virtual file.

[0018] According to a fourth aspect of the present disclosure, a data processing apparatus is provided, comprising:

[0019] The receiving module is configured to receive multiple data processing requests, wherein each of the multiple data processing requests carries a corresponding virtual file identifier and data processing content;

[0020] The determination module is configured to determine, based on the virtual file identifier, the virtual file corresponding to the data processing request and the target file set corresponding to the virtual file, wherein the target file set includes at least one physical file in the target data storage system, and at least two virtual files in the multiple virtual files correspond to the same target file set, and one target file set corresponds to one file handle;

[0021] The execution module is configured to open the target file set using the file handle of the target file set, and execute the multiple data processing requests according to the data processing content, the virtual file, and the target file set corresponding to the virtual file.

[0022] According to a fifth aspect of the present disclosure, a computing device is provided, comprising:

[0023] Memory and processor;

[0024] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0025] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0026] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0027] One embodiment of this disclosure provides a data processing method that, upon receiving multiple data processing requests for multiple virtual files, determines the corresponding multiple virtual files and the target file set corresponding to the multiple virtual files. When at least two virtual files can correspond to the same target file set, data processing for multiple virtual files corresponding to the same target file set can be determined as data processing for physical files within that target file set. For example, if the data processing request is a data write request, the write data for the multiple virtual files corresponding to the same target file set can be written to the physical files of that target file set. That is, when opening a physical file in a file system for writing, a handle is obtained for actual data processing. Writing is straightforward. By merging writes, when a large number of virtual files are written simultaneously, only a small number (or even just one) of physical files are actually written to the file system. The file system does not need to allocate handle resources for each of the large number of write streams, reducing resource consumption while supporting more concurrent writes. It also reduces the number of physical files in the file system, increasing the upper limit of the number of files the entire system can accommodate. Correspondingly, when the data processing request is a data read request, the reads for multiple virtual files are merged into the reads for a small number (or even just one) of physical files. This allows only a small number (or even just one) of physical files to be opened in the file system, without having to allocate handle resources for each of the large number of read streams. This reduces resource consumption while supporting more concurrent reads. Attached Figure Description

[0028] Figure 1 is a schematic diagram of a data processing method provided in an embodiment of this disclosure;

[0029] Figure 2 is a flowchart of a data processing method provided in an embodiment of this disclosure;

[0030] Figure 3a is a schematic flowchart of a data processing method for creating files according to an embodiment of this disclosure;

[0031] Figure 3b is a schematic flowchart of a data writing method provided in an embodiment of this disclosure;

[0032] Figure 3c is a schematic diagram of the structure of a plug-in provided in an embodiment of this disclosure;

[0033] Figure 3d is a schematic flowchart of a data processing method for reading data provided in an embodiment of this disclosure;

[0034] Figure 4 is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of the present disclosure;

[0035] Figure 5 is a structural block diagram of a computing device provided in an embodiment of this disclosure. Detailed Implementation

[0036] Numerous specific details are set forth in the following description to provide a full understanding of this disclosure. However, this disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this disclosure. Therefore, this disclosure is not limited to the specific implementations disclosed below.

[0037] The terminology used in one or more embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this disclosure. The singular forms “a,” “the,” and “the” as used in one or more embodiments of this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this disclosure refers to and includes any or all possible combinations of one or more associated listed items.

[0038] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this disclosure, and similarly, second may also be referred to as first. Depending on the context, the word “if” as used herein may be interpreted as “when”, “in response to a determination”, or “when…”.

[0039] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this disclosure are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0040] First, the terms and concepts involved in one or more embodiments of this disclosure will be explained.

[0041] File system: A file system provides a user interface through which users can manipulate a directory tree, create and delete directories, and create, delete, write, and read files. The underlying file system may be a local file system (i.e., a local disk and its associated file interface); a remote file system; a distributed file system, such as HDFS; or some other form.

[0042] Handle: When opening a file in a file system for reading or writing, the file must first be opened to obtain a handle. The file is then accessed through this handle to perform the actual read / write operations. This handle maps to the file currently opened by the user and stores corresponding metadata information such as operation permissions. The handle persists until the user closes the file. Generally, a handle allocates certain resources for this read / write operation, such as memory, threads, and network resources. These resources are necessary for accessing and writing files, and the specific amount depends on the file system implementation. The upper limit of handle resources is the upper limit of the number of files that the entire system can handle for concurrent read / write operations.

[0043] Atomic writes refer to files where, after a file write stream is opened, the file is written incrementally without immediately responding to user read requests. The file becomes visible and readable only after the user closes the write stream. If a failure occurs during the write process, the file becomes unusable. Because atomic writes do not require immediate reading, background operations such as data rearrangement can be performed during the write process. Therefore, they are more compatible with advanced features requiring data rearrangement, such as EC (Erasure Coding; EC is a redundancy protection mechanism that achieves data redundancy protection by calculating parity slices), RAID (Redundant Array of Independent Disks; RAID technology combines multiple hard drives into a larger, more secure disk array to improve read / write speeds and data redundancy), and compression.

[0044] Incremental writes refer to a write method where the user can call `flush` (a flush operation that typically flushes data from the cache to disk) at any time during the write process, making the written portion immediately visible to the reader. Because this method requires high real-time performance, rearranging the data structure in the background is more difficult, often necessitating the faithful execution of the user-written data into the actual background storage. Therefore, it has fewer compatible advanced features compared to atomic writes.

[0045] Virtual files and real files: A real file refers to a file in the file system, while a virtual file refers to a file created by the user. Due to merging, data from multiple virtual files may actually reside in the same real file. In the following description, the read and write streams corresponding to virtual files are called virtual read and write streams; the read and write streams corresponding to real files are called real read and write streams.

[0046] Atomicity data encapsulation: An atomic data encapsulation refers to the encapsulation of a piece of user-written data. After encapsulation, this data can either be fully read by the user or be completely invisible to the user. There is no case where the beginning of the data can be read.

[0047] Real file set: refers to a collection of real files. The content of the same virtual file is located in a real file set and may be located in any one or more real files in that real file set.

[0048] This disclosure provides a data processing method, and also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0049] Referring to Figure 1, Figure 1 shows a scenario diagram of a data processing method provided according to an embodiment of the present disclosure.

[0050] The data processing method provided in this disclosure is applied to a read / write plugin. Specifically, in the prior art, users directly read and write to real files in the file system. In this case, opening a real file requires a file handle. The data processing method provided in this disclosure allows users to interact with the read / write plugin. Files written or read by the plugin are virtual files. These virtual files do not correspond to real files in the file system but are simulated by the plugin. This allows multiple virtual files to correspond to the same real file, so when processing data on multiple virtual files, the user only needs to open one real file in the file system. Users can perform read / write operations on virtual files as if they were reading and writing real files; that is, the process of virtual file conversion and interaction with real files is transparent to the user.

[0051] The read / write plugin uses the data processing method provided in this embodiment to receive multiple data processing requests sent by the user. These multiple data processing requests are all data processing requests for virtual files. For example, the multiple data processing requests are data write requests to write data to multiple virtual files, or data read requests to read data from multiple virtual files. Of course, after being processed by the read / write plugin, these data processing requests from the user will all be implemented as processing of files in the file system.

[0052] In practical applications, one virtual file corresponds to one set of real files, and one set of real files corresponds to one or more virtual files. When the read / write plugin receives multiple data processing requests, it determines the virtual file identifiers carried in the multiple data processing requests, thereby identifying the corresponding multiple virtual files, and returns virtual write streams for multiple virtual files so that users can write files through the virtual write streams. When multiple virtual files can correspond to the same set of real files (i.e., the target file set), the multiple virtual write streams corresponding to the same set of real files will be merged into a single file system real file set write stream. That is, the write data of the multiple virtual files is actually written to only one real file in the file system at the same time.

[0053] By mapping multiple virtual files to a single set of real files, operations on multiple virtual files can be merged into a small number of operations, or even operations on a single real file. This eliminates the need for the file system to allocate handles and other resources separately for a large number of concurrent read and write streams, reducing resource consumption while supporting more concurrent reads and writes.

[0054] In Figure 1, the third-party database or other storage medium serves as an auxiliary medium for storing metadata that may require fast access. Examples include the path to a virtual file and its mapping to a set of real files; for a set of real files, the data of which virtual files correspond to each real file, and the offsets of this data relative to the virtual files. This metadata can also be stored in a file system, for example, by creating one or more files within the file system to store the metadata. However, file systems generally have poor random access performance. Therefore, storing frequently and quickly accessed metadata in a third-party database or other storage medium optimized for random read / write operations offers better query and update performance compared to storing it in files.

[0055] The data processing method for read / write plugins provided in this disclosure allows multiple virtual files to correspond to the same real file set. Based on this, the data processing for multiple virtual files can be merged into a small number, or even a single, processing for real files in the real file set. This allows users to write and read a large number of files simultaneously, but only write and read a small number (or one) of files in the file system. The file system does not need to allocate handles and other resources for a large number of write streams and read streams, reducing resource consumption while supporting more concurrent writes and reads.

[0056] Referring to Figure 2, which shows a flowchart of a data processing method provided in an embodiment of the present disclosure, the method specifically includes the following steps.

[0057] Step 202: Receive multiple data processing requests, wherein each of the multiple data processing requests carries a corresponding virtual file identifier and data processing content.

[0058] The virtual file identifier can be understood as an identifier used to uniquely identify a specific virtual file. This identifier can be a file name, file path, or any other mark that can uniquely identify a virtual file, without any restrictions here.

[0059] The data processing content can be understood as the specific operations to be performed on the virtual file as specified in the data processing request. The data processing content varies depending on the data processing request. For example, when the data processing request is a data write request, the data processing content can be the data to be written. When the data processing request is a data read request, the data processing content can be the data information of the data to be read.

[0060] Specifically, before receiving multiple data processing requests for virtual files, multiple virtual files are created, and corresponding real file sets are assigned to each of the created virtual files, thereby establishing a correspondence between virtual files and real file sets. The specific implementation method is as follows:

[0061] Before determining the virtual file corresponding to the data processing request and the target file set corresponding to the virtual file based on the virtual file identifier, the method further includes:

[0062] Receive the creation request of the virtual file, wherein the creation request carries the file path of the virtual file;

[0063] Based on the file path, create the virtual file and assign a corresponding set of target files to the virtual file;

[0064] Write the correspondence between the virtual file and the target file set corresponding to the virtual file into the target storage location, and determine the corresponding virtual file identifier for the virtual file.

[0065] The target file set can be understood as the real file set. One real file set corresponds to one or more virtual files, and one real file set consists of one or more real files.

[0066] Specifically, for ease of description, we will take the creation of a virtual file as an example for detailed explanation. The creation process of each virtual file can be referred to in the following description. A virtual file system is established on the read / write plugin in the above embodiment.

[0067] In practice, users can send a virtual file creation request to the read / write plugin. This request includes the file path of the virtual file and namespace information. Alternatively, if the user doesn't specify a namespace, a corresponding set of real files can be assigned to the virtual file from the default namespace. A namespace is a logical unit used to organize and manage real file sets. Each namespace can contain multiple real file sets, and sets of the same type can be organized within a single namespace. Each virtual file is mapped to a unique namespace, and the real file sets within each namespace are independent. The threads and memory resources corresponding to each real file set are also independent, thus allowing namespaces to isolate the resources of different virtual files.

[0068] When the read / write plugin receives a creation request, it creates a corresponding virtual file on the file path corresponding to the virtual file system, and allocates a corresponding real file set for the created virtual file in the namespace corresponding to the namespace information. Of course, the allocated real file set is the active real file set, that is, there are active real files that can be appended to in the active real file set, so as to ensure that the write data for the virtual file can be written to the real files in the active real file set.

[0069] Specifically, when creating a new virtual file, an active set of real files can be allocated to the new virtual file using an allocation strategy (such as round-robin, random, hash, etc.).

[0070] The mapping relationship between virtual files and their corresponding real file sets is written as metadata to the target storage location, and a virtual handle is returned to the user. This virtual handle is used for the user to access and manipulate the virtual files. Subsequent access or manipulation of virtual files will require the use of this virtual handle to access the corresponding virtual file. Therefore, this virtual handle can also be understood as a virtual file identifier for the virtual file. When this virtual handle references a real file set, opening a virtual file through this virtual handle will open the real file in the corresponding real file set, thus converting operations on virtual files into operations on real files. The target storage location can be one or more files in the file system, or a third-party database or other storage media, without limitation.

[0071] The data processing method provided in this disclosure creates virtual files and assigns corresponding real file sets to them, enabling multiple virtual files to correspond to the same real file set. This allows subsequent processing of multiple virtual files to be converted into processing of the same real file set, thus merging multiple virtual files.

[0072] Step 204: Based on the virtual file identifier, determine the virtual file corresponding to the data processing request and the target file set corresponding to the virtual file, wherein the target file set is a set of physical files in the target data storage system, and at least two virtual files in the multiple virtual files correspond to the same target file set, and one target file set corresponds to one file handle.

[0073] The target data storage system can be understood as a file system, and the physical file can be understood as a file in the file system.

[0074] Specifically, the corresponding virtual file is determined based on the virtual file identifier. Since the mapping relationship between the virtual file and the corresponding real file set is recorded when the virtual file is created, the real file set corresponding to the virtual file can be determined through the mapping relationship based on the determination of the virtual file.

[0075] In practical applications, when there are at least two virtual files corresponding to the same target file set in the virtual files corresponding to multiple data processing requests, it is possible to merge multiple virtual files of the same target file set. That is, the processing of multiple virtual files of the same target file set corresponds to the processing of the same target file set, thereby reducing the occupation of handle resources.

[0076] Step 206: Open the target file set using the file handle of the target file set, and execute the multiple data processing requests according to the data processing content, the virtual file, and the target file set corresponding to each virtual file.

[0077] Specifically, the process of executing multiple data processing requests differs depending on the data processing request. This will be explained in detail using data write requests and data read requests as examples.

[0078] In one or more embodiments of this disclosure, the plurality of data processing requests are plurality of data write requests, and the data processing content is data to be written; the data to be written corresponding to the plurality of virtual files is written to the target buffer of the target file set corresponding to the virtual files, thereby writing the data in the target buffer to the corresponding target file set. Specific implementation methods are as follows:

[0079] The step of executing the multiple data processing requests based on the data processing content, the virtual file, and the set of target files corresponding to the virtual file includes:

[0080] Based on the correspondence between the virtual file and the data processing request, and the association between the data processing request and the data to be written, the data to be written corresponding to the virtual file is obtained;

[0081] The data to be written corresponding to the virtual file is encapsulated to obtain the data encapsulation package corresponding to the virtual file;

[0082] Write the data package corresponding to the virtual file into the target buffer corresponding to the target file set corresponding to the virtual file;

[0083] Under the condition that the preset writing conditions are met, the data package to be written to the target buffer is written to the target file set corresponding to the target buffer.

[0084] The preset write conditions can be, for example, reaching a preset interval time (which can be set according to actual conditions), the amount of data in the target buffer reaching a preset threshold (which can be set according to actual conditions), or the write thread detecting that there is a newly written data package in the target buffer. No restrictions are imposed here.

[0085] The target buffer can be understood as a temporary storage area used to temporarily store data packages to be written into the target file set.

[0086] Specifically, a data processing request carries a corresponding virtual file identifier and data to be written. The virtual file identifier in the data processing request is used to determine the processing object of the data processing request, that is, to process the virtual file corresponding to the virtual file identifier. The data to be written is used to determine the specific processing content of the data processing request, that is, to write the data to be written to the virtual file corresponding to the data processing request.

[0087] For ease of description, we will still use the execution process of a single data processing request as an example for detailed explanation. The execution process of each data processing request is the same, that is, the execution process of each data processing request can be referred to in the following description.

[0088] In practice, when a data processing request carries a corresponding virtual file identifier and the data to be written, the virtual file corresponding to the data processing request and the data to be written corresponding to the virtual file can be determined.

[0089] For example, when a user sends a data write request for virtual file A to the read / write plugin, the data write request carries the virtual file identifier of virtual file A and the data to be written to virtual file A. The read plugin returns virtual write stream 1 of virtual file A to the user, and the user can write the corresponding data to be written to virtual file A through virtual write stream 1 returned by the read plugin.

[0090] For each set of real files in the file system, the read plugin maintains a corresponding target buffer to buffer the data to be written to one or more virtual files corresponding to that set of real files. Each time a user writes data via a virtual write stream, this data is encapsulated into a data packet and placed in the target buffer of the corresponding set of real files. This data packet can be an atomic data packet, thus being read by the user as a whole. The atomic data packet includes a header containing information such as the data length, the virtual file to which the data belongs, the offset of the data within the virtual file, and a checksum (e.g., CRC). The reader can use the information in the header to determine whether the data packet in the file system is complete. If it is incomplete, it may indicate an error during the writing process or that writing is in progress. In this case, the data packet should be discarded during reading, rather than only retrieving the first part of the valid data, otherwise, atomicity semantics would not be met. Of course, the header of the atomic data packet can also store user-defined data, including but not limited to structured and unstructured data (such as key-value data). Furthermore, the header of the atomic data packet is not limited to a single header format; it does not necessarily have to be completely placed at the beginning of the data packet, as long as it can store the corresponding information and be read.

[0091] Taking the preset write condition that the write thread listens to the target buffer for a newly written data package as an example, the write thread of the target file set can listen to the target buffer of the target file set. When it listens to the target buffer of the target file set and finds that there is an unwritten data package, it writes the unwritten data package to the corresponding target file set.

[0092] When multiple virtual files can correspond to a set of target files, the data packages of multiple virtual files are placed into the target buffer of the same set of target files, so that the writing thread can write the data packages from multiple virtual files in the target buffer to the same set of target files.

[0093] The data processing method provided in this embodiment includes a target buffer for a target file set, which can buffer multiple data packages from multiple virtual files corresponding to the target file set. This allows the data packages from multiple virtual files in the target buffer to be written into the same target file set, thereby merging the virtual files.

[0094] In one or more embodiments of this disclosure, the physical files in the target file set include open and closed states. When writing a data encapsulation package to the target file set, the data encapsulation package is written to the target files in the target file set that are in the open state.

[0095] Under the condition that the preset write conditions are met, the data package to be written to the target buffer is written to the target file set corresponding to the target buffer, including:

[0096] The physical files that are in the open state in the target file set are identified as target files, wherein the open state is used to indicate that the physical file has been opened and data can be written, and the file handle corresponding to the target file set is the file handle corresponding to the target file;

[0097] Under the condition that the preset writing conditions are met, the data of the data package in the target buffer is written to the target file in the target file set corresponding to the target buffer using the writing thread corresponding to the target file set.

[0098] The "on" state can be understood as an active, append-only write state, meaning that newly written data can be written to the active, append-only write target file. Since there is one target file in the target file set, the file handle corresponding to the target file is the file handle corresponding to the target file set.

[0099] In practical applications, the most recently created real file in the real file set (usually the last real file in the real file set) is the target file, while other real files are closed and cannot be modified. During writing, the writing thread corresponding to the target file set writes the data packaged in the target buffer into the target file in the corresponding target file set.

[0100] Therefore, multiple virtual write streams of the same real file set will be merged into a real file set write stream of the file system. This real file set write stream is actually the target file write stream. That is, the data to be written to multiple virtual files corresponding to the same real file set is actually written to the same target file in the file system at the same time, so that opening a target file and obtaining a file handle is sufficient.

[0101] The data processing method provided in this disclosure has an active, append-writeable target file in the target file set. Therefore, the data to be written to multiple virtual files corresponding to the same real file set can be written to the same target file in the file system at the same time. That is, a target file is opened in the file system and a handle is obtained, which reduces the occupation of handle resources.

[0102] In one or more embodiments of this disclosure, when a target file meets preset switching conditions, a new target file can be created in the corresponding target file set, and the open state of the target file can be updated to a closed state. Furthermore, the closed file can be processed. Specific implementation methods are as follows:

[0103] If the target file meets the preset switching conditions, a new target file is created in the target file set corresponding to the target file, and the open state of the target file is updated to the closed state to obtain the physical file in the closed state. The preset switching conditions are that the amount of data in the target file is greater than a preset threshold, or the creation time of the target file exceeds a preset time. The closed state is used to indicate that the target file is no longer allowed to write data.

[0104] If a physical file in the closed state meets a preset deletion condition, delete the file data in any one of the closed physical files.

[0105] The preset switching conditions can be that the amount of data in the target file is greater than a preset threshold, the creation time of the target file exceeds a preset time, or other user-defined conditions, which are not limited here.

[0106] Specifically, since multiple data processing requests correspond to multiple virtual files and multiple target file sets, for any target file in the multiple target file sets, if the preset switching conditions are met, a new target file is created in the target file set corresponding to that target file, and the open status of that target file is updated to the closed status, that is, no data can be appended to that target file.

[0107] At this point, if there are virtual files being written to the real file set, it is not possible to directly switch to other real file sets because one virtual file maps to one real file set. When writing data to a virtual file, the data is written to a new target file. When the number of real files in the real file set is greater than 1, the corresponding real file set has dedicated metadata to store the virtual file corresponding to each real file, as well as information such as the data offset and total data length of each virtual file, so that when reading, it can locate which real file in the real file set the data of the required virtual file is located.

[0108] Similarly, when certain conditions are met (such as the number of virtual files in the current real file set reaching a specified size, or a specified time elapsed since the creation of the current real file set), the current real file set should be prohibited from continuing to accommodate new virtual files. Newly created virtual files should be mapped to another newly created real file set. In this way, the current real file set will not expand indefinitely, but will enter an unmodifiable state as all its virtual files are closed and written.

[0109] By creating a new target file and updating the open state of the previous target file to the closed state, the problem of excessively large files caused by a single target file being constantly open and appended with data is avoided. Furthermore, garbage collection can be performed on physical files in the closed state, and the garbage collection process is transparent to the reader. In contrast, the latest target file being written to in the real file set cannot be garbage collected because it is still being dynamically modified.

[0110] The preset deletion conditions can include conditions such as the proportion of junk data in the data stored in the closed physical file being greater than a certain value, the size of the junk data being greater than a certain value, the interval between the last garbage collection exceeding a certain time, or other user-defined garbage collection trigger conditions. Junk data can be understood as the data of the deleted virtual file when the user deletes the virtual file.

[0111] Specifically, since no more data is appended to a closed physical file, garbage collection can be performed on the data within it, that is, garbage data is deleted from the closed physical file.

[0112] The data processing method provided in this disclosure can rearrange the data in a closed physical file by creating a new target file and updating the open state of the previous target file to a closed state. This can improve writing efficiency, optimize storage space utilization, improve data access performance, and enhance data quality and consistency. Furthermore, by performing garbage collection on the data in the closed physical file, resource consumption can be reduced.

[0113] In one or more embodiments of this disclosure, when an error occurs in the write stream of a target file currently being written to in the real file set, the error handling described above can be achieved by creating a new target file and writing the new data to the new target file. Specific implementation methods are as follows:

[0114] The step of using the writing thread corresponding to the target file set to write the data packaged in the target buffer to the target file in the target file set corresponding to the target buffer includes:

[0115] Using the writing thread corresponding to the target file in the target file set, the data package to be written to the target buffer is written to the target file in the target file set corresponding to the target buffer;

[0116] If the write thread corresponding to a target file in the target file set encounters an error, a new target file is created in the target file set. The data package corresponding to the target buffer in the target file set and the new target file is written to the new target file using the write thread corresponding to the new target file.

[0117] Specifically, for each real file set write stream in the file system, there is one write thread, one or more flush threads, and one notification thread in the background. When the real file set write stream writes the data package in the target buffer to the target file in the real file set, the real file set write stream is actually the target file write stream. It uses the write thread corresponding to the target file in each target file set to write the data package written to each target buffer to the target file in the target file set corresponding to each target buffer.

[0118] If the write thread corresponding to a target file in the target file set encounters an error (when the target file write stream encounters an error), a new target file needs to be created in the target file set. Then, the write thread corresponding to the new target file is used to write the newly written data into the new target file. At this time, the target file write stream is the write stream that writes the newly written data in the target buffer to the new target file.

[0119] The data processing method provided in this disclosure can continue data writing by creating a new target file when an error occurs in the target file write stream during the data writing process, thus avoiding interruption of the entire data processing flow and ensuring data continuity.

[0120] In one or more embodiments of this disclosure, if the data package is successfully written to the target file set of the file system, the data package successfully written to the file system can be further flushed to persistent storage medium. Specific implementation methods are as follows:

[0121] After writing the data package to each target buffer into the target file set corresponding to each target buffer, under the condition that the preset writing conditions are met, the method further includes:

[0122] If the data package written to the target buffer is successfully written to the target file set corresponding to the target buffer, the flashing thread calls the flashing interface to store the data package successfully written to the target file set to persistent storage medium.

[0123] The flashing thread is the thread corresponding to the target file set, and the flashing interface is the interface provided by the target data storage system that corresponds to the target file set.

[0124] The flash interface can be understood as the flush interface, which is used to flash data in the file system into the corresponding persistent storage medium. The persistent storage medium can be a disk, optical disk, network attached storage, etc., and there are no restrictions here.

[0125] Taking a set of target files as an example, the above steps will be described in detail.

[0126] The thread corresponding to the target file set is actually the thread that writes the real file set to the target file set. Therefore, the flushing thread of the real file set write stream uses the flushing interface of the target file set provided by the file system to package the data successfully written to the target file set and store it in the persistent storage medium.

[0127] In one or more embodiments of this disclosure, the data package written to the target buffer and the set of target files that have been successfully written to the target buffer are determined in the following manner:

[0128] The step of successfully writing the data package to the target buffer into the target file set corresponding to the target buffer includes:

[0129] Determine the amount of data to be written in the data encapsulation packet written to the target buffer;

[0130] Based on the amount of data to be written in the data encapsulation packet, update the written points corresponding to the target file set corresponding to the target buffer;

[0131] Upon completion of the update, it is confirmed that the data package to be written to the target buffer has been successfully written to the target file set corresponding to the target buffer.

[0132] The written points are used to record the amount of data to be written to the target file set, and are updated incrementally according to the amount of data to be written to the target file set successfully.

[0133] The written point can be understood as a counter or pointer used to record the amount of data that has been successfully written to the target file set. That is, the point will increment every time a data packet is successfully written.

[0134] Specifically, the write thread of the real file set write stream monitors the target buffer of the corresponding real file set. When it finds that there is an unwritten atomic data package in the target buffer, it performs the following operations:

[0135] Retrieve the data package from the target buffer and write its contents to the target file. If the write is successful, update the write position based on the amount of data to be written in the data package and increment the write position by the amount of data in the data package. After updating the write position, it can be assumed that the data package has been successfully written to the target file set (here, "success" means that the data has been written to the actual file. Depending on the file system implementation, after appending, the data may have already fallen into the persistent storage medium or may have only been placed in the file system buffer. Therefore, appending does not guarantee that the data has been persisted and can be read by the user).

[0136] Therefore, to store the successfully written data package in persistent storage, the flashing thread calls the flashing interface to store the data package successfully written to the target file set in persistent storage. The specific implementation is as follows:

[0137] The step of using the flashing thread to call the flashing interface, encapsulating the data successfully written to the target file set, and storing it in a persistent storage medium includes:

[0138] The target file set is used to monitor the written points and triggered flash points, and the triggered flash points are used to record the amount of data to be written that has been triggered to be flashed to the persistent storage medium.

[0139] Based on the written points and the triggered flashing points, if it is determined that the data package successfully written to the target file set has not been stored in the persistent storage medium, the flashing interface is called to store the data package successfully written to the target file set in the persistent storage medium.

[0140] The flush thread of the real file set write stream monitors the written point and the triggered flush point of the real file set write stream. When it finds that the written point is greater than the triggered flush point, it updates the triggered flush point to the written point through atomic update (such as CAS technology, i.e., Compare And Swap). After the update of the triggered flush point is successful, it calls the flush interface of the real file set write stream. When the flush interface returns successfully, it updates the flush point to the previously updated triggered flush point.

[0141] For example, if the write point is 50 (meaning that 50 bytes of data have been successfully written to the real file set) and the trigger flush point is 30 (meaning that only 30 bytes of data have been flushed to the persistent storage medium), then the trigger flush point needs to be updated to 50, and the flush interface of the real file set write stream needs to be called to flush 30 to 50 bytes of data to the persistent storage medium. If the flush is successful, then the flush point is updated to 50.

[0142] In practical applications, after a successful flash, the flashing thread can build index information in memory based on the header information of the atomic data package. This index information consists of a set of information such as the virtual file corresponding to each atomic data package, the data offset of the virtual file, and the data offset of the real file.

[0143] When the target file's open state is updated to closed state, if the corresponding index data exists in memory, the corresponding index data will also be persisted. This persistence location can be another third-party database, a file in the real file system specifically used to store metadata, or other readable forms, which are not limited here.

[0144] Of course, in addition to the index data mentioned above, other indexes should be created before the index data is created to speed up reading. Such indexes include, but are not limited to, the index created when the virtual file is closed, the memory index updated in real time, etc. These indexes are called front indexes. When the above index data is persisted, since there is already complete index data in memory, it should be persisted using a structured method such as sorting so that it can be quickly located when reading.

[0145] Furthermore, this disclosure provides an index reconstruction mechanism. If the writing process encounters an unexpected event and fails to build an index for a real file, the index can be rebuilt for the real file after ensuring that the real file will no longer be updated.

[0146] It should be noted that, to ensure the correctness of the written data, when an error occurs in the currently writing real file stream, a new target file is created, and the newly written data is written to the new target file. The written and flushed points are checked; data after the written point and before the flushed point is considered unwritten, so that this data can be rewritten later. All threads are paused, and the triggered flushed point and the written point are rolled back to the flushed point. Then, a new real file is created as the target file, and all threads are resumed. After resumption, all threads will re-check that the written point is the same as the previous flushed point, and therefore will rewrite the previously written but unflushed data.

[0147] However, the above steps may cause some new target file header data to appear in the previous real file. This is because although the points that have been flashed may not have been updated to the specified location, they may actually have been flashed, but the system itself is unaware of this and treats them as not having been flashed. To avoid this problem, in the metadata and index data, for duplicate data content at the junction, the data in the next target file shall prevail.

[0148] The data processing method provided in this disclosure ensures the successful writing of data to be written and its successful storage to the persistent storage medium through mechanisms such as writing threads, flushing threads, written points, and triggered flushing points.

[0149] In one or more embodiments of this disclosure, when writing data from the data package to the target file, the data package is not removed from the target buffer corresponding to the target file set in the read / write plugin. Instead, the data package is removed from the target buffer after the notification thread calls the callback function of the data package. Specific implementation details are as follows:

[0150] After storing the data package successfully written to each of the target file sets into a persistent storage medium, the process further includes:

[0151] Based on the callback function corresponding to the data package that is persistently stored in the persistent storage medium, delete the data package in the target buffer corresponding to the target file set.

[0152] The notification thread corresponding to the real file set write stream will monitor the flushed point and the notified point. When the notified point is less than the flushed point, it will calculate the atomic data packages that have been persisted but have not yet been notified in the target buffer corresponding to the real file set write stream based on the notified point and the flushed point. Calling the callback function of these atomic data packages is considered successful, and then these atomic data packages are deleted from the target buffer corresponding to the real file set write stream.

[0153] The data processing method provided in this disclosure embodiment has a target buffer in the read / write plugin for the real file set write stream. This buffer is generally placed in memory because the background is constantly flushing data from the target buffer to the file system. Therefore, the data package in the target buffer will be deleted in time after the notification callback to release memory space.

[0154] In one or more embodiments of this disclosure, when performing garbage collection on a closed physical file, the process can be performed by creating a new target file, writing non-garbage data from the closed physical file into the new target file, and then replacing the closed physical file with the new target file. Specific implementation methods are described below:

[0155] The step of reclaiming file data from any one of the closed physical files when it is determined that the closed physical file meets the preset reclamation conditions includes:

[0156] If it is determined that the physical file in the closed state meets the preset recycling conditions, a new target file is created in the target file set corresponding to the physical file in the closed state, and the physical file in the closed state is determined as a file to be recycled;

[0157] The non-junk data in the file to be recycled is written into the new target file, and the new target file is used to replace the file to be recycled. The non-junk data is the file data in the file to be recycled excluding the junk data, and the junk data is the deleted file data.

[0158] Specifically, if a closed physical file meets the preset recycling conditions, a new target file is created in the target file set corresponding to the closed physical file. The non-garbage data in the closed physical file is written into the new target file. Then, the closed physical file is replaced by the new target file, so that the currently closed physical file includes non-garbage data but does not include garbage data.

[0159] In practice, the newly created target file is atomic; either the entire file is successfully written and replaces the closed physical file, or the file becomes unreadable. Because the newly created target file is an atomically written file, the write format can be reorganized more easily during the write process. For example, the newly created target file can use features such as EC, RAID, and compression to write, thereby better reducing costs or meeting other needs. Since the newly created target file is written atomically, there is no need for incremental write support. The storage format of the newly created target file is not limited to a file system; it can also be a third-party database or other formats, which are not limited here.

[0160] In practical applications, during garbage collection, in order to reclaim the last real file currently being written to, a new real file can be created in advance as the target file, while the last real file currently being written to is closed, so that the last real file currently being written to can be garbage collected as a file to be reclaimed.

[0161] In one or more embodiments of this disclosure, in order to fully utilize the reordering capability of atomic file writing, a copy of the data can be buffered in memory as mirror data during writing; the mirror data can be reordered before being written to the file, and can be written atomically. Specific implementation methods are described below:

[0162] The data to be written corresponding to each virtual file is written into the memory buffer and identified as mirror data;

[0163] If the mirrored data in the memory buffer exceeds a preset threshold, the mirrored data is rearranged to obtain rearranged data;

[0164] If the target file is updated to a closed state, the data to be written in the data encapsulation package written in the target file is replaced with the rearranged data.

[0165] Reordering can be understood as data rearrangement, which refers to adjusting the order of data during the data writing process to optimize performance.

[0166] Specifically, during the actual file writing process, to fully leverage the reordering capability of atomic file writes, a copy of the data can be buffered in memory as mirror data during the write operation. Unlike the data written to the actual file, the mirror data doesn't need to be immediately flushed to disk; instead, it can be accumulated to a certain size, reordered, and then written. Furthermore, because mirror data doesn't require the immediate visibility of flushes during incremental writes, it can be written atomically. Features such as EC, RAID, and compression can be used, and it can be stored not only in the file system but also in other forms such as third-party databases. When the actual file is closed, if the mirror data is persisted normally, it can replace the original data in the actual file.

[0167] In practical applications, in addition to using the aforementioned mirrored data and data created after recycling to achieve the EC, RAID, and compression functions required for atomic writing, the above capabilities can also be achieved by asynchronously reading data in the background and rewriting it to the atomic file after the real file is closed. Since the index data needs to be rearranged for easy querying after being written to disk, the index of the real file is not visible before the real file is successfully written. The system can use some pre-indexing methods to accelerate data reading.

[0168] Of course, a timely updated index can be stored in memory or other fast databases (including but not limited to high-speed databases such as Redis). Because these media sort very quickly, the index can be updated immediately after each successful atomic data encapsulation write. If the reader and writer can access this media, they can access part of the index before the real file is closed. When a virtual file is closed, the corresponding real file in the real file set can still receive writes from other virtual files and therefore may not be closed. In this case, a sorted index can be created in advance for the closed virtual file and stored in a dedicated location in the file system or a third-party database; this is not limited to these specific locations.

[0169] The data processing method provided in this disclosure, by mirroring data, can fully leverage the reordering capability of atomic file writing. By reordering data, disk fragmentation can be reduced, cache hit rate can be improved, and data consistency and integrity can be guaranteed through atomic writing.

[0170] In one or more embodiments of this disclosure, the plurality of data processing requests are plurality of data reading requests, and the data processing content is the data information of the data to be read; the corresponding virtual file and target data package information are determined by the data information of the data to be read, the corresponding real file is determined by the virtual file, and the data is retrieved from the real file according to the target data package information.

[0171] Accordingly, executing the multiple data processing requests based on the data processing content, the virtual file, and the set of target files corresponding to the virtual file includes:

[0172] Based on the data information of the data to be read, determine the target data encapsulation package information corresponding to the data to be read in the virtual file;

[0173] Determine the set of target files corresponding to the virtual file, and determine the physical file corresponding to the virtual file from the set of target files corresponding to the virtual file;

[0174] Read the target data package corresponding to the target data package information from the physical file corresponding to the virtual file;

[0175] Based on the data information of the data to be read, the data to be read is read from the target data package.

[0176] The data information to be read can be understood as the virtual file offset value and the reading length. The virtual file offset value is used to specify from which position in the virtual file to start reading data.

[0177] Specifically, when a user reads a virtual file, they need to open the corresponding virtual file through a read / write plugin. The read / write plugin will return a virtual handle to the user, who will then read the corresponding virtual file through the virtual handle. If multiple virtual files share the same set of real files, then when opening these virtual file handles simultaneously, it is actually only necessary to open the handles of one or more real files in the set of real files.

[0178] By using the virtual file offset and read length, the virtual read stream can calculate which data package(s) this data should be located in (the target data package), obtain the corresponding target data package information, determine the real file set corresponding to the virtual file, and then determine the real file corresponding to the virtual file in the real file set. Thus, the corresponding atomic data package is read from one or more real files. Then, based on the virtual file offset and read length, the atomic data package is truncated to form the data required by the user and returned.

[0179] The target data package information includes: the virtual file corresponding to the data package, the data offset corresponding to the virtual file, and the data offset of the target file corresponding to the virtual file.

[0180] Specifically, the target data encapsulation package information can be understood as the index data in the above embodiments.

[0181] Therefore, the location of the target data package within the actual file can be quickly located based on the index data. The specific implementation method is as follows:

[0182] The step of reading the target data package corresponding to the target data package information from the target file corresponding to the virtual file includes:

[0183] Based on the target data package information, determine the target file corresponding to the target data package and the data offset of the target file;

[0184] Based on the data offset of the target file, the target data package at the data offset position is read from the target file corresponding to the virtual file.

[0185] In practical applications, when the number of real files in the real file set is greater than 1, the real file set has dedicated metadata to store the virtual file corresponding to each real file, as well as the data offset and total data length of each virtual file, so that when reading, it can locate which real file in the real file set the data of the required virtual file is located.

[0186] Therefore, based on the virtual file offset value and the read length, it is possible to determine which one or more atomic data packages the data to be read is located in. By using the index data, the data offset of the atomic package in the real file can be quickly located, thereby quickly reading the corresponding atomic package from the real file and truncating the atomic package to read the corresponding data.

[0187] The data processing method provided in this disclosure can quickly locate the real file and the specific location of the data to be read in the real file by reading the length based on the virtual file offset value through various index information, thereby efficiently reading the data.

[0188] The data processing method provided in this disclosure, through merged writing, ensures that when a large number of files are written simultaneously, only one or a small number of files are actually written to the file system. The file system does not need to allocate handles and other resources for a large number of write streams, reducing resource consumption while supporting more concurrent writes. Since the actual number of files is small, when a large number of files are opened for reading simultaneously, only one or a small number of files are actually opened in the file system. The file system also does not need to allocate handles and other resources for a large number of concurrent read streams, reducing resource consumption while supporting more concurrent reads.

[0189] This disclosure provides a data processing method according to one embodiment, which can be applied to log file scenarios, including:

[0190] Receive multiple log data processing requests, wherein each of the multiple data processing requests carries a corresponding virtual log file identifier and log data processing content;

[0191] Based on the virtual log file identifier, the virtual log file corresponding to the log data processing request and the target log file set corresponding to the virtual log file are determined. The target log file set is a set of physical log files in the target data storage system, and at least two virtual log files correspond to the same target log file set. One target log file set corresponds to one file handle.

[0192] The target log file set is opened using the file handle of the target log file set, and the multiple log data processing requests are executed according to the log data processing content, the virtual log file, and the target log file set corresponding to the virtual log file.

[0193] Specifically, the log file scenario can be understood as a WAL log scenario. In this case, when the log data processing request is a log data write request, the log data processing content is the WAL log data to be written; when the log data processing request is a log data read request, the log data processing content is the data information of the WAL log to be read. See the above embodiments for details, which will not be repeated here.

[0194] The data processing method provided in this embodiment, when applied to log file scenarios, can achieve the goal of opening a separate WAL log for each shard because concurrent writing of multiple files no longer requires a large number of handle resources from the underlying file system.

[0195] Referring to Figure 3a, Figure 3a shows a schematic flowchart of the file creation process of a data processing method provided in an embodiment of the present disclosure.

[0196] The user sends a file creation request to the plugin, which carries path information and a namespace (optional; if not specified, the default namespace is used, as shown in Figure 3a. Since the user carries namespace 188, an active real file set is allocated to the virtual file from multiple active real file sets in namespace 188). Based on the path information in the file creation request, the plugin creates the virtual file on the file path corresponding to the virtual file system and determines the corresponding namespace from the virtual file system, thereby allocating an active real file set to the newly created virtual file from the corresponding namespace.

[0197] Specifically, when a new virtual file is created, a strategy (such as rotation, random, hash, etc.) can be used to select which active real file set the new virtual file should belong to, and then the virtual file is mapped to that real file set; the mapping relationship between the virtual file and the real file set is written into the metadata, and a handle is returned to the plugin, so that the plugin can access the active real file set corresponding to the virtual file based on the handle.

[0198] Referring to Figure 3b, Figure 3b shows a schematic flowchart of data writing in a data processing method provided in an embodiment of the present disclosure.

[0199] The user sends a write request to the plugin, which is used to write data to a virtual file. The plugin returns a virtual write stream for the corresponding virtual file. Specifically, the user can write data to the virtual write stream by calling the write interface. Each time the user writes a piece of data to the virtual write stream, the data will be placed as an atomic package (i.e., the atomic data package in the above embodiment) into the buffer of the corresponding real file set write stream. The atomic package includes the actual data, metadata containing the mapping relationship, and a callback function. The callback function will be called after the atomic package has been persisted to the file system, or after the write fails, and the callback will be returned to the user.

[0200] The atomic encapsulation packet described above is a single unit after being written to disk, including a header. The header contains the data length, the virtual file to which the data belongs, the data's offset within the virtual file, and a checksum. Therefore, the reader can use the information in the header to determine if the encapsulated data packet in the file system is complete. If it is incomplete, it may indicate an error during the writing process or that writing is in progress. In this case, the encapsulation packet should be discarded during reading, rather than only retrieving the first part of the valid data; otherwise, it would violate atomic semantics. Optionally, the atomic encapsulation packet header can also store user-defined data, including but not limited to structured and unstructured data (such as key-value data). The atomic encapsulation packet header is not limited to a single form; it does not necessarily have to be completely placed at the beginning of the data packet, as long as it can store the corresponding information and be read.

[0201] Referring to Figure 3c, Figure 3c shows a schematic diagram of the structure of a plug-in provided in an embodiment of the present disclosure.

[0202] Specifically, the plugin can merge multiple virtual write streams into a single real file write stream for a file system. Specifically, for each real file set write stream in the file system, there is a write thread, one or more flush threads, and a notification thread in the background. The write thread will monitor the buffer of the real file set write stream (i.e., the write buffer in Figure 3c). When there is an unwritten atomic data package in the buffer, it will be appended to the real file (the last active real file in the real file set). Each time an atomic data package is successfully written, the number of written points will be incremented by 1.

[0203] It's important to note that when the plugin writes an atomic package A to the file system, it actually calls the `write` function of the real file write stream to write the data of atomic package A to the real file and then updates the write point. At this point, atomic package A is not removed from the plugin's buffer but remains there. Subsequently, the flush thread calls the `flush` function of the real file write stream to persist atomic package A, update the flush point, notify the thread to call the callback function of atomic package A, and then remove atomic package A from the plugin's buffer. The file system provides an interface that returns a real file write stream `o` after file creation. Calling `o.write` writes data to the real file, but may only place it in the file system's buffer. Calling `o.flush` flushes the previously written data to disk, ensuring it's written to disk.

[0204] Since a set of real files corresponds to one or more virtual files, atomic data packets written by different virtual write streams in the buffer of the real file set write stream can be written to the real files of the file system by the same real file write stream.

[0205] In practical applications, a set of real files includes one or more real files. The plugin can set preset conditions (such as the data size of the real file being greater than the maximum write limit, the creation time of the real file exceeding a certain time, or other user-defined thresholds). When a real file meets the preset conditions, the status of the real file can be updated to closed, and a new active real file can be created.

[0206] Furthermore, when certain conditions are met (such as the number of virtual files in the current real file set reaching a specified size, or a specified time elapsed since the current real file set was created), the current real file set can be prevented from accommodating new virtual files. Newly created virtual files will be mapped to another newly created real file set, thus preventing the current real file set from expanding indefinitely, and instead causing it to enter an unmodifiable state once all its virtual files are closed and written to.

[0207] For example, if there are two virtual files, virtual file A is four bytes long and virtual file B is also four bytes long, and four bytes are written to virtual file A, then the written position of the corresponding real file set will be increased by four bytes; if four bytes are also written to the other virtual file B at the same time, then the written position will be increased by 4, that is, the written position will be 8.

[0208] The data has been written to the file system buffer, and the flush thread is used to process the file system buffer. The data written is eight bytes, so it is necessary to flush the eight bytes of data to the disk by calling the flush interface of the file system's real file write stream, and update the triggered flush point to 8.

[0209] The flush thread can monitor the written point and the triggered flush point. When it is determined that there is written data that has not been flushed based on the written point and the triggered flush point, the triggered flush point is updated to the written point. When the update of the triggered flush point is successful, the flush interface of the real file set write stream is called. When the flush interface returns successfully, the flush point is updated to the triggered flush point, which means that the eight bytes of data have been flushed to the disk.

[0210] The notification thread will monitor the flushed points and the notified points. When the notified points are less than the flushed points, it will retrieve the atomic wrappers that have been successfully flushed but have not yet received a notification callback from the real file set write stream buffer. Calling the callback functions of these atomic wrappers will be successful. After execution, the notified points can be updated. Then, these atomic wrappers will be deleted from the buffer corresponding to the real file set write stream of the plugin in order to free up memory space.

[0211] Referring to Figure 3d, Figure 3d shows a schematic flowchart of a data processing method for reading data according to an embodiment of the present disclosure.

[0212] When a user needs to read a virtual file created by a plugin, the user needs to open the virtual file through the plugin. The plugin obtains a virtual file handle through the virtual file system and returns a virtual file handle for reading data to the user (the handle has different functions depending on the data processing request; for example, in the case of a data write request, the handle is used for writing data, and in the case of a data write request, the handle is used for reading data). The user reads the virtual file through the virtual file handle. When reading a virtual file, the user needs to include the virtual file offset value and the read length in the read request. The mapping relationship in the metadata determines the real file set corresponding to the virtual file. Since the flashing thread can build index information in memory based on the atomic package header information, that is, a set of information consisting of the virtual file corresponding to each atomic package, the offset of the virtual file data, and the offset of the real file data.

[0213] Furthermore, when the number of real files in the real file set is greater than 1, the real file set has dedicated metadata to store the corresponding virtual file in each real file, as well as the data offset and total data length of each virtual file, so that when reading, it can locate which real file in the real file set the data of the required virtual file is located in.

[0214] Therefore, based on the offset value and read length, it is possible to determine which atomic package(s) the data to be read is located in, and to locate one or more real files in the real file set. By using index data or pre-index, the offset of the atomic package in the real file can be quickly located, the position of the corresponding atomic package in the real file can be found, and the corresponding atomic package can be quickly read from the corresponding real file. The atomic package is then truncated to read the corresponding data.

[0215] It should be noted that, apart from the actual file itself needing to rely on the incremental write feature to support flush and be immediately visible to the user, data that does not need to be written incrementally in this disclosure embodiment (including but not limited to the above-mentioned mapping, index, pre-index, garbage collection, mirror data, atomic files after asynchronous background rewriting, or other reasonable data) can be stored in the file system, other third-party databases, or other reasonable locations according to actual needs and implementation, without any restrictions.

[0216] The data processing method provided in this disclosure increases the maximum number of concurrent file writes supported by the system and reduces resource consumption when writing a large number of files. Furthermore, due to the reduction in the number of files, the metadata pressure on the file system itself is reduced. It supports progressive writing (files that are immediately visible after flushing), integrates some capabilities that rely on atomic writing, and supports the capability of atomic writing (writing a piece of data either makes it completely invisible or makes it completely visible), so that developers do not have to adapt for incomplete data left at the end.

[0217] Corresponding to the above method embodiments, this disclosure also provides a data processing apparatus embodiment. Figure 4 shows a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this disclosure. As shown in Figure 4, the apparatus includes:

[0218] The receiving module 402 is configured to receive multiple data processing requests, wherein each of the multiple data processing requests carries a corresponding virtual file identifier and data processing content;

[0219] The determination module 404 is configured to determine, based on the virtual file identifier, the virtual file corresponding to the data processing request and the target file set corresponding to the virtual file, wherein the target file set is a set of physical files in the target data storage system, and at least two virtual files in the multiple virtual files correspond to the same target file set, and one target file set corresponds to one file handle;

[0220] The execution module 406 is configured to open the target file set using the file handle of the target file set, and execute the multiple data processing requests according to the data processing content, the virtual file, and the target file set corresponding to the virtual file.

[0221] Optionally, the execution module 406 is further configured to:

[0222] Based on the correspondence between the virtual file and the data processing request, and the association between the data processing request and the data to be written, the data to be written corresponding to the virtual file is obtained;

[0223] The data to be written corresponding to the virtual file is encapsulated to obtain the data encapsulation package corresponding to the virtual file;

[0224] Write the data package corresponding to the virtual file into the target buffer corresponding to the target file set corresponding to the virtual file;

[0225] Under the condition that the preset writing conditions are met, the data package to be written to the target buffer is written to the target file set corresponding to the target buffer.

[0226] The device further includes:

[0227] The creation module is configured to receive the creation request of the virtual file, wherein the creation request carries the file path of the virtual file; create the virtual file according to the file path, and allocate a corresponding target file set for the virtual file; write the correspondence between the virtual file and the target file set corresponding to the virtual file into the target storage location, and determine a corresponding virtual file identifier for the virtual file.

[0228] Optionally, the execution module 406 is further configured to:

[0229] The physical files that are in the open state in the target file set are identified as target files, wherein the open state is used to indicate that the physical file has been opened and data can be written, and the file handle corresponding to the target file set is the file handle corresponding to the target file;

[0230] Under the condition that the preset writing conditions are met, the data of the data package in the target buffer is written to the target file in the target file set corresponding to the target buffer using the writing thread corresponding to the target file set.

[0231] The device further includes:

[0232] The flashing module is configured to, upon successfully writing the data package to the target buffer and then to the target file set corresponding to the target buffer, use the flashing thread to call the flashing interface to store the data package successfully written to the target file set to persistent storage media.

[0233] The flashing thread is the thread corresponding to the target file set, and the flashing interface is the interface provided by the target data storage system that corresponds to the target file set.

[0234] Optionally, the flashing module is further configured to:

[0235] Determine the amount of data to be written in the data encapsulation packet written to the target buffer;

[0236] Based on the amount of data to be written in the data encapsulation packet, update the written points corresponding to the target file set corresponding to the target buffer;

[0237] Upon completion of the update, it is confirmed that the data package to be written to the target buffer has been successfully written to the target file set corresponding to the target buffer.

[0238] The written points are used to record the amount of data to be written to the target file set, and are updated incrementally according to the amount of data to be written to the target file set successfully.

[0239] Optionally, the flashing module is further configured to:

[0240] The target file set is used to monitor the written points and triggered flash points, where the triggered flash points are used to record the amount of data to be written that has been flashed into the persistent storage medium.

[0241] Based on the written points and the triggered flashing points, if it is determined that the data package successfully written to the target file set has not been stored in the persistent storage medium, the flashing interface is called to store the data package successfully written to the target file set in the persistent storage medium.

[0242] The device further includes:

[0243] The deletion module is configured to delete the data package in the target buffer corresponding to the target file set based on the callback function corresponding to the data package persistently stored in the persistent storage medium.

[0244] The device further includes:

[0245] The switching module is configured to, when determining that a target file meets preset switching conditions, create a new target file in the target file set corresponding to the target file, update the open state of the target file to the closed state, and obtain a closed physical file. The preset switching conditions are that the amount of data in the target file is greater than a preset threshold, or the creation time of the target file exceeds a preset time. The closed state is used to indicate that the target file is no longer allowed to write data. When determining that the closed physical file meets preset recycling conditions, the file data in the closed physical file is recycled.

[0246] Optionally, the switching module is further configured to:

[0247] If it is determined that the physical file in the closed state meets the preset recycling conditions, a new target file is created in the target file set corresponding to the physical file in the closed state, and the physical file in the closed state is determined as a file to be recycled;

[0248] The non-junk data in the file to be recycled is written into the new target file, and the new target file is used to replace the file to be recycled. The non-junk data is the file data in the file to be recycled excluding the junk data, and the junk data is the deleted file data.

[0249] Optionally, the execution module 406 is further configured to:

[0250] Using the writing thread corresponding to the target file in the target file set, the data package to be written to the target buffer is written to the target file in the target file set corresponding to the target buffer;

[0251] If the write thread corresponding to a target file in the target file set encounters an error, a new target file is created in the target file set. The data package corresponding to the target buffer in the target file set and the new target file is written to the new target file using the write thread corresponding to the new target file.

[0252] The device further includes:

[0253] The rearrangement module is configured to write the data to be written corresponding to each virtual file into a memory buffer and determine it as mirror data; if the mirror data in the memory buffer exceeds a preset threshold, rearrange the mirror data to obtain rearranged data; if the target file is updated to a closed state, use the rearranged data to replace the data to be written in the data encapsulation package written in the target file.

[0254] Optionally, the execution module 406 is further configured to:

[0255] Based on the data information of the data to be read, determine the target data encapsulation package information corresponding to the data to be read in the virtual file;

[0256] Determine the set of target files corresponding to the virtual file, and determine the physical file corresponding to the virtual file from the set of target files corresponding to the virtual file;

[0257] Read the target data package corresponding to the target data package information from the physical file corresponding to the virtual file;

[0258] Based on the data information of the data to be read, the data to be read is read from the target data package.

[0259] Optionally, the execution module 406 is further configured to:

[0260] Based on the target data package information, determine the target file corresponding to the target data package and the data offset of the target file;

[0261] Based on the data offset of the target file, the target data package at the data offset position is read from the target file corresponding to the virtual file.

[0262] One embodiment of this disclosure provides a data processing apparatus that, upon receiving multiple data processing requests for multiple virtual files, determines the corresponding multiple virtual files and a target file set corresponding to the multiple virtual files. When at least two virtual files can correspond to the same target file set, data processing for multiple virtual files corresponding to the same target file set can be determined as data processing for physical files within that target file set. For example, if the data processing request is a data write request, the write data for multiple virtual files corresponding to the same target file set can be written to the physical files of that target file set. That is, when opening a physical file in a file system for writing, a handle is obtained for actual data writing; this is achieved through merging writes. The first method, merging, allows multiple virtual files to write data into a single file in the file system. This means that when a large number of files are written simultaneously, only one (or a few) files are actually written to the file system. The file system doesn't need to allocate handles and other resources for each of the numerous write streams, reducing resource consumption while supporting more concurrent writes. It also reduces the number of files in the file system, increasing the overall file capacity. The second method, merging, allows multiple virtual files to write data into a single file in the file system. Therefore, when multiple data processing requests are actually multiple data read requests, and a large number of files are opened simultaneously, only one (or a few) files need to be opened in the file system. The file system also doesn't need to allocate handles and other resources for each of the numerous concurrent read streams, reducing resource consumption while supporting more concurrent reads.

[0263] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0264] Corresponding to the above method embodiments, this disclosure also provides a data storage system embodiment, the system comprising:

[0265] The receiving unit is used to receive multiple data processing requests, wherein each of the multiple data processing requests carries a corresponding virtual file identifier and data processing content;

[0266] The determining unit is configured to determine, based on the virtual file identifier, the virtual file corresponding to the data processing request and the target file set corresponding to the virtual file, wherein the target file set is a set of physical files in the target data storage system, and at least two virtual files in the multiple virtual files correspond to the same target file set, and one target file set corresponds to one file handle;

[0267] An execution unit is configured to open the target file set using the file handles of the target file set, and execute the multiple data processing requests according to the data processing content, the virtual file, and the target file set corresponding to the virtual file.

[0268] The above is an illustrative scheme of a data processing system according to this embodiment. It should be noted that the technical solution of this data processing system and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing system, please refer to the description of the technical solution of the data processing method described above.

[0269] Figure 5 shows a structural block diagram of a computing device 500 according to an embodiment of the present disclosure. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0270] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0271] In one embodiment of this disclosure, the aforementioned components of the computing device 500, as well as other components not shown in FIG. 5, may also be connected to each other, for example, via a bus. It should be understood that the computing device structural block diagram shown in FIG. 5 is merely for illustrative purposes and is not intended to limit the scope of this disclosure. Those skilled in the art can add or replace other components as needed.

[0272] Computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). Computing device 500 can also be a mobile or stationary server.

[0273] The processor 520 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0274] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the data processing method embodiments.

[0275] An embodiment of this disclosure also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0276] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the data processing method embodiments.

[0277] An embodiment of this disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0278] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0279] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0280] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0281] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this disclosure are not limited to the described order of actions, because according to the embodiments of this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this disclosure.

[0282] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0283] The preferred embodiments disclosed above are merely illustrative of this disclosure. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments of this disclosure. These embodiments are selected and specifically described in this disclosure to better explain the principles and practical applications of the embodiments of this disclosure, thereby enabling those skilled in the art to better understand and utilize this disclosure. This disclosure is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Receive multiple data processing requests, wherein each of the multiple data processing requests carries a corresponding virtual file identifier and data processing content; Based on the virtual file identifier, the virtual file corresponding to the data processing request and the target file set corresponding to the virtual file are determined. The target file set is a set of physical files in the target data storage system, and at least two virtual files in the multiple virtual files correspond to the same target file set. One target file set corresponds to one file handle. The target file set is opened using the file handle of the target file set, and the multiple data processing requests are executed according to the data processing content, the virtual file, and the target file set corresponding to the virtual file.

2. The data processing method according to claim 1, wherein the plurality of data processing requests are plurality of data write requests, and the data processing content is data to be written; Accordingly, executing the multiple data processing requests based on the data processing content, the virtual file, and the target file set corresponding to the virtual file includes: Based on the correspondence between the virtual file and the data processing request, and the association between the data processing request and the data to be written, the data to be written corresponding to the virtual file is obtained; The data to be written corresponding to the virtual file is encapsulated to obtain the data encapsulation package corresponding to the virtual file; Write the data package corresponding to the virtual file into the target buffer corresponding to the target file set corresponding to the virtual file; Under the condition that the preset writing conditions are met, the data package to be written to the target buffer is written to the target file set corresponding to the target buffer.

3. The data processing method according to claim 1 or 2, before determining the virtual file corresponding to the data processing request and the target file set corresponding to the virtual file based on the virtual file identifier, further comprising: Receive the creation request of the virtual file, wherein the creation request carries the file path of the virtual file; Based on the file path, create the virtual file and assign a corresponding set of target files to the virtual file; Write the correspondence between the virtual file and the target file set corresponding to the virtual file into the target storage location, and determine the corresponding virtual file identifier for the virtual file.

4. The data processing method according to claim 2 or 3, wherein, under the condition of satisfying a preset write condition, the data package to be written to the target buffer is written to the target file set corresponding to the target buffer, comprising: The physical files that are in the open state in the target file set are identified as target files, wherein the open state is used to indicate that the physical file has been opened and data can be written, and the file handle corresponding to the target file set is the file handle corresponding to the target file; Under the condition that the preset writing conditions are met, the data of the data package in the target buffer is written to the target file in the target file set corresponding to the target buffer using the writing thread corresponding to the target file set.

5. The data processing method according to any one of claims 2-4, further comprising, after writing the data package to the target buffer into the target file set corresponding to the target buffer when the preset writing conditions are met, the method further comprises: If the data package written to the target buffer is successfully written to the target file set corresponding to the target buffer, the flashing thread calls the flashing interface to store the data package successfully written to the target file set to persistent storage medium. The flashing thread is the thread corresponding to the target file set, and the flashing interface is the interface provided by the target data storage system that corresponds to the target file set.

6. The data processing method according to claim 5, wherein successfully writing the data package written to the target buffer into the target file set corresponding to the target buffer includes: Determine the amount of data to be written in the data encapsulation packet written to the target buffer; Based on the amount of data to be written in the data encapsulation packet, update the written points corresponding to the target file set corresponding to the target buffer; Upon completion of the update, it is confirmed that the data package to be written to the target buffer has been successfully written to the target file set corresponding to the target buffer. The written points are used to record the amount of data to be written to the target file set, and are updated incrementally according to the amount of data to be written to the target file set successfully.

7. The data processing method according to claim 6, wherein the step of using the flashing thread to call the flashing interface to encapsulate the data package successfully written to the target file set and store it in a persistent storage medium includes: The target file set is used to monitor the written points and triggered flash points, where the triggered flash points are used to record the amount of data to be written that has been flashed into the persistent storage medium. Based on the written points and the triggered flashing points, if it is determined that the data package successfully written to the target file set has not been stored in the persistent storage medium, the flashing interface is called to store the data package successfully written to the target file set in the persistent storage medium.

8. The data processing method according to any one of claims 5-7, further comprising, after storing the data package successfully written to the target file set into a persistent storage medium: Based on the callback function corresponding to the data package that is persistently stored in the persistent storage medium, delete the data package in the target buffer corresponding to the target file set.

9. The data processing method according to any one of claims 4-8, further comprising: If the target file meets the preset switching conditions, a new target file is created in the target file set corresponding to the target file, and the open state of the target file is updated to the closed state to obtain the physical file in the closed state. The preset switching conditions are that the amount of data in the target file is greater than a preset threshold, or the creation time of the target file exceeds a preset time. The closed state is used to indicate that the target file is no longer allowed to write data. If the physical files in the closed state meet the preset recycling conditions, the file data in the physical files in the closed state will be recycled.

10. The data processing method according to claim 9, wherein reclaiming file data in the closed physical file when it is determined that the closed physical file meets preset reclamation conditions includes: If it is determined that the physical file in the closed state meets the preset recycling conditions, a new target file is created in the target file set corresponding to the physical file in the closed state, and the physical file in the closed state is determined as a file to be recycled; The non-junk data in the file to be recycled is written into the new target file, and the new target file is used to replace the file to be recycled. The non-junk data is the file data in the file to be recycled excluding the junk data, and the junk data is the deleted file data.

11. The data processing method according to any one of claims 4-10, wherein the step of using the writing thread corresponding to the target file set to write the data of the data package in the target buffer to the target file in the target file set corresponding to the target buffer includes: Using the writing thread corresponding to the target file in the target file set, the data package to be written to the target buffer is written to the target file in the target file set corresponding to the target buffer; If the write thread corresponding to a target file in the target file set encounters an error, a new target file is created in the target file set. The data package corresponding to the target buffer in the target file set and the new target file is written to the new target file using the write thread corresponding to the new target file.

12. The data processing method according to any one of claims 2-11, further comprising: The data to be written corresponding to the virtual file is written to the memory buffer and identified as mirror data; If the mirrored data in the memory buffer exceeds a preset threshold, the mirrored data is rearranged to obtain rearranged data; When the target file is updated to a closed state, the data to be written in the data encapsulation package written in the target file is replaced with the rearranged data.

13. The data processing method according to any one of claims 1-12, wherein the plurality of data processing requests are plurality of data reading requests, and the data processing content is the data information of the data to be read; Accordingly, executing the multiple data processing requests based on the data processing content, the virtual file, and the set of target files corresponding to the virtual file includes: Based on the data information of the data to be read, determine the target data encapsulation package information corresponding to the data to be read in the virtual file; Determine the set of target files corresponding to the virtual file, and determine the physical file corresponding to the virtual file from the set of target files corresponding to the virtual file; Read the target data package corresponding to the target data package information from the physical file corresponding to the virtual file; Based on the data information of the data to be read, the data to be read is read from the target data package.

14. The data processing method according to claim 13, wherein the target data package information includes a virtual file corresponding to the data package, a data offset corresponding to the virtual file, and a data offset of the target file corresponding to the virtual file.

15. The data processing method according to claim 14, wherein reading the target data package corresponding to the target data package information from the target file corresponding to the virtual file comprises: Based on the target data package information, determine the target file corresponding to the target data package and the data offset of the target file; Based on the data offset of the target file, the target data package at the data offset position is read from the target file corresponding to the virtual file.

16. A data processing method applied to log file scenarios, comprising: Receive multiple log data processing requests, wherein each of the multiple data processing requests carries a corresponding virtual log file identifier and log data processing content; Based on the virtual log file identifier, the virtual log file corresponding to the log data processing request and the target log file set corresponding to the virtual log file are determined. The target log file set is a set of physical log files in the target data storage system, and at least two virtual log files correspond to the same target log file set. One target log file set corresponds to one file handle. The target log file set is opened using the file handle of the target log file set, and the multiple log data processing requests are executed according to the log data processing content, the virtual log file, and the target log file set corresponding to the virtual log file.

17. A data storage system, comprising: The receiving unit is used to receive multiple data processing requests, wherein each of the multiple data processing requests carries a corresponding virtual file identifier and data processing content; The determining unit is configured to determine, based on the virtual file identifier, the virtual file corresponding to the data processing request and the target file set corresponding to the virtual file, wherein the target file set is a set of physical files in the target data storage system, and at least two virtual files in the multiple virtual files correspond to the same target file set, and one target file set corresponds to one file handle; An execution unit is configured to open the target file set using the file handles of the target file set, and execute the multiple data processing requests according to the data processing content, the virtual file, and the target file set corresponding to the virtual file.

18. A data processing apparatus, comprising: The receiving module is configured to receive multiple data processing requests, wherein each of the multiple data processing requests carries a corresponding virtual file identifier and data processing content; The determination module is configured to determine, based on the virtual file identifier, the virtual file corresponding to the data processing request and the target file set corresponding to the virtual file, wherein the target file set includes at least one physical file in the target data storage system, and at least two virtual files in the multiple virtual files correspond to the same target file set, and one target file set corresponds to one file handle; The execution module is configured to open the target file set using the file handle of the target file set, and execute the multiple data processing requests according to the data processing content, the virtual file, and the target file set corresponding to the virtual file.

19. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1 to 15.

20. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 15.

21. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 15.

Images