Data caching method and distributed caching system
By dividing columnar storage files into cache blocks by row groups or column blocks, and extending the table metadata and data access interface of the distributed caching system, the problem that cached data access does not support block access and data skipping in the existing technology is solved, realizing efficient data management and access, and improving the performance of AI large model training and big data computing and analysis.
Patent Information
- Application Number
- PCT/CN2026/090821
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-14
- Filing Date
- 2026-04-15
- Publication Date
- 2026-11-19
AI Technical Summary
Existing distributed caching acceleration technologies suffer from several drawbacks: they do not support block access and data skipping for cached data, resulting in poor random access performance; they do not support batch access of cached data files, leading to memory overflow; they do not support compression of cached data, resulting in high resource consumption; and they do not provide a direct access interface for columnar objects, which affects the efficiency of AI large-scale model training and big data computing and analysis.
The columnar storage file is divided into multiple cache data blocks by row groups or column blocks and stored in a distributed caching system. It supports data skipping and filtering, and extends the distributed caching system to provide table metadata, block storage format and data access interface. It achieves efficient data management and access through table extension client and format conversion layer.
It improves data access efficiency, reduces the risk of memory overflow, enhances the speed and performance of AI large model training and big data computing and analysis, and supports efficient access and management of massive amounts of structured data.
Smart Images

Figure CN2026090821_19112026_PF_FP_ABST
Abstract
Description
Data caching methods and distributed caching systems
[0001] Cross-reference to related applications
[0002] This disclosure is based on and claims priority to Chinese Patent Application No. 2025106213866, entitled “Data Caching Method and Distributed Caching System”, filed on May 14, 2025, the entire contents of which are incorporated herein by reference. Technical Field
[0003] This disclosure relates to the field of artificial intelligence, and more specifically, to a data caching method and a distributed caching system. Background Technology
[0004] Data and intelligence are evolving towards the convergence of data and intelligence. Data platforms can provide massive data storage and access, as well as high-quality training data, for artificial intelligence platforms. In turn, artificial intelligence platforms use the data from data platforms for data training and provide the trained algorithm models to the data platforms, thereby making the management and operation of data platforms more intelligent.
[0005] In the evolution of digital intelligence convergence, distributed caching acceleration technology is an important means to accelerate the training of large-scale artificial intelligence models and the computation and analysis of big data. Using distributed caching acceleration technology can reduce waiting time during training and improve GPU utilization. However, existing distributed caching acceleration technologies have the following problems: cached data access does not support block access and data skipping, resulting in poor random access performance; cached data files do not support batch access, and loading a large amount of cached data at once can significantly impact application memory, easily causing Out Of Memory (OOM) problems; cached data does not support compression, consuming significant storage resources (memory, SSDs), and in resource-constrained environments, the amount of cacheable data is small, the hit rate is low, and the acceleration effect is poor; and no direct access interface for columnar storage objects is provided, requiring applications to pay attention to the underlying storage format and perform format conversion, which is not friendly to AI large-scale model training frameworks. Summary of the Invention
[0006] This disclosure provides a data caching method and an electronic device.
[0007] According to one embodiment of this disclosure, a data caching method is provided, applied to a table extension client, comprising: dividing a columnar storage file into multiple cache data blocks according to column block data of the columnar storage file; caching the multiple cache data blocks into multiple storage blocks of a distributed caching system respectively; and storing the mapping relationship between the multiple cache data blocks and the multiple storage blocks into the block metadata of the corresponding storage blocks.
[0008] According to another embodiment of this disclosure, a distributed caching system is also provided, which includes a table extension client. The table extension client includes: a cache extension layer, used to obtain column block data that meets the filtering conditions and send the column block data to a format conversion layer; a format conversion layer, used to convert the column block data into batch data of a first columnar memory format object and send the batch data of the first columnar memory format object to an application programming interface (API) layer; and an API layer, used to send the batch data of the first columnar memory format object to the application layer of the distributed caching system.
[0009] According to yet another embodiment of this disclosure, a computer-readable storage medium is also provided, which stores a computer program configured to perform the steps in any of the above method embodiments when executed.
[0010] According to yet another embodiment of this disclosure, an electronic device is also provided, including a memory and a processor, the memory storing a computer program, the processor being configured to run the computer program to perform the steps in any of the above method embodiments.
[0011] According to yet another embodiment of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments. Attached Figure Description
[0012] Figure 1 is a hardware structure block diagram of the computer terminal used in the embodiments of the method disclosed herein;
[0013] Figure 2 is a schematic diagram of the network architecture of a distributed caching acceleration enhancement system according to an embodiment of the present disclosure;
[0014] Figure 3 is a schematic diagram of InodeTree table metadata expansion according to an embodiment of the present disclosure;
[0015] Figure 4 is a flowchart of a data caching method according to an embodiment of the present disclosure;
[0016] Figure 5 is a schematic diagram of a columnar storage file divided into rows according to an embodiment of the present disclosure;
[0017] Figure 6 is a schematic diagram of a storage block structure divided into rows according to an embodiment of the present disclosure;
[0018] Figure 7 is a schematic diagram of a columnar storage file divided into rows and columns according to an embodiment of the present disclosure;
[0019] Figure 8 is a schematic diagram of a storage block structure divided into columns according to an embodiment of the present disclosure;
[0020] Figure 9 is a schematic diagram of the file metadata reading format corresponding to the columnar storage file according to an embodiment of the present disclosure;
[0021] Figure 10 is a schematic diagram of the skip-read filtering process for cached column block data according to an embodiment of the present disclosure;
[0022] Figure 11 is a schematic diagram of the cache read acceleration process according to an embodiment of the present disclosure;
[0023] Figure 12 is a schematic diagram of the cache miss acceleration process according to an embodiment of the present disclosure;
[0024] Figure 13 is a schematic diagram of a cache write acceleration process according to an embodiment of the present disclosure. Detailed Implementation
[0025] The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings and examples.
[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0027] The methods and embodiments provided in this disclosure can be executed in a computer terminal or similar computing device. Taking running on a computer terminal as an example, FIG1 is a hardware structure block diagram of the computer terminal running in the method embodiments of this disclosure. As shown in FIG1, the computer terminal may include one or more (only one is shown in FIG1) processors 102 (processor 102 may include, but is not limited to, processing devices such as microprocessors MCUs or programmable logic devices FPGAs) and a memory 104 for storing data. The computer terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that the structure shown in FIG1 is only illustrative and does not limit the structure of the computer terminal. For example, the computer terminal may also include more or fewer components than shown in FIG1, or have a different configuration than shown in FIG1.
[0028] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data caching method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0029] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.
[0030] This disclosure proposes an accelerated AI training method and a big data computing and analysis method based on a distributed cache enhancement system to enhance the distributed cache acceleration scheme in the context of data-intelligence fusion.
[0031] This embodiment of the disclosure can be run on the network architecture of the distributed cache acceleration enhancement system shown in Figure 2. As shown in Figure 2, the distributed cache acceleration enhancement system includes: an application layer, a distributed cache part, and an underlying storage part.
[0032] The application layer of this distributed caching acceleration and enhancement system includes: a large model training framework or a big data computing and analysis engine. The AI model training framework in the large model training framework includes: TensorFlow, PyTorch, HuggingFace, Ray, etc.; the big data computing and analysis engine includes: Spark, Flink, OLAP, etc.
[0033] The distributed caching component of this distributed caching acceleration and enhancement system includes: existing distributed caches (e.g., Alluxio, JuiceFS, etc.), including: distributed cache cluster server and various types of clients.
[0034] The distributed cache cluster server further includes two types of nodes: Master (master node) and Worker (worker node). The Master is a metadata storage cluster responsible for managing cache metadata, including file directory tree (Inode Tree) and block metadata. The Worker is a cache data storage cluster responsible for managing cache data blocks (writing, reading, lifecycle management, etc.).
[0035] Among them, the Fuse Client, a distributed caching client of type Fuse, provides a POSIX-compliant interface that allows applications to read and write cached data in a manner similar to accessing the local file system. It can be used in sequential file access scenarios. The Fuse Client can handle both structured and unstructured data files, but does not support skipping.
[0036] As shown in Figure 2, in this embodiment of the disclosure, the distributed caching acceleration enhancement system has been extended in addition to the above settings. The specific extended functions include Arrow RecordBatch, Table Extender Client, Table Meta, and caching extension (plugin package).
[0037] The Table Extender Client is a new device added to the embodiments of this disclosure. It supports multiple deployment methods such as containers, physical processes, and cloud computing. It provides an efficient batch access interface for Arrow objects for massive structured data and supports data skipping and filtering, thus making up for the shortcomings of traditional cache Fuse Client, which does not support skipping and has poor random access.
[0038] Table Extender Client includes an Application Programming Interface (API) layer, a format conversion layer, and a cache extension layer.
[0039] The cache extension layer is used to retrieve column block data that meets the filtering conditions and send the column block data to the format conversion layer; it is also used to extend and enhance the functionality of mainstream distributed caching products in the industry, for example, by extending the following aspects:
[0040] Table metadata extensions support table structure field information and partition information. For example, extensions support Hive and Iceberg table metadata.
[0041] This table metadata extension is based on the existing distributed cache file system directory tree (inode tree), which adds newly defined table and partition directory type iNode nodes to the directory tree.
[0042] Specifically, the existing distributed cache file system's Inode Tree contains two types of nodes: branch nodes and leaf nodes. Branch nodes are directories that organize / contain other nodes; for example, they can contain other branch nodes (i.e., subdirectories) or leaf nodes (i.e., files). Leaf nodes are files and refer to the terminal nodes in the Inode Tree. Leaf nodes do not contain other nodes. Each leaf node contains the metadata of the corresponding file, such as the filename, size, and permissions, as well as pointers to the file's content data blocks. These pointers contain the storage block's ID and its physical location on the storage device.
[0043] Figure 3 is a schematic diagram of the metadata extension of the Inode Tree table according to an embodiment of the present disclosure. As shown in Figure 3, in addition to regular branch nodes and leaf nodes, the Inode Tree can attach nodes of table and partition directory types. Specifically, the Inode Tree includes the following metadata:
[0044] 1. Table Inode metadata is used to store table structure field information and partition information. The table structure field information is a description of the structured data table, including multiple subfields. Each subfield carries at least one of the following information: table ID, table type, partition ID, partition type, etc. As shown in Figure 3, the table structure field information includes the following subfields: field 1, field 2, and partition field 1. Partition information exists in the form of directories P1 and P2, where each directory represents a partition, and the directory names follow the key-value pair naming convention.
[0045] 2. Partition Inode metadata is used to store a list of all file Inodes under a partition, including information on all files contained in each partition. The naming convention of the partition directory reflects the specific value of the partition field. The partition directory is named using the common key-value pair naming convention for table partitions: <partition field name> = <partition value> [, <partition field name> = <partition value>]; this helps to quickly locate a specific data partition.
[0046] 3. File Inode metadata, which stores information such as the ID, size, and storage path of the storage block.
[0047] In an exemplary embodiment, suppose "TableX" is a table that records log data, partitioned by day with the partition field being date. Then, the table inode metadata of "TableX" might contain a description similar to the following:
[0048] Table name: "TableX";
[0049] Fields:
[0050] Field 1: ID, type INT;
[0051] Field 2: Name, type VARCHAR;
[0052] Partition field 1: Date, type DATE;
[0053] Partitioning strategy: Partitions are based on date, with one partition per day.
[0054] Partition:
[0055] Partition P1: Date=2023-01-01 Inode metadata:
[0056] File1: Type = Parquet, Size = 10MB, Storage Path = " / data / Date=2023-01-01 / File1.parquet";
[0057] File2: Type = ORC, Size = 15MB, Storage Path = " / data / Date = 2023-01-01 / File2.orc";
[0058] Partition P2: Partition Date = 2023-02-02 Inode metadata:
[0059] File3: Type = Parquet, Size = 20MB, Storage Path = " / data / Date=2023-02-02 / File3.parquet";
[0060] File4: Type = ORC, Size = 25MB, Storage Path = " / data / Date = 2023-02-02 / File4.orc".
[0061] The above design allows the system to locate the correct partition based on the table inode metadata when processing query or data access requests, and then quickly obtain the specific file information within the partition through the partition inode metadata. This allows the system to selectively read or manipulate the required data instead of blindly traversing all data.
[0062] In this embodiment, by attaching tables and partition nodes to the directory tree, the distributed caching acceleration enhancement system can efficiently organize structured data, making it easier to query and access. To support these newly introduced concepts, the system also extends the original metadata structure to include storage table field information, partition information, and file block information, enabling efficient data management and access. Through these extensions, the system can better understand the organization of structured data, thereby quickly locating the required data blocks during query execution using metadata information, enabling data skipping and filtering, and ultimately accelerating data reading speed during AI large-scale model training and big data computing and analysis, thus improving overall performance.
[0063] Block storage format extensions, namely, support for managing and parsing columnar storage files, such as: extensions to support management and parsing commonly used columnar storage file formats in the industry, including but not limited to: Parquet, ORC;
[0064] Compression algorithm extension, which supports column-wise compression of columnar storage files, that is, extended support for column-wise compression of files / data in columnar storage file format, and supports, but is not limited to, the following compression algorithms: Snappy, GZP, Zstd;
[0065] The data access extension supports skipping through cached data using data filtering methods; that is, it extends data access filtering and enhances the random access performance of cached data skipping through data filtering techniques such as partitioning, column pruning, and predicate pushdown.
[0066] The plugin package deployment mode supports deployment on remote worker nodes (i.e., worker nodes that are not deployed with Table Extender Client but only with cache) via plugin packages.
[0067] The format conversion layer is used to convert the column block data into batch data of a first column-formatted memory object, and send the batch data of the first column-formatted memory object to the application programming interface (API) layer.
[0068] In one embodiment, the format conversion layer is also used to convert the batch data of the second columnar memory format object into a columnar storage format and send it to the storage layer for caching and finally synchronize it to the underlying storage.
[0069] The format conversion layer can convert between common columnar storage formats (Parquet, ORC, etc.) and Arrow object access formats.
[0070] The API layer is used to send batch data of the first columnar in-memory format object to the application layer of the distributed caching system, providing an Arrow RecordBatch access interface that supports multiple languages such as Python and Rust. The API layer also enables seamless integration between the Table Extender Client and the training framework.
[0071] In one embodiment, the API layer is also used to receive batch data of second columnar memory-formatted objects from the large model training framework or big data computing and analysis engine of the application layer.
[0072] This distributed caching accelerates and enhances the underlying storage components of the system, including file storage (HDFS, Ceph, NAS) and object storage (S3, OSS).
[0073] In the distributed caching acceleration enhancement system of this disclosure embodiment, Table Extender Client supports commonly used development languages (Python, Rust, Java, etc.) in the fields of AI model training and big data computing and analysis, and can be deployed in various cloud and on-premises (physical machines, Docker containers) AI large model training and big data computing and analysis environments such as K8s.
[0074] This embodiment provides a data caching method running on the aforementioned computer terminal or distributed caching acceleration enhancement system. Specifically, this method can be applied to the aforementioned Table Extender Client. Figure 4 is a flowchart of the data caching method according to an embodiment of this disclosure. As shown in Figure 4, the process includes the following steps:
[0075] Step S402: Divide the columnar storage file into multiple cache data blocks according to the column block data of the columnar storage file;
[0076] Step S404: The plurality of cached data blocks are cached into the plurality of storage blocks of the distributed caching system respectively;
[0077] Step S406: The mapping relationship between the plurality of cached data blocks and the plurality of storage blocks is stored in the block metadata of the corresponding storage block.
[0078] As shown in Figure 2, the table extension client is set up in the distributed cache system. As shown in Figure 3, the file directory tree metadata of the distributed cache system includes table metadata.
[0079] In related technologies, distributed caches are usually divided into blocks according to file size, for example, splitting into multiple blocks of 64M each (except for the last block). This kind of splitting cannot guarantee that the same batch of accessed data blocks / the same batch of record data / the same batch of column data will be placed in the same cache block for structured data. As a result, there will be cross-cache block access problems, which will make it impossible to filter and skip reads by block (column pruning, predicate pushdown, etc.).
[0080] To address this problem, this disclosure proposes an innovative method for dividing columnar storage files into cache blocks by row groups or column blocks. This ensures that data from the same row group or column block is in the same cache block file, avoiding the problem of cross-block reading.
[0081] In one embodiment, dividing the columnar storage file into multiple cache data blocks based on the column block data of the columnar storage file includes: dividing the columnar storage file into multiple cache data blocks by using one column block data as a cache data block; or, dividing the columnar storage file into multiple cache data blocks by using multiple column block data of any row group as a cache data block.
[0082] In one embodiment, caching the plurality of cached data blocks into a plurality of storage blocks in a distributed caching system includes: caching the plurality of cached data blocks into a plurality of storage blocks in a distributed caching system, wherein, except for the last cached data block in the plurality of cached data blocks, the data volume of the other cached data blocks in the plurality of cached data blocks is the same as the data volume that can be cached by the storage blocks, and the data volume that can be cached by each storage block is the same; and storing the file metadata corresponding to the columnar storage file in the last storage block.
[0083] In one embodiment, storing the file metadata corresponding to the columnar storage file in the end storage block includes: if the data size of the end cache data block is the same as the cacheable data size of the storage block, storing only the file metadata in the end storage block; or, if the data size of the end cache data block is less than the cacheable data size of the storage block, and the total data size of the end cache data block and the file metadata is less than the cacheable data size of the storage block, storing the file metadata and the end cache data block in the end storage block; or, if the data size of the end cache data block is less than the cacheable data size of the storage block, and the total data size of the end cache data block and the file metadata is greater than the cacheable data size of the storage block, storing the file metadata in the end storage block, and storing the end cache data block and padding data in the storage block preceding the end storage block.
[0084] In one embodiment, when a cached data block corresponds to a row group, the mapping relationship between the plurality of cached data blocks and the plurality of storage blocks includes: the mapping relationship between the row group number corresponding to each row group and the storage information of the corresponding storage block; the storage information of the corresponding storage block includes: the identification information of the corresponding storage block, the storage path of the corresponding storage block, the file offset, the offset within the block, and the row group data volume.
[0085] In an exemplary embodiment, FIG5 is a schematic diagram of a columnar storage file divided into rows according to an embodiment of the present disclosure. As shown in FIG5, multiple column block data (referred to as column block) of any row group are used as a cache data block. The columnar storage file is divided into multiple cache data blocks, that is, each row group data of the columnar storage file is divided into a cache data block. The resulting multiple cache data blocks are cached into multiple storage blocks of the distributed cache system to generate a distributed cache storage block file. At the same time, a mapping relationship between the row group number and the storage information of the storage block file is established in the block metadata of the storage block so as to store the mapping relationship between the cache data block and the storage block in the block metadata of the storage block.
[0086] The storage block file contains information including the identification information of the corresponding storage block, the storage path of the corresponding storage block, the file offset, the offset within the storage block, and the amount of row group data. The file offset is the offset position of the corresponding row group in the original columnar storage file, and the position offset of the corresponding row group in the storage block is 0 by default.
[0087] Except for the last line group, this splitting strategy requires that the size of the generated line group be the same as the size of the cache block.
[0088] In this embodiment, the file metadata corresponding to the columnar storage file is stored in the end storage block. The end storage block includes two types: metadata storage block and mixed storage block. The metadata storage block stores only file metadata, while the mixed storage block stores both the end row group and file metadata.
[0089] Figure 6 is a schematic diagram of a storage block structure divided by row groups according to an embodiment of the present disclosure. As shown in Figure 6, the storage block structure divided by row groups includes the following cases:
[0090] When the amount of data in the last row group is the same as the amount of data that can be cached in the storage block, the last storage block is a metadata storage block;
[0091] If the amount of data in the last line group is less than the amount of data that the storage block can cache, and the total amount of data in the last line group and the file metadata is less than the amount of data that the storage block can cache, then the last storage block is a hybrid storage block.
[0092] If the amount of data in the last line group is less than the amount of data that the storage block can cache, and the total amount of data in the last line group and the file metadata is greater than the amount of data that the storage block can cache, then the last storage block is the metadata storage block, the last line group is stored in the second to last storage block, and any part that is not within the cache size is filled with padding data, for example, 0x00.
[0093] In one embodiment, when one cached data block corresponds to one column block data, the mapping relationship between the multiple cached data blocks and the multiple storage blocks includes: the mapping relationship between the row group number, column block number, and storage information of the corresponding storage block corresponding to each column block data; the storage information of the corresponding storage block includes: the identification information of the corresponding storage block, the storage path of the corresponding storage block, the file offset, the offset within the block, and the data volume of the column block data.
[0094] In an exemplary embodiment, FIG7 is a schematic diagram of a columnar storage file divided into row and column blocks according to an embodiment of the present disclosure. As shown in FIG7, when the data volume of each column block is relatively large, a column block data (referred to as a column block) is used as a cache data block, and the columnar storage file is divided into multiple cache data blocks. That is, each column block data of the columnar storage file is divided into a cache data block, generating a distributed cache storage block file, and at the same time, a mapping relationship between the column block number, row group number and storage information of the storage block file is established in the block metadata of the storage block.
[0095] The storage block file contains information including the identifier of the corresponding storage block, the storage path of the corresponding storage block, the file offset, the offset within the storage block, and the data volume of the column block data. The file offset is the offset position of the corresponding column block in the original columnar storage file. The position offset of the corresponding column block in the storage block is 0 by default.
[0096] Except for the last column block, this splitting strategy requires that the amount of data in the generated column block be consistent with the amount of data that the storage block can cache.
[0097] In this embodiment, the file metadata corresponding to the columnar storage file is stored in the end storage block. The end storage block includes two types: metadata storage block and mixed storage block. The metadata storage block stores only file metadata, while the mixed storage block stores both the end column block of the end row group and file metadata.
[0098] Figure 8 is a schematic diagram of a storage block structure partitioned by column blocks according to an embodiment of the present disclosure. As shown in Figure 8, the storage block structure partitioned by column blocks includes the following cases:
[0099] When the amount of data in the last column block of the last row group is the same as the amount of data that can be cached in the storage block, the last storage block is a metadata storage block;
[0100] When the amount of data in the last column block is less than the amount of data that the storage block can cache, and the total amount of data in the last cached data block and the file metadata is less than the amount of data that the storage block can cache, the last storage block is a mixed storage block.
[0101] When the amount of data in the last column block is less than the amount of data that the storage block can cache, and the total amount of data in the last cached data block and the file metadata is greater than the amount of data that the storage block can cache, the last storage block is a metadata storage block; the last column block of the last row group is stored in the second to last storage block, and the part that is not full is filled with padding data, for example, 0x00.
[0102] In one embodiment, the method further includes: receiving a read request to read cached data; and filtering data that meets the data filtering conditions from the storage blocks of the distributed cache system according to the data filtering conditions in the read request, wherein the data filtering conditions include table metadata information.
[0103] In a distributed caching system, each table contains multiple columnar storage files (Parquet / ORC), each of which is divided into fixed-size storage blocks. Cached data skipping filtering uses table metadata, innode file metadata, block metadata, and columnar storage file (Parquet / ORC) metadata to select rows and columns that meet specific criteria, avoiding the reading of the entire table.
[0104] In one embodiment, filtering data that meets the data filtering conditions from the storage blocks of the distributed caching system according to the data filtering conditions in the read request includes: parsing the table metadata in the data filtering conditions and determining a target file list for a specified storage partition, wherein the specified storage partition and the target file list are specified by the table metadata; and filtering the columnar storage files to be read from the storage blocks of the distributed caching system according to the target file list. Here, "to be read" refers to files that meet the data filtering conditions.
[0105] In one embodiment, after selecting the columnar storage files to be read from the storage blocks of the distributed caching system, the method further includes: parsing the row group metadata in the data filtering conditions and determining the target row group, wherein the target row group is specified by the row group metadata; and selecting the column data to be read from the columnar storage files to be read based on the target row group, wherein each columnar storage file includes multiple columns of data. Here, "to be read" refers to data that meets the data filtering conditions.
[0106] In one embodiment, after filtering out the column data to be read from the columnar storage file to be read, the method further includes: parsing the column block metadata in the data filtering conditions and determining the target column block, wherein the target column block is specified by the column block metadata; and filtering out the column block data to be read from the column data to be read based on the target column block, wherein each column data includes multiple column block data. Here, "to be read" refers to data that meets the data filtering conditions.
[0107] Through the above embodiments of this disclosure, since the columnar storage file is divided into multiple cache data blocks according to the column block data of the columnar storage file and then stored in the distributed caching system, the function of supporting data skipping can be realized. Therefore, the problem that the existing distributed caching acceleration technology in the related art does not support block access and data skipping, and has poor random access performance can be solved, thereby improving data access efficiency.
[0108] The file metadata parsing process of a columnar storage file is a layer-by-layer parsing process from the outside in. Figure 9 is a schematic diagram of the file metadata reading format corresponding to a columnar storage file according to an embodiment of this disclosure. As shown in Figure 9, parsing the file metadata can obtain the row group list, table structure (i.e., table Scheme) and total number of records; parsing the row group metadata can obtain the column block list and row number contained in the row group; parsing the column block metadata can obtain the compression encoding, offset, and column value statistics of the column block (e.g., numeric type: maximum, minimum; character type: constant value).
[0109] Parsing column block metadata information helps to filter reads of cached data. For example, after parsing column block metadata information, cache block data can be filtered through methods such as column pruning and value range filtering.
[0110] Data skipping filtering achieved through partitioning, column pruning, and predicate pushdown can significantly reduce data access I / O in AI large model training frameworks / big data computing and analysis engines, thereby accelerating model training / computation and analysis.
[0111] In an exemplary embodiment, FIG10 is a schematic diagram of a skip-read filtering process for cached column block data according to an embodiment of the present disclosure. As shown in FIG10, the process includes the following steps:
[0112] Step S1001: Based on the data filtering conditions in the read request, parse the table metadata in the data filtering conditions; based on the partition information in the table metadata, filter out the list of Inode files for the specified storage partition, wherein the list of Inode files is functionally equivalent to the target file list in the above embodiment.
[0113] Step S1002: Read the list of Inode files in a loop, and obtain the corresponding storage block metadata information from the metadata of each Inode file in the list in turn; wherein, the Inode file is a columnar storage file.
[0114] As mentioned earlier, the corresponding underlying columnar storage file metadata information can be obtained from the tail block.
[0115] Step S1003: For any Inode file in the Inode file list, parse the row group metadata and read each row group in sequence.
[0116] Step S1004: For any row group of the Inode file, parse the column block metadata and read the data of each column block in sequence.
[0117] Step S1005: Based on the parsed row group metadata and column block metadata, determine whether the currently read column block data meets the data filtering conditions. If the current column block data meets the data filtering conditions, proceed to step S1006; otherwise, proceed to step S1007.
[0118] Step S1006: Add the data in this column block to the result location list.
[0119] Step S1007: Determine whether there are any unread column blocks in the currently read row group data. If there are unread column blocks in the currently read row group data, return to step S1003; if there are no unread column blocks in the currently read row group data, proceed to step S1008.
[0120] Step S1008: Determine whether there are any unread row groups in the currently read Inode file; if there are unread row groups in the currently read Inode file, return to step S1002; if there are no unread column block data in the currently read row group data, execute step S1009.
[0121] Step S1009: Based on the correspondence between row group, column block data and storage block location, read the datasets that meet the data filtering conditions from the result location list and add them to the result set.
[0122] Step S1010: Determine whether there are any unread Inode files in the Inode file list. If there are unread Inode files, return to step S1002; if there are no unread Inode files, proceed to step S1011.
[0123] Step S1011: Convert the result set into columnar in-memory format object batch data.
[0124] In one exemplary embodiment, in addition to skip-read filtering based on column block data, data skip-read filtering can also be performed based on row group data or columnar storage files. For example, by setting both file metadata and row group metadata in the data filtering conditions, data skip-read filtering based on row group data can be achieved; while by setting only file metadata in the data filtering conditions, data skip-read filtering based on columnar storage files can be achieved. Data skip-read filtering based on row group data or columnar storage file data is essentially the same as skip-read filtering based on column block data, only the unit of data skip-read filtering is different. Therefore, the specific process of data skip-read filtering based on row group data or columnar storage file data can be referred to the above process, and will not be repeated here.
[0125] This disclosure also provides a complete process for accelerating and enhancing AI large model training / big data computing and analysis. In this process, the distributed cache acceleration and enhancement system based on the above embodiment provides a batch access interface for Arrow objects that is more suitable for training frameworks / computation and analysis engines. This interface enables access to massive structured data files and can achieve data skipping and filtering functions through techniques such as column pruning and predicate pushdown. Compared with the Fuse / Posix access interface provided by traditional distributed cache, this interface can greatly reduce file I / O access and improve access speed.
[0126] Specifically, the process includes two parts: cache read acceleration and cache write acceleration. Cache read acceleration includes two cases: cache hit (reading from the cache) and cache miss (reading data from the underlying storage).
[0127] In one exemplary embodiment, FIG11 is a schematic diagram of a cache read acceleration process for cache hits according to an embodiment of the present disclosure. As shown in FIG11, the process includes the following steps:
[0128] In step S1101, the training framework / computation analysis engine sends a request to the Table Extender Client to request the reading of table data from the distributed cache acceleration enhancement system cache. The request includes the table name and data filtering conditions, which include: specifying the partition, specifying the column, and the column value range.
[0129] In step S1102, the Table Extender Client queries the Master for table metadata to obtain the Worker node where the cache is located and its storage path.
[0130] When the cached data is deployed on the same physical or virtual node as the local worker (i.e., the training framework or computational analysis engine and the cached data access component (local worker), the process also includes steps S11031 and S11032.
[0131] Step S11031: The Table Extender Client obtains the storage block where the metadata of the table on this node is located, and performs a predicate pushdown operation to obtain a list of information of column block data that meets the filtering conditions.
[0132] In step S11032, the Table Extender Client reads column block data that meets the data filtering conditions based on the information list and the block metadata mapping relationship.
[0133] When the cached data is deployed on a remote worker, i.e., the training framework or computational analysis engine, and the cached data access component are deployed on different physical or virtual nodes, the process includes step S11033.
[0134] Step S11033: The Table Extender Client delegates to the remote Worker node to obtain column block data that meets the data filtering conditions from the remote node and provides a batch interface for batch retrieval. The remote Worker node to which the delegation is made needs to have the Table Extender Client plugin installed.
[0135] In step S1104, the Table Extender Client converts the acquired column block data into Arrow RecordBatch objects and returns them to the training framework / computation analysis engine in batches.
[0136] In an exemplary embodiment, FIG12 is a schematic diagram of a cache miss acceleration process according to an embodiment of the present disclosure. As shown in FIG12, the process includes the following steps:
[0137] In step S1201, the training framework / computation analysis engine sends a request to the Table Extender Client to request the reading of table data from the distributed cache acceleration enhancement system cache. The request includes the table name and data filtering conditions, which include: specifying partitions, specifying columns, column value ranges, etc.
[0138] Step S1202: The table metadata that the Table Extender Client queried from the Master does not exist.
[0139] In step S1203, the Table Extender Client obtains table metadata information and the storage path of the table data in the underlying storage from the metadata service of the underlying data lake.
[0140] Step S1204: The Table Extender Client retrieves the table data file from the underlying storage according to the specified partition. The table data file is a columnar storage file such as Parquet or ORC.
[0141] In step S1205, the Table Extender Client creates table metadata and writes the table data file to the local cache.
[0142] Step S1206: The Table Extender Client performs predicate pushdown based on the filter condition column clipping and column value range to obtain a list of column block information that meets the data filtering conditions.
[0143] In step S1207, the Table Extender Client reads the column block data that meets the conditions based on the information list and the block metadata mapping relationship.
[0144] Step S1208: Convert the acquired column block data into Arrow RecordBatch objects and return them to the training framework / computation analysis engine in batches.
[0145] In this embodiment of the disclosure, for structured data that needs to be persisted for model training / computational analysis, the training framework / computational analysis engine can also write Arrow objects directly to the underlying storage in batches. This can greatly simplify the writing process of the training framework and improve computational efficiency.
[0146] In an exemplary embodiment, FIG13 is a schematic diagram of a cache write acceleration process according to an embodiment of the present disclosure. As shown in FIG13, the process includes the following steps:
[0147] Step S1301: The training framework / computational analysis engine sends an Arrow RecordBatch write request to the Table Extender Client.
[0148] In step S1302, the Table Extender Client converts the Arrow RecordBatch into a columnar storage format (Parquet, ORC, etc.).
[0149] In step S1303, the Table Extender Client sends a request to the Master to create table metadata, and generates table metadata iNode nodes (including partition directories) on the distributed cache directory tree iNodeTree.
[0150] In step S1304, the Table Extender Client requests a storage block ID and a Wroker node that can be used to store table data cache from the Master, preferably the local Wroker node.
[0151] In step S1305, the Table Extender Client divides the written data / column storage file into cached data blocks of a specified size (default 64M) according to the block division strategy (by row group, by column block).
[0152] When the data is written to a local worker, the process also includes:
[0153] In step S13061, the Table Extender Client requests the storage block ID and column block data block storage path from the local Worker. This request can be a batch operation, meaning that when data needs to be stored in different storage blocks, multiple storage block IDs and column block data block storage paths can be requested at once.
[0154] In step S13062, the Table Extender Client writes the data blocks that need to be cached to the local storage blocks.
[0155] In step S13063, the Table Extender Client submits the updated block metadata information to the Master and returns a success message to the training framework / computation analysis engine.
[0156] When the data is written to a remote worker, the process also includes:
[0157] In step S13071, the Table Extender Client requests a storage block ID and block storage path from the remote Worker, and delegates the remote Worker to write the cached data block to the remote cache. The remote Worker node to which the delegation is made needs to have the Table Extender Client plugin installed.
[0158] In step S13072, the Table Extender Client submits the updated block metadata information to the Master and returns a success message to the training framework / computation analysis engine.
[0159] By deploying Table Extender Client on the application computing side and innovatively using a block-based method for structured data columnar storage format files to accelerate AI large-scale model training and big data computing and analysis, seamless integration with application computing frameworks is achieved. This enables the following functional extensions: expanded support for table metadata, allowing caching of structured data by table; expanded cache block file storage format, supporting common columnar storage formats such as Parquet and ORC; support for storage compression, supporting efficient compression algorithms such as Snappy and ZSTD; support for data skipping and filtering access, providing data filtering methods such as column pruning and predicate pushdown; and a batch access interface for Arrow columnar storage format objects, supporting multiple programming languages such as Python and Rust.
[0160] The Table Extender Client extension enhances the distributed cache's support for table metadata and columnar storage formats, significantly improving current distributed cache acceleration solutions and accelerating model training and big data computing and analysis.
[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.
[0162] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.
[0163] Embodiments of this disclosure also provide a computer-readable storage medium storing a computer program configured to perform the steps in any of the above method embodiments when executed.
[0164] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0165] Embodiments of this disclosure also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.
[0166] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.
[0167] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.
[0168] It is obvious to those skilled in the art that the modules or steps of this disclosure described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this disclosure is not limited to any particular combination of hardware and software.
[0169] The above description is merely an exemplary embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A data caching method, applied to a table extension client, comprising: Based on the column block data of the columnar storage file, the columnar storage file is divided into multiple cache data blocks; The multiple cached data blocks are respectively cached into multiple storage blocks of the distributed caching system; The mapping relationship between the multiple cached data blocks and the multiple storage blocks is stored in the block metadata of the corresponding storage block.
2. The method of claim 1, wherein, The table extension client is set up in the distributed caching system, and the file directory tree metadata of the distributed caching system includes table metadata.
3. The method of claim 1, wherein, The step of dividing the columnar storage file into multiple cache data blocks based on the column block data of the columnar storage file includes: The columnar storage file is divided into multiple cache data blocks, with each column block of data serving as a cache data block; or, The columnar storage file is divided into multiple cache data blocks, with each row group containing multiple column blocks as a single cache data block.
4. The method of claim 1, wherein, The step of caching the plurality of cached data blocks into the plurality of storage blocks of the distributed caching system includes: The plurality of cached data blocks are respectively cached in a plurality of storage blocks of a distributed caching system. Except for the last cached data block in the plurality of cached data blocks, the data volume of the other cached data blocks in the plurality of cached data blocks is the same as the data volume that can be cached in the storage blocks, and the data volume that can be cached in each storage block is the same. The file metadata corresponding to the columnar storage file is stored in the end storage block.
5. The method of claim 4, wherein, The file metadata corresponding to the columnar storage file is stored in the end storage block, including: If the amount of data in the final cached data block is the same as the amount of data that can be cached in the storage block, then only the file metadata is stored in the final storage block; or If the amount of data in the end-of-storage cached data block is less than the amount of data that the storage block can cache, and the total amount of data in the end-of-storage cached data block and the file metadata is less than the amount of data that the storage block can cache, then the file metadata and the end-of-storage cached data block are stored in the end-of-storage storage block; or If the amount of data in the last cached data block is less than the amount of data that the storage block can cache, and the total amount of data in the last cached data block and the file metadata is greater than the amount of data that the storage block can cache, then the file metadata is stored in the last storage block, and the last cached data block and padding data are stored in the storage block preceding the last storage block.
6. The method of claim 3, wherein, When a cached data block corresponds to a row group, the mapping relationship between the multiple cached data blocks and the multiple storage blocks includes: the mapping relationship between the row group number corresponding to each row group and the storage information of the corresponding storage block.
7. The method of claim 6, wherein, The storage information of the corresponding storage block includes: the identification information of the corresponding storage block, the storage path of the corresponding storage block, the file offset, the offset within the block, and the amount of row group data.
8. The method of claim 3, wherein, When a cached data block corresponds to a column block of data, the mapping relationship between the multiple cached data blocks and the multiple storage blocks includes: the mapping relationship between the row group number, column block number, and storage information of the corresponding storage block for each column block of data.
9. The method of claim 8, wherein, The storage information of the corresponding storage block includes: the identification information of the corresponding storage block, the storage path of the corresponding storage block, the file offset, the offset within the block, and the data volume of the column block data.
10. The method of claim 1, wherein, The method further includes: Receive read requests for reading cached data; Based on the data filtering conditions in the read request, data that meets the data filtering conditions is selected from the storage blocks of the distributed cache system, wherein the data filtering conditions include table metadata information.
11. The method of claim 10, wherein, Based on the data filtering conditions in the read request, data that meets the data filtering conditions is selected from the storage blocks of the distributed caching system, including: Parse the table metadata in the data filtering conditions and determine the target file list for the specified storage partition, wherein the specified storage partition and the target file list are specified by the table metadata; Based on the target file list, the columnar storage files to be read are selected from the storage blocks of the distributed caching system.
12. The method of claim 11, wherein, After selecting the columnar storage files to be read from the storage blocks of the distributed caching system, the method further includes: Parse the row group metadata in the data filtering conditions and determine the target row group, wherein the target row group is specified by the row group metadata; Based on the target row group, the column data to be read is selected from the columnar storage file to be read, wherein each columnar storage file includes multiple columns of data.
13. The method of claim 12, wherein, After filtering out the column data to be read from the columnar storage file to be read, the method further includes: Parse the column block metadata in the data filtering conditions and determine the target column block, wherein the target column block is specified by the column block metadata; Based on the target column block, the column block data to be read is filtered out from the column data to be read, wherein each column data to be read includes multiple column block data.
14. A distributed cache system provided with a table extension client, wherein, The table extension client includes: A cache extension layer is used to obtain column block data that meets the filtering conditions and send the column block data to the format conversion layer; The format conversion layer is used to convert the column block data into a first column-format memory object batch data, and send the first column-format memory object batch data to the application programming interface (API) layer. The API layer is used to send batch data of the first columnar memory format object to the application layer of the distributed caching system.
15. The system of claim 14, wherein, The API layer is also used to receive batch data of second columnar memory format objects from the large model training framework or big data computing and analysis engine of the application layer.
16. The system of claim 14, wherein, The format conversion layer is also used to convert batch data of the second columnar memory format object into columnar storage format and send it to the storage layer for caching and finally synchronize it to the underlying storage.
17. The system of claim 14, wherein, The cache extension layer is also used to extend the functionality of the distributed caching system, wherein the extended functionality includes at least one of the following: Supports table structure field information and partition information; Supports the management and parsing of columnar storage files; Supports column-wise compression of columnar storage files; It supports skipping through cached data using data filtering methods; It supports deployment via plugin packages on remote worker nodes.
18. A computer readable storage medium having stored therein a computer program, wherein, The computer program, which is executed by a processor, realizes the steps of the method as claimed in any one of claims 1 to 13.
19. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor, which executes the computer program, realizes the steps of the method as claimed in any one of claims 1 to 13.
20. A computer program product comprising a computer program which, when executed by a processor, realizes the steps of the method as claimed in any one of claims 1 to 13.
