Data processing method and apparatus, computer, storage medium, and program product

Abstract

The present application discloses a data processing method and apparatus, a computer, a storage medium, and a program product. The method comprises: a batch-stream processing server acquires t pieces of queue data sent by a batch-stream processing client, each piece of queue data being one piece of data among first service data generated by the batch-stream processing client for original data, and the first service data being generated by the batch-stream processing client on the basis of structural information of the original data and data record information for the original data; writing the t pieces of queue data into a service queue, and by means of a lake synchronization service, converting the t pieces of queue data into second service data in a columnar storage format, the service queue being used for providing data for a data stream-read request initiated by the batch-stream processing client; and writing the second service data into a data lake by means of the lake synchronization service, the data lake being used for providing data for a data batch-read request initiated by the batch-stream processing client. By using the present application, the efficiency and flexibility of data processing can be improved.

Description

Data processing methods, apparatus, computers, storage media and software products

[0001] This application claims priority to Chinese Patent Application No. 2025101163292, filed on January 23, 2025, entitled “Data Processing Method, Apparatus, Computer, Storage Medium and Program Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer, storage medium and program product. Background Technology

[0003] To accommodate the demands of massive data processing and real-time performance, the data platform architectures widely used in the big data field are highly complex, and their data processing is also quite intricate. Therefore, data lakes are generally used for integrated batch and stream storage. However, data lakes rely on distributed file systems (Hadoop Distributed File System, HDFS), simple storage services (S3 Simple Storage Service, S3), or cloud object storage (COS) to access data. These storage architectures prioritize throughput over latency, making current data access methods unable to meet the real-time data processing requirements brought about by the development of big data systems. Summary of the Invention

[0004] This application provides a data processing method, apparatus, computer, storage medium, and program product, which can improve the efficiency and flexibility of data processing.

[0005] This application provides a data processing method, which is executed by a batch stream processing server, and includes:

[0006] Retrieve t queue data sent by the batch stream processing client; the batch stream processing client provides an access interface for the data processing engine; each queue data is a piece of data in the first business data generated by the batch stream processing client for the original data; t is a positive integer; the first business data is generated by the batch stream processing client based on the structure information of the original data and the data record information for the original data sent by the data processing engine; the first business data is in row storage format.

[0007] Write t queue data into the service queue, call the lake synchronization service, and convert the t queue data into second business data in columnar storage format through the lake synchronization service; the service queue is used to provide data for data stream read requests initiated by batch stream processing clients;

[0008] The second business data is written into the data lake through the lake synchronization service; the data lake is used to provide data for batch read requests initiated by batch processing clients.

[0009] This application provides a data processing method, executed by a batch stream processing client, comprising:

[0010] Obtain the raw data sent by the data processing engine, as well as the data record information generated by the data processing engine based on the raw data;

[0011] Obtain the structural information of the original data, and generate t queue data based on the structural information and data record information; t is a positive integer;

[0012] t queues of data are sent to the batch stream processing server, which then writes the t queues of data into a service queue. The server then uses a lake synchronization service to convert the t queues of data into second business data in columnar storage format. This second business data is written to the data lake by the batch stream processing server via the lake synchronization service. The service queue provides data for data stream read requests initiated by the batch stream processing client. The data lake provides data for batch read requests initiated by the batch stream processing client.

[0013] Continue generating the next queue data based on the structure information and data record information, until all queue data corresponding to the original data are obtained; all queue data corresponding to the original data constitute the first business data; the first business data is data in row storage format.

[0014] One embodiment of this application provides a data processing apparatus, the apparatus comprising:

[0015] The queue acquisition module is used to acquire t queue data sent by the batch stream processing client; the batch stream processing client provides an access interface for the data processing engine; each queue data is a piece of data in the first business data generated by the batch stream processing client for the original data; t is a positive integer; the first business data is generated by the batch stream processing client based on the structure information of the original data and the data record information for the original data sent by the data processing engine; the first business data is in row storage format.

[0016] The data conversion module is used to write t queues of data into the service queue, call the lake synchronization service, and convert the t queues of data into second business data in columnar storage format through the lake synchronization service; the service queue is used to provide data for data stream read requests initiated by batch stream processing clients;

[0017] The data lake module is used to write the second business data into the data lake through the lake synchronization service; the data lake is used to provide data for batch read requests initiated by batch processing clients.

[0018] The data conversion module can be used for:

[0019] From the N batch stream processing service partitions included in the batch stream processing server, determine the target batch stream processing service partition, and write t queue data into the service queue in the target batch stream processing service partition; N is a positive integer;

[0020] Call the lake synchronization service in the target batch stream processing service partition, and convert t queue data into second business data in columnar storage format through the lake synchronization service in the target batch stream processing service partition.

[0021] Among them, N batch stream processing service partitions each correspond to different partition values; when determining the target batch stream processing service partition from the N batch stream processing service partitions included in the batch stream processing server, this data transformation module can be used for:

[0022] The batch stream processing server performs round-robin processing on its N batch stream processing service partitions to obtain the target batch stream processing service partition for t queues of data; or,

[0023] Obtain the queue processing parameters, perform a hash transformation on the first business data using the queue processing parameters to obtain the business hash value, and perform a modulo operation on the business hash value using N to obtain the target partition value; the queue processing parameters are preset hash values ​​used for hash transformation;

[0024] Among the N batch stream processing service partitions, the batch stream processing service partition corresponding to the target partition value is determined as the target batch stream processing service partition.

[0025] Specifically, when calling the lake synchronization service to convert t queue data into columnar storage format for second business data, this data conversion module can be used for:

[0026] When calling the lake synchronization service, if the amount of data in t queues exceeds the data write threshold, then the t queues will be split into M business sub-data based on the data write threshold; M is a positive integer.

[0027] In the lake synchronization service, M write threads are used to perform parallel columnar storage conversion on M business sub-data to obtain second business data with columnar storage format corresponding to each of the M business sub-data.

[0028] The data landing module can be used for:

[0029] The M write threads in the lake synchronization service generate commit information for one of the M second business data, and obtain the data write path corresponding to the M second business data.

[0030] Using M write threads, the commit information corresponding to each of the M write threads is submitted to the coordinating node corresponding to the data write path.

[0031] The commit thread scans the coordinating node. If commit information is found in the coordinating node, it is written into the data lake. The commit thread continues to scan the coordinating node until no commit information is found in the coordinating node.

[0032] The number of data lakes is K, where K is a positive integer; one data lake corresponds to one type of lake.

[0033] This data dumping module can be used for:

[0034] Obtain the lake configuration data from the lake synchronization service, parse the second business data, and determine the data storage parameters of the second business data;

[0035] Based on the lake configuration data, the target data lake corresponding to the data storage parameters is determined from K data lakes, and the second business data is written to the target data lake through the lake synchronization service.

[0036] Specifically, when writing t queue data into the service queue, this data transformation module can be used for:

[0037] Obtain the business object information that generates the raw data, and determine the data storage space associated with the business object information from the data storage space included in the batch stream processing server; the batch stream processing server includes multiple mutually isolated data storage spaces;

[0038] Write the t queue data into the service queue in the data storage space associated with the business object information in sequence.

[0039] One embodiment of this application provides a data processing apparatus, the apparatus comprising:

[0040] The data acquisition module is used to acquire the raw data sent by the data processing engine, as well as the data record information generated by the data processing engine based on the raw data;

[0041] The data construction module is used to obtain the structural information of the raw data and generate t queue data based on the structural information and data record information; t is a positive integer.

[0042] The queue generation module sends t queue data to the batch stream processing server, so that the server writes the t queue data into a service queue and converts it into second business data in columnar storage format via the lake synchronization service. The second business data is written to the data lake by the batch stream processing server via the lake synchronization service. The service queue provides data for data stream read requests initiated by the batch stream processing client. The data lake provides data for batch read requests initiated by the batch stream processing client.

[0043] The structural information includes queue structure information and message structure information;

[0044] This data building module can be used for:

[0045] The original data is parsed to construct queue structure information for the original data; the queue structure information is used to split the original data into row-based storage format data.

[0046] Message structure information is constructed based on queue structure information and data record information. Data record information is then serialized based on message structure information to obtain t queue data in sequence.

[0047] The number of data processing engines is H; H is a positive integer.

[0048] This data building module can be used for:

[0049] Call the data writing function associated with the data processing engine that sends the raw data, and obtain the structural information of the raw data through the data writing function; H data processing engines include the data processing engine used to send the raw data;

[0050] The data writing function converts the structural information and data record information into t queues of data.

[0051] The device also includes:

[0052] The request parsing module is used to respond to data read requests sent by the data processing engine, and to obtain the data read method and data read parameters from the data read request; the data read parameters are obtained by the data processing engine from parsing the structured query statement;

[0053] The data reading module is used to determine the area to be read from the batch stream processing server and the data lake based on the data reading method, and to obtain the third business data requested by the data reading request from the area to be read based on the data reading parameters.

[0054] The data sending module is used to send third-party business data to the data processing engine.

[0055] The data reading module can be used for:

[0056] If the data reading method is streaming, then the batch stream processing server is identified as the area to be read, a data read request is sent to the batch stream processing server, and the third business data returned by the batch stream processing server based on the data read request is obtained; the third business data is obtained by the batch stream processing server from the service queue based on the data read parameters.

[0057] If the data reading method is batch reading, then the data lake is identified as the area to be read, and the third business data requested by the data reading request is obtained from the data lake based on the data reading parameters.

[0058] The device also includes:

[0059] The write processing module is used to obtain write configuration parameters. If the write configuration parameter is the first write configuration parameter, the step of obtaining the structure information of the original data is executed. The write configuration parameter is used to indicate the data storage method.

[0060] The write processing module is also used to write the original data into the data lake if the write configuration parameter is the second write configuration parameter.

[0061] One embodiment of this application provides a computer device, including a processor, a memory, and an input / output interface;

[0062] The processor is connected to a memory and an input / output interface, respectively. The input / output interface is used to receive and output data, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device containing the processor executes the data processing method in one aspect of the embodiments of this application.

[0063] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having the processor performs the data processing method of one aspect of this application.

[0064] One aspect of this application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional embodiments of this application. In other words, when the computer instructions are executed by the processor, they implement the methods provided in various optional embodiments of this application.

[0065] Implementing the embodiments of this application will have the following beneficial effects:

[0066] In this embodiment, a batch stream processing client can be constructed, which connects the data processing engine and the batch stream processing server. When the data processing engine obtains raw data, it can send the data record information for the raw data to the batch stream processing client. The batch stream processing client generates first business data based on the structure information of the raw data and the data record information for the raw data. The first business data includes multiple queue data, and when the queue data is obtained, it sends the queue data to the batch stream processing server. The batch stream processing server can write the obtained queue data into a service queue and can call the lake synchronization service to convert the obtained queue data into second business data in columnar storage format. The second business data is then written into the data lake through the lake synchronization service. By constructing a batch-stream processing client and integrating a data lake synchronization service into the batch-stream processing server, unified batch-stream and data access is achieved. Specifically, the raw data sent by the data processing engine is synchronously stored based on a service queue and a data lake. This allows subsequent data retrieval based on stream read requests from the service queue and batch read requests from the data lake. Due to the first-in-first-out (FIFO) and read-as-you-go characteristics of the service queue, real-time data stream processing is ensured, resolving issues such as high latency and sequentiality in data lake stream writing and reading, thus improving the efficiency and real-time performance of data stream processing. Simultaneously, the use of the data lake is retained, enabling the processing of various data types and handling of large volumes of data while ensuring data integrity. In this embodiment, the simultaneous use of a service queue and a data lake for data retrieval processing allows the application to retain the characteristics of both. Data can be retrieved from either the service queue or the data lake as needed, thereby improving the efficiency and flexibility of data processing. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 is a network interaction architecture diagram for data processing provided in an embodiment of this application;

[0069] Figure 2 is a system architecture diagram of data processing provided in an embodiment of this application;

[0070] Figure 3 is a schematic diagram of a data processing functional architecture provided in an embodiment of this application;

[0071] Figure 4 is a flowchart of a data processing method provided in an embodiment of this application;

[0072] Figure 5 is a schematic diagram of a batch stream processing server scenario provided in an embodiment of this application;

[0073] Figure 6 is a schematic diagram of a data writing processing scenario provided in an embodiment of this application;

[0074] Figure 7 is a schematic diagram of another batch stream processing server-side processing scenario provided in an embodiment of this application;

[0075] Figure 8 is a schematic diagram of a data lake processing scenario provided in an embodiment of this application;

[0076] Figure 9 is a schematic diagram of another batch stream processing server-side processing scenario provided in an embodiment of this application;

[0077] Figure 10 is a flowchart of another data processing method provided in an embodiment of this application;

[0078] Figure 11 is an interactive flowchart of a data writing method provided in an embodiment of this application;

[0079] Figure 12 is an interactive flowchart of a data reading method provided in an embodiment of this application;

[0080] Figure 13 is a schematic diagram of a data processing device provided in an embodiment of this application;

[0081] Figure 14 is a schematic diagram of another data processing device provided in an embodiment of this application;

[0082] Figure 15a is a schematic diagram comparing the read / write throughput of this application and a data lake according to an embodiment of this application;

[0083] Figure 15b is a comparison of the streaming read latency of this application and a data lake provided in an embodiment of this application;

[0084] Figure 15c is a comparison of the system throughput of this application and a data lake provided in an embodiment of this application;

[0085] Figure 15d is a schematic diagram comparing the throughput of the client and the server during data processing according to an embodiment of this application;

[0086] Figure 16 is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0087] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0088] If this application requires the collection of object data (such as user data), a prompt interface or pop-up window will be displayed before and during the collection process. This prompt interface or pop-up window is used to inform the user that certain data is being collected. The data acquisition steps will only begin after the user confirms the prompt interface or pop-up window; otherwise, the process will end. Furthermore, the acquired user data (such as the raw data in this application) will be used in reasonable and legal scenarios or for legitimate purposes. Optionally, in scenarios where user data needs to be used but user authorization has not been obtained, authorization can be requested from the user, and the user data can only be used after authorization is granted.

[0089] In this application embodiment, please refer to Figure 1. Figure 1 is a network interaction architecture diagram of data processing provided by this application embodiment. As shown in Figure 1, this application is used to implement batch & stream storage (BSS), which aims to unify real-time stream processing and offline batch processing. The batch & stream storage system implemented by this application may include a batch & stream processing client 101 (which can be referred to as a BSS client), a batch & stream processing server 102 (which can be referred to as a BSS server), and a data lake. Specifically, this application can provide an access interface to the data processing engine through a batch-stream processing client 101. The batch-stream processing client 101 can continuously serialize the raw data sent by the data processing engine to obtain queue data, and send the queue data to the batch-stream processing server 102 until all queue data corresponding to the raw data is obtained. Then, it can serialize the next raw data sent by the data processing engine. The batch-stream processing server 102 can write the obtained queue data into the service queue, and through the lake synchronization service, convert the obtained queue data into second business data in columnar storage format, and write the second business data into the data lake. This realizes the synchronous storage of data in the service queue and the data lake, so that the batch-stream integrated storage system in this application can simultaneously possess the characteristics of service queue and data lake. This enables batch-stream integrated access, provides high-throughput and low-latency access services, meets the performance and timeliness requirements of business processing, improves the efficiency and flexibility of data processing, and saves storage and maintenance costs.

[0090] Specifically, the batch stream processing client 101, or BSS Client, can be denoted as BSS Client and may include a core processing unit (BS Core) and associated engine units (Clients). The core processing unit is used to generate queue data and interact with the batch stream processing server 102, such as sending queue data to the batch stream processing server 102 and retrieving data from the service queue from the batch stream processing server 102. The associated engine units are used to provide access interfaces for the data processing engines. The data processing engines can be considered as big data processing frameworks. The number of data processing engines can be one or more, denoted as H, where H is a positive integer, such as data processing engine 1 and data processing engine 2 shown in Figure 1. For example, data processing engines can include, but are not limited to, batch and micro-batch processing engines (Spark) and event-driven streaming engines (Flink). Spark is an open-source project for distributed data processing and parallel computing, characterized by high efficiency, versatility, and ease of programming, and is suitable for scenarios such as machine learning, big data processing, real-time data processing, and distributed applications. Flink is a distributed big data computing engine that can perform stateful computations on bounded and unbounded data, can be deployed in various cluster environments, and can perform fast computations on data of various sizes. The batch stream processing client 101 may further include a local proxy mode 103 for the data lake. This local proxy mode 103 includes lake information for the data lake, which may include, but is not limited to, the corresponding data lake's metadata catalog, table structure, and data connection information (connector). The metadata catalog consists of the metadata of the corresponding data lake instance, including but not limited to base tables, synonyms, indexes, and users. The table structure represents the tables included in the data lake, the fields that make up the tables, and the relationships between tables and fields. The data connection information represents the access method to the corresponding data lake. The batch stream processing client 101 can access the data lake and read data from the data lake through the local proxy mode 103. For example, the batch stream processing client 101 can access data lake 1 through the lake information of data lake 1, and access data lake 2 through the lake information of data lake 2, and so on.

[0091] The batch stream processing server 102, also known as the BS Server or BSS server, can provide one or more batch stream processing services, such as batch stream processing service 1, batch stream processing service 2, and batch stream processing service 3 as shown in Figure 1. Each batch stream processing service can provide a data lake synchronization service, which can be used to perform data lake-writing processing on the acquired queue data. Data lake-writing processing refers to the process of converting the acquired queue data into columnar storage format data that can be processed by the data lake, and writing the converted data into the data lake. Columnar storage format is a physical data storage method corresponding to row storage format, which refers to a storage format in which each column of data is stored together.

[0092] The number of data lakes is K, where K is a positive integer. Each data lake corresponds to a specific data lake format, such as data lake 1041 and data lake 1042 shown in Figure 1. Optionally, a data lake can simultaneously provide data lake services and data warehouse services, achieving lake-warehouse integration. Each data lake can include d tables, where d is a positive integer. Optionally, when K is greater than 1, the d corresponding to each data lake can be different, and multiple data lakes can have the same d; this is not restricted. Tables are used to store the data in the data lake to which the table belongs. For example, the K data lakes can include, but are not limited to, Iceberg (an open table format for large analytical datasets) and Paimon (a streaming data lake storage system).

[0093] Optionally, a table metadata management service can be built for each data lake and its service queue, which can be considered as a batch-stream storage (BSS). For example, a metadata construction method (such as the "CREATE CATALOG default_catalog_t" method) can be called to build a table metadata management service for each data lake or service queue. The database table metadata of each data lake or service queue can be associated through the configuration parameters of the table metadata management service. These configuration parameters may include, but are not limited to, data processing type (type), storage format (bs.table.format), and data address (url). For example, it may also include data service cluster (bs.cluster), metadata management type (catalog-type), default database (default-database), warehouse, and metadata processing component (metastore). In this application, `type` represents the data lake's data processing method, which is batch-stream storage (BSS). The data address can include metadata addresses (bs.catalog.url) and data storage addresses (url). `data service cluster` represents the storage cluster allocated to the data. `catalog-type` represents the tools or methods used for storing, querying, and analyzing the data lake's metadata, such as Hive. `default-database` represents the default database used by the corresponding data lake for storing data. `warehouse` represents the system used by the corresponding data lake for processing data, such as a distributed file system (Hadoop Distributed File System, HDFS). `metastore` represents the tools or methods used in the corresponding data lake for storing, querying, and analyzing big data, such as Hive. Hive is a distributed data warehouse tool that can map structured data into a database table and provide services such as data storage, data querying, and data analysis.

[0094] For example, BS can be the batch-stream integrated storage format implemented in this application, and a table metadata management service can be built for this BS. In one optional way, this process can be written as "CREATE CATALOG default_catalog_t WITH('type'='bss', 'bs.table.format'='bs', 'bs.catalog.url'='http: / / …', 'bs.cluster'='…', …)". For Iceberg, in another optional way, this process can be written as "CREATE CATALOG default_catalog_t WITH('type'='bss', 'bs.table.format'='iceberg', 'catalog-type'='hive', 'url'='thrift: / / …', 'default-database'='…', 'warehouse'='hdfs: / / … / ', …)". In Paimon, an optional approach, this process can be written as "CREATE CATALOG default_catalog_t WITH('type'='bss', 'bs.table.format'='paimon', 'default-database'='…', 'warehouse'='hdfs: / / … / ', 'metastore'='hive', 'url'='thrift: / / …', …)". The above is merely one possible way to construct a data lake and its service queue's table metadata management service.

[0095] Please refer to Figure 2, which is a system architecture diagram of data processing provided in an embodiment of this application. As shown in Figure 2, the data processing engine 201 can interact with the batch stream processing client 202. The batch stream processing client 202 provides an access interface for the data processing engine 201, allowing the data processing engine 201 to access the batch stream processing server 203. The batch stream processing client 202 can serialize data to obtain queue data and send the queue data to the batch stream processing server 203. It can also read data from the service queue in the batch stream processing server 203 and perform data storage and retrieval on the data lake 204. The batch stream processing server 203 can perform "data lake writing" processing on the acquired queue data, which involves writing the data into the data lake 204. The number of data processing engines 201 can be H, such as data processing engines 201a and 201b in Figure 2; the number of data lakes 204 can be K, such as data lakes 204a and 204b in Figure 2.

[0096] Specifically, please refer to Figure 3, which is a schematic diagram of the functional architecture of data processing provided in an embodiment of this application. As shown in Figure 3, the batch-stream integrated storage system implemented in this application may include a data computing layer 301, a data access layer 302, a data service layer 303, and a data storage layer 304. The data computing layer 301 provides multi-engine support and may include H data processing engines, such as data processing engine 1 and data processing engine 2. The data access layer 302 is a batch-stream processing client. The data service layer 303 is a batch-stream processing server, providing multiple functional services, such as load balancing service, multi-tenant service, table metadata management service (Catalog Manager), row storage service, and Lakesync service. Load balancing ensures efficient utilization of system resources and performance optimization. The BSS server can dynamically adjust the load distribution based on this load balancing service, ensuring uniform distribution of data sent by the BSS client and high system availability. Multi-tenancy services support multiple independent users or organizations sharing the same BSS cluster without interfering with each other. The BSS cluster can be divided into multiple isolated data storage spaces using tenants and namespaces to achieve multi-tenancy isolation and improve space utilization. Table metadata management services provide metadata management for tables in the data lake, including but not limited to basic table information such as table name, schema, primary key, and attributes. It can be used to generate and manage the metadata of service queues and their corresponding database tables in each data lake. For example, when a batch / stream processing server converts queue data into columnar storage format for second business data, it can obtain the database table metadata of the data lake to which the second business data will be written through this table metadata management service. Based on this metadata, the queue data is converted into columnar storage format for the second business data. Similarly, when a batch / stream processing client obtains the structure information of raw data, it can construct the structure information of the raw data using the database table metadata corresponding to the message queue. The structure information of the raw data represents its data composition. Row-based storage services provide the functionality to store queue data, i.e., provide storage space for service queues. Lakesync is used to process queued data by writing it to the data lake. The data storage layer 304 is used to implement batch storage of data and can include K data lakes, such as data lake 1 and data lake 2. The BSS client and BSS server can perform stream writing and reading of data. The BSS server can synchronously write queued data to the data lake, and the BSS client can batch read data from the data lake.

[0097] It is understood that the batch streaming client or batch streaming server mentioned in the embodiments of this application can be a computer device, including but not limited to terminal devices or servers. In other words, the computer device can be a server or a terminal device, or a system composed of a server and a terminal device. Among them, the terminal device mentioned above can be an electronic device, including but not limited to mobile phones, tablets, desktop computers, laptops, handheld computers, in-vehicle devices, augmented reality / virtual reality (AR / VR) devices, head-mounted displays, smart TVs, wearable devices, smart speakers, digital cameras, webcams and other mobile internet devices (MIDs) with network access capabilities, or terminal devices in scenarios such as trains, ships, and flights. The servers mentioned above can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, vehicle-road cooperation, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0098] Further, please refer to Figure 4, which is a flowchart of a data processing method provided in an embodiment of this application. As shown in Figure 4, the method is executed by a batch processing server, and the data processing process includes the following steps:

[0099] Step S401: Obtain t queue data sent by the batch stream processing client.

[0100] In this embodiment, the batch stream processing client provides an access interface for the data processing engine; each queue data is a piece of data in the first business data generated by the batch stream processing client for the original data; t is a positive integer; the first business data is generated by the batch stream processing client based on the structure information of the original data and the data record information for the original data sent by the data processing engine; the first business data is data in row storage format.

[0101] Specifically, the data processing engine can generate raw data, create data records for that raw data, and send the raw data and data records to the batch / stream processing client. Alternatively, it can directly send the data records to the batch / stream processing client. The data records represent complete information about the raw data and can be in any form, including but not limited to text, numbers, and dates. The data records can be of a composite data type. For example, the raw data might be as shown in Table 1 below:

[0102] Table 1

[0103] For the original data shown in Table 1, the data record information can include at least "Name: Zhao, Student ID: 201501, Class: Grade 5, Contact: Mobile Number 1; Name: Qian, Student ID: 201502, Class: Grade 5, Contact: Mobile Number 2", etc. Furthermore, the batch processing client can generate the first business data based on the structure information and data record information of the original data. The structure information of the original data is used to represent the parameter objects included in the original data, such as "Name, Student ID, Class, and Contact Information" in the original data shown in Table 1. The first business data can include A queues of data, where A is a positive integer.

[0104] Furthermore, the batch streaming client can send t queue data to the batch streaming server, where t is a positive integer; the batch streaming server can obtain the t queue data and execute step S402. Optionally, the batch streaming client can send the queue data one by one. Specifically, the batch streaming client can serialize the data record information according to the structure information of the original data to obtain the h-th queue data, and send the h-th queue data to the batch streaming server, where h is a positive integer less than or equal to A. At this time, t is 1. That is to say, the batch streaming client can serialize the data record information according to the structure information of the original data, and send one queue data to the batch streaming server for each queue data obtained, until all queue data corresponding to the original data is sent to the batch streaming server. Alternatively, the batch streaming client can send queue data in batches. Specifically, the client can serialize the data record information based on the structure information of the original data to obtain the h-th queue data; it can then cache the h-th queue data. If the number of cached queue data is greater than or equal to the number of batch queue elements, or if the sending wait time is greater than or equal to the data sending period, the cached queue data is sent to the batch streaming server. In this case, the number of cached queue data can be denoted as t, where t is a positive integer less than or equal to the number of batch queue elements. Here, the number of batch queue elements refers to the number of queue data sent in each batch when sending queue data to the batch streaming server; the data sending period represents the maximum time interval between two data transmissions from the batch streaming client to the batch streaming server.

[0105] Step S402: Write t queue data into the service queue, call the lake synchronization service, and convert the t queue data into second business data in columnar storage format through the lake synchronization service.

[0106] In this embodiment, the batch stream processing server can write the acquired queue data into a service queue. The service queue is used to provide data for data stream read requests initiated by the batch stream processing client. The service queue is a message queue, which provides stream processing capabilities. Based on the first-in-first-out and read-as-you-go characteristics of the queue, it can solve the problems of high latency and sequentiality of data lake stream writing and reading, and realize real-time data processing, thereby improving the real-time performance of data processing. At the same time, due to the finite sequence of the queue, the data in the queue exhibits linear characteristics, and the amount of data corresponding to the queue will not be too large. Thus, when reading data from the queue, it is read sequentially from one position, and when writing data to the queue, it is written directly to the tail of the queue without the need for other position determination, thereby improving the efficiency of data processing.

[0107] Optionally, the batch stream processing server can obtain the business object information that generated the raw data and determine the data storage space associated with the business object information from the data storage spaces included in the batch stream processing server. The batch stream processing server includes multiple isolated data storage spaces, and the business object information is used to indicate the business object that generated the raw data. Specifically, the batch stream processing server can determine the data storage space associated with the business object information from the data storage spaces included in the batch stream processing client through a multi-tenant service. The batch stream processing server can sequentially write t queues of data into the service queue of the data storage space associated with the business object information. Optionally, if the data storage space associated with the business object information is not found in the data storage spaces included in the batch stream processing client, a data storage space can be allocated for the business object corresponding to the business information through a multi-tenant service; if the data storage space associated with the business object information is found, the step of writing t queues of data into the service queue of the data storage space associated with the business object information can be executed.

[0108] Optionally, the batch / stream processing server may include one or more batch / stream processing service partitions. Each partition provides batch / stream processing services; that is, one partition can be considered as providing one batch / stream processing service. Each partition includes a lake synchronization service. Specifically, the server can determine the target partition from the N partitions and write t queues of data into the service queue of the target partition (where N is a positive integer). The server then calls the lake synchronization service in the target partition to convert the t queues of data into columnar storage format second business data, making the data more suitable for the data lake's format requirements. Among them, columnar storage format is a data storage format corresponding to row storage format. Taking the data shown in Table 1 as an example, the data stored in row storage format includes two sets of data: "Name: Zhao; Student ID: 201501; Class: Class 5-1; Contact Information: Mobile Phone Number 1" and "Name: Qian; Student ID: 201502; Class: Class 5-1; Contact Information: Mobile Phone Number 2". The data stored in columnar storage format includes four sets of data: "Name: Zhao, Qian", "Student ID: 201501, 201502", "Class: Class 5-1, Class 5-1", and "Contact Information: Mobile Phone Number 1, Mobile Phone Number 2".

[0109] For example, referring to Figure 5, which is a schematic diagram of a batch stream processing server scenario provided by an embodiment of this application. As shown in Figure 5, the batch stream processing server may include N batch stream processing service partitions, such as batch stream processing service partition 1 (which can be denoted as Partition 1), batch stream processing service partition 2 (which can be denoted as Partition 2), and batch stream processing service partition N (which can be denoted as Partition N) shown in Figure 5. Each Partition includes a Lakesync service. The batch stream processing server can determine a target batch stream processing service partition from the N batch stream processing service partitions and store queue data in the target batch stream processing service partition. Specifically, each batch stream processing service partition may include service queues, such as service queue 5011 in Partition 1, service queue 5012 in Partition 2, and service queue 5013 in Partition N. The batch stream processing server can write t queue data into the service queues in the target batch stream processing service partition.

[0110] Optionally, the N batch stream processing service partitions can each correspond to different partition values. When determining the target batch stream processing service partition from the N batch stream processing service partitions included in the batch stream processing server, the batch stream processing server can perform round-robin processing on the N batch stream processing service partitions included in the batch stream processing server to obtain the target batch stream processing service partition for t queues of data. As shown in Figure 5, when processing data, the batch stream processing server can use Partition1 to PartitionN as the target batch stream processing service partitions in sequence. Alternatively, the batch stream processing server can randomly select one batch stream processing service partition from the N batch stream processing service partitions included in the batch stream processing server segment as the target batch stream processing service partition for t queues of data. Alternatively, the batch streaming server can obtain queue processing parameters, perform hash transformation on the first business data using these parameters to obtain a business hash value, and then perform a modulo operation on the business hash value using N, obtaining the target partition value. The queue processing parameters are preset hash values ​​used for hash transformation (which can be denoted as hash keys). The batch streaming service partition corresponding to the target partition value among the N batch streaming service partitions is determined as the target batch streaming service partition. Optionally, the process of determining this target batch streaming service partition can also be performed by the batch streaming client.

[0111] The batch processing server can obtain the data storage duration of historical business data stored in the service queue and delete historical business data whose data storage duration is greater than or equal to the storage duration threshold. The historical business data stored in the service queue includes the first business data. This enables real-time storage and management of data, ensuring that the amount of data in the service queue does not become too large, guaranteeing the finite sequence of the service queue, improving data processing efficiency and resource utilization, and saving data storage space.

[0112] Furthermore, in one scenario, where there is no batch / stream processing service partition (equivalent to the batch / stream processing server directly providing lake synchronization service), or when there is one batch / stream processing service partition, the batch / stream processing server can call the lake synchronization service (Lakesync) to convert t queue data into second business data in columnar storage format. Here, the first business data is in row-oriented storage format and includes A queue data. By converting each queue data into second business data in columnar storage format, the storage format of the second business data better meets the format requirements of data stored in the data lake, making the second business data easier to trim or compress, etc., suitable for big data processing. The format conversion process for the queue data is automatically executed by the batch / stream processing server through the lake synchronization service after obtaining the queue data, enabling different storage of the same data, thereby improving the flexibility of data processing.

[0113] In this system, the batch / stream processing server can employ a writer thread within the lake synchronization service to convert t queues of data into second business data in columnar storage format. This writer thread can be a single thread, allowing for single-threaded data processing. Alternatively, it can be multiple threads, enabling multi-threaded processing (MultiWriter). Specifically, the batch / stream processing server can call the lake synchronization service. If the amount of data in the t queues exceeds a data write threshold, the t queues can be split into M business sub-data pieces based on this threshold; M is a positive integer, where the data write threshold represents the upper limit of the amount of data processed by a single writer thread. The lake synchronization service uses M writer threads to perform parallel columnar storage conversion on the M business sub-data pieces, resulting in second business data with columnar storage format for each of the M sub-data pieces. This achieves multi-threaded concurrent data writing to the lake, improving the throughput of the integrated batch / stream storage system. In other words, a single lake synchronization service can provide one or more writer threads.

[0114] In another scenario, when multiple batch stream processing service partitions exist, each partition's lake synchronization service can have a write thread. This write thread handles data format conversion and lake-based processing. Specifically, each batch stream processing service partition can provide a lake synchronization service, and each lake synchronization service within that partition can provide one or more write threads, as shown in Figure 5, such as write thread 5021 in lake synchronization service 1, write thread 5022 in lake synchronization service 2, and write thread 5023 in lake synchronization service N. In this case, the batch stream processing server can determine the target batch stream processing service partition from the N partitions, write t queues of data into the service queue of the target partition, and then call the lake synchronization service within that partition. The write thread in the target partition's lake synchronization service converts the t queues of data into columnar storage format second business data. Of course, when there are multiple batch stream processing service partitions, data can also be processed using multiple threads. Specifically, the batch stream processing server can call the lake synchronization service in the target batch stream processing service partition. If the amount of data in t queues is greater than the data writing threshold, then the t queues are split into M business sub-data based on the data writing threshold. In the lake synchronization service of the target batch stream processing service partition, M write threads are used to perform parallel columnar storage conversion on the M business sub-data to obtain the second business data with columnar storage format corresponding to the M business sub-data. As shown in Figure 5, assuming that the target batch stream processing service partition is Partition2, the write thread 5022 in the lake synchronization service 2 can convert the t queues into second business data in columnar storage format.

[0115] Step S403: Write the second business data into the data lake through the lake synchronization service.

[0116] In this embodiment, the data lake is used to provide data for batch read requests initiated by batch processing clients. As shown in Figure 5, when storing multiple partitions, the second business data can be written to the data lake 503 through the write thread in the target batch processing service partition.

[0117] Specifically, the batch / stream processing server can write the second business data into the data lake through the write threads in the lake synchronization service. Alternatively, when processing data using multiple threads, the batch / stream processing server can use M write threads in the lake synchronization service to generate commit messages for one of the M second business data sets. Each of the M write threads can then request a write lock from the lake lock component in the data lake. The write thread that obtains the write lock can then write its corresponding second business data into the data lake. The write lock provides data write permissions; that is, a write thread with a write lock can write its corresponding second business data into the data lake. For example, referring to Figure 6, which is a schematic diagram of a data writing process scenario provided by an embodiment of this application, as shown in Figure 6, assuming M is 2, that is, there are writing threads 6011 and 6012, specifically including: S1, writing threads 6011 and 6012 send a write lock request to the lake lock component 602; S12, assuming writing thread 6011 acquires the write lock; S13, writing thread 6011 writes the second business data into the data lake 603 based on the write lock; S22, since writing thread 6012 did not request the write lock, writing thread 6012 will resend the write lock request to the lake lock component 602; S23, when writing thread 6012 acquires the write lock, it can execute S24 to write the second business data into the data lake 603.

[0118] When there are N batch stream processing service partitions, refer to Figure 7, which is a schematic diagram of another batch stream processing server scenario provided by an embodiment of this application. Each batch stream processing server may include a service queue and a lake synchronization service. Each lake synchronization service may include a write thread. As shown in Figure 7, Partition 1 includes service queue 7011 and lake synchronization service 1, which may include write thread 7021; Partition 2 includes service queue 7012 and lake synchronization service 2, which may include write thread 7022; ...; Partition N includes service queue 7013 and lake synchronization service N, which may include write thread 7023. Each lake synchronization service may include one or more write threads. The batch stream processing server can perform single-threaded or multi-threaded processing of data through the lake synchronization service. For example, assuming the target batch stream processing service partition is Partition2, Partition2 can request a write lock from the lake lock component 703 through the write thread 7022 in the lake synchronization service 2; the write thread 7022 in the lake synchronization service 2 writes the second business data into the data lake based on the write lock.

[0119] The write thread 7022 can include a single write thread. In this case, the batch stream processing server can perform single-threaded data processing on the data through the lake synchronization service in the target batch stream processing service partition. Alternatively, the write thread 7022 can include multiple write threads, such as M write threads. In this case, the batch stream processing server can perform multi-threaded data processing on the data through the lake synchronization service. Specifically, in the target batch stream processing service partition (such as Partition2 in this example), the batch stream processing server can request write locks from the lake lock component 703 through the M write threads in the lake synchronization service 2. Based on the acquired write locks, the M write threads in the lake synchronization service 2 sequentially write the second business data into the data lake. The order in which the second business data is written into the data lake is the order in which the M write threads acquired the write locks. Since the second business data corresponding to the M write threads is obtained from the same original data, it is generally written into a single table. In this case, the M write threads sequentially write the second business data into the data lake based on the order in which they acquired the write locks. Of course, it's also possible that the second business data corresponding to M write threads will be written to different tables. In this case, among the M write threads, the write thread that acquires the write lock for the same table can write the second business data to the data lake based on the order in which the write lock for that table is acquired. For example, suppose there are write threads 1 to 4, where write threads 1 and 4 need to write the second business data to table 1 of the data lake, and write threads 2 and 3 need to write the second business data to table 2 of the data lake. Write thread 1 acquires the write lock for table 1 before write thread 4, and write thread 3 acquires the write lock for table 2 before write thread 2. Then, after write thread 1 writes its corresponding second business data to table 1, write thread 4 writes its corresponding second business data to table 1; after write thread 3 writes its corresponding second business data to table 2, write thread 2 writes its corresponding second business data to table 2. The writing processes of write threads 1 and 4 do not affect the writing processes of write threads 2 and 3.

[0120] In this context, when write threads write data to the same table, they need to compete for write locks. That is, if multiple batch processing service partitions have write threads that need to write data to the same table, these threads can write their data to the data lake based on the order in which they acquire write locks for that table. For example, if write threads 1 and 2 in a batch processing service partition, and write thread 3 in a batch processing service partition, need to write their data to table 1 in the data lake, assuming the order in which they acquire write locks for table 1 is write thread 1, write thread 3, and write thread 2, then the write order for table 1 in this example is: write thread 1 writes its data to table 1 based on its acquired write lock -> write thread 3 writes its data to table 1 based on its acquired write lock -> write thread 2 writes its data to table 1 based on its acquired write lock.

[0121] Alternatively, when processing data using multiple threads, the M write threads in the lake synchronization service can each generate commit information for one of the M second business data items, obtaining the data write path corresponding to the M second business data items. This data write path represents the table to which the corresponding second business data item is to be written. The batch stream processing server can use the M write threads to submit the commit information corresponding to each of the M write threads to the coordinating node corresponding to the data write path. The coordinating node can be considered as a node that can store data, and can be denoted as zNode. Here, since the M second business data items are obtained from the same data (i.e., the original data), the data write path corresponding to the M second business data items is generally the same. In this case, the M second business data items correspond to one data write path. Of course, the M second business data items can also be partitioned into multiple tables. In this case, the M second business data items can correspond to multiple data write paths, and any write thread can submit the commit information corresponding to that write thread to the coordinating node of the data write path corresponding to that write thread. Furthermore, the coordinating node can be scanned by the commit thread. If commit information is found in the coordinating node, it will be written into the data lake. The coordinating node can then be scanned by the commit thread until no commit information is found in the coordinating node. In this system, the submission thread and each write thread can be considered to run independently. When writing data (such as the second business data) to the data lake, an intermediate storage node (i.e., a coordinating node) is introduced. This allows the batch stream processing server to submit the submission information to the coordinating node under the corresponding data write path without needing to request a lock. Subsequently, when writing data from the intermediate storage node to the data lake, data that needs to be written to the same data write path will be written to the data lake by the same submission thread, naturally without needing to request a lock. Even if a lock is needed, with only one thread, it is equivalent to simply requesting the lock and obtaining it, thus achieving lock-free or lock-less data entry processing. This can improve the throughput of data entry into the data lake for the batch stream processing server, especially under high concurrency. It can save resources used by the write thread to wait for locks or trigger submission retries when submission information fails, thereby improving the data entry performance and resource utilization of the batch stream processing server.

[0122] For example, please refer to Figure 8, which is a schematic diagram of a data write-to-lake processing scenario provided by an embodiment of this application. As shown in Figure 8, M write threads can submit their respective commit information to the coordination node corresponding to the data write path. Here, it is assumed that M is 3. The M write threads include write thread 8011, write thread 8012, and write thread 8013. Specifically, each write thread can submit its own commit information to the global coordination service 802, which is the coordination node corresponding to the data write path. The global coordination service 802 can be considered as a distributed application coordination service (such as ZooKeeper) used to manage the global commit information submitted by the batch stream processing server. The global commit information is used to indicate all commit information submitted by the batch stream processing server. The global coordination service 802 can include one or more coordination nodes, such as coordination node 8021 and coordination node 8022 shown in Figure 8. Optionally, when the data lake includes multiple tables, as shown in Figure 8, data lake 804 includes tables 8041 and 8042, etc. Each table corresponds to a data write path, and each data write path can be associated with one or more coordinating nodes. In this case, it can be considered that each write thread can submit its corresponding commit information to the global coordination service 802, and the coordinating node corresponding to the data write path of this commit information. Simply put, the i-th write thread can submit the i-th commit information corresponding to the i-th write thread to the coordinating node corresponding to the data write path of the i-th commit information in the global coordination service 802, where i is a positive integer less than or equal to M. Here, it is assumed that coordinating node 8021 is associated with table 8041, coordinating node 8022 is associated with table 8042, and M commit information needs to be written to table 8042, then all M commit information are submitted to coordinating node 8022.

[0123] The batch processing server can scan the coordination nodes in the global coordination service 802 using submission thread 803. If submission information is found in a coordination node, it is written to the data lake, and the server continues scanning coordination nodes using submission thread 803 until no submission information is found in the coordination node. Optionally, submission thread 803 can periodically scan the coordination nodes in the global coordination service 802. If submission information is found, it is written to the data lake; if no submission information is found, it waits for the next scan of the coordination nodes in the global coordination service 802.

[0124] Optionally, a commit thread 803 can scan all coordination nodes in the global coordination service 802. If commit information is found, it is written to the table corresponding to that commit information in the data lake. As shown in Figure 8, the commit thread 803 scans coordination nodes 8021 and 8022 in the global coordination service 802. If commit information is found in coordination node 8021, it is written to table 8041 associated with coordination node 8021. If commit information is found in coordination node 8022, it is written to table 8042 associated with coordination node 8022. If no commit information is found, the commit thread 803 waits for the next scan of the coordination nodes in the global coordination service 802.

[0125] Alternatively, each table in the data lake corresponds to a commit thread. In this case, each commit thread scans the coordinating node of the table it corresponds to. If the data lake includes d tables, the batch processing server can use the j-th commit thread to scan the coordinating node associated with the j-th table, where j is a positive integer less than or equal to d. If commit information is found, it is written to the j-th table; otherwise, it waits for a second scan of the coordinating node associated with the j-th table. As shown in Figure 8, commit thread 803 includes commit thread 1 corresponding to table 8041 and commit thread 2 corresponding to table 8042. The batch processing server can use commit thread 1 to scan the coordinating node 8021. If commit information is found, it is written to table 8041 via commit thread 1; otherwise, commit thread 1 waits for a second scan of node 8021. Similarly, the batch processing server can scan the coordinating node 8022 through submission thread 2. If submission information is found, it is written to table 8042 through submission thread 2; if no submission information is found, submission thread 2 waits to scan node 8022 again.

[0126] When there are N batch stream processing service partitions, refer to Figure 9, which is a schematic diagram of another batch stream processing server processing scenario provided by the embodiments of this application. Figure 9 adds a global coordination service 901 to Figure 7. Specifically, assuming the target batch stream processing service partition is Partition2, Partition2 can submit the commit information corresponding to the write thread 7022 in the lake synchronization service 2 to the coordination node corresponding to the data write path of the global coordination service 901 through the write thread 7022. The write thread 7022 can include a single write thread. In this case, the batch stream processing server can perform single-threaded data processing on the data through the lake synchronization service in the target batch stream processing service partition. Alternatively, the write thread 7022 can include multiple write threads, such as M write threads. In this case, the batch stream processing server can perform multi-threaded data processing on the data through the lake synchronization service. Specifically, the batch stream processing server can submit the commit information corresponding to the M write threads to the global coordination service 901 and the coordination node corresponding to the data write path through the M write threads in the target batch stream processing service partition (such as Partition2 in this example). When writing data to the data lake, it is written to a table in the data lake. Therefore, the coordination nodes in the global coordination service 901 can be grouped based on tables, that is, a table can be associated with one or more coordination nodes. Furthermore, the batch / stream processing server can scan the coordination nodes in the global coordination service 901 via a submission thread. When submission information is detected, the submission thread can request a write lock from the lake lock component 703 and write the detected submission information into the data lake based on the requested write lock; alternatively, the submission thread can write the detected submission information into the data lake. The process of the submission thread scanning the coordination nodes in the global coordination service 901 can be described in Figure 8 above. The coordination nodes in the global coordination service 901 can include submission information submitted by one or more write threads in each batch / stream processing service partition.

[0127] Optionally, the number of data lakes is K, where K is a positive integer; each data lake corresponds to a specific lake format. For example, the K data lakes can include, but are not limited to, batch / stream processing data lakes (BS), Iceberg, and Paimon. Iceberg is a high-performance tabular data lake used for big data analytics, supporting efficient querying and event travel, and is compatible with multiple data processing engines. It can be used for large-scale dataset management, providing data consistency and high-performance query capabilities. Paimon is a high-performance data lake storage system that enables both stream and batch processing, optimizing data storage and query performance, and ensuring data consistency. Specifically, when writing the second business data into the data lake, the batch / stream processing server can obtain the lake configuration data from the lake synchronization service, parse the second business data, and determine the data storage parameters for the second business data. This lake configuration data instructs the batch / stream processing server to write the data to the appropriate data lake. Further, based on the lake configuration data, the target data lake corresponding to the data storage parameters can be determined from the K data lakes, and the second business data can be written to the target data lake through the lake synchronization service. This process can be used in any of the embodiments shown in Figures 5 to 9 above.

[0128] In this embodiment of the application, the efficiency and performance of lake landing processing can be improved by multi-threaded processing and unified submission (i.e., the use of submission threads).

[0129] Further, please refer to Figure 10, which is a flowchart of another data processing method provided in an embodiment of this application. As shown in Figure 10, this method is executed by a batch processing client, and the data processing process includes the following steps:

[0130] Step S1001: Obtain the raw data sent by the data processing engine, and the data record information generated by the data processing engine for the raw data.

[0131] In this embodiment, the batch stream processing client can obtain the raw data sent by the data processing engine, as well as the data record information generated by the data processing engine for the raw data. Further, the batch stream processing client can execute step S1002. Alternatively, the batch stream processing client can obtain write configuration parameters. If the write configuration parameter is a first write configuration parameter, then step S1002 is executed to obtain the structure information of the raw data; the write configuration parameter is used to indicate the data storage method; if the write configuration parameter is a second write configuration parameter, then the raw data is written to the data lake. In this way, the batch stream processing client can synchronously write data to the service queue and the data lake based on the first write configuration parameter, and directly write data to the data lake based on the second write configuration parameter. This allows users to directly use the data lake through simple write configuration parameters, improving the flexibility of data storage.

[0132] Step S1002: Obtain the structure information of the original data, and generate t queue data based on the structure information and data record information.

[0133] In this embodiment, when the batch stream processing client sends queue data one by one (i.e., t = 1), the client can serialize the data record information according to the structure information. For each queue data item obtained, step S1003 is executed, sending t queue data items to the batch stream processing server. When the batch stream processing client sends queue data in batches (i.e., t > 1), the client can serialize the data record information according to the structure information. For each queue data item generated, it is cached. If the number of cached queue data items is greater than or equal to the number of batch queue elements, or if the sending wait time is greater than or equal to the data sending cycle, step S1003 is executed, sending the cached queue data to the batch stream processing server.

[0134] Specifically, the structural information can include queue structure information and message structure information. The batch stream processing client can parse the raw data and construct queue structure information for the raw data. This queue structure information is used to split the raw data into row-oriented data, that is, to break the raw data into individual data entries. Further, the batch stream processing client can construct message structure information based on the queue structure information and data record information, and then serialize the data record information based on the message structure information to obtain t queue data entries sequentially. The message structure information represents the data parameters that make up the queues. Specifically, the batch stream processing client can parse the raw data and construct queue structure information for the raw data; based on the queue structure information, the raw data is split into A sub-data entries, where A is a positive integer; based on the queue structure information, the A sub-data entries are parsed to obtain data parameters, which represent the data composition of each sub-data entry. As shown in Table 1, these data parameters can be considered to include name, student ID, class, and contact information, etc.; the parameter values ​​corresponding to the data parameters are sequentially obtained from the A sub-data entries to obtain the queue data corresponding to each sub-data entry. The A queue data entries constitute the first business data. Specifically, when the batch stream processing client sends queue data one by one, the batch stream processing client executes step S1003 every time it generates a queue data; when the batch stream processing client sends queue data in batches, the batch stream processing client executes step S1003 every time it generates queue data corresponding to the number of batch queue elements, or every data sending cycle.

[0135] Optionally, the number of data processing engines is H; H is a positive integer. The batch / stream processing client can invoke the data write function associated with the data processing engine that sends the raw data, and obtain the structural information of the raw data through the data write function; the H data processing engines include those used to send the raw data. Through the data write function, the structural information and data record information are converted into the first business data; this process can be referred to the process described in the previous paragraph.

[0136] Step S1003: Send t queue data to the batch stream processing server so that the batch stream processing server writes the t queue data into the service queue and converts the t queue data into second business data in columnar storage format through the lake synchronization service.

[0137] In this embodiment, the second business data is written to the data lake by the batch stream processing server through the lake synchronization service; the service queue is used to provide data for data stream read requests initiated by the batch stream processing client; the data lake is used to provide data for data batch read requests initiated by the batch stream processing client. Specifically, if t is 1, the batch stream processing client can send t queue data to the batch stream processing server; if t is greater than 1, the batch stream processing client can integrate t queue data into a batch of data and send the batch data to the batch stream processing server.

[0138] Furthermore, the batch / stream processing client can perform data reading. Specifically, the batch / stream processing client can respond to data reading requests sent by the data processing engine, obtaining the data reading method and data reading parameters from the data reading request. These data reading parameters are obtained by the data processing engine from parsing the structured query statement, or they can be generated by the data processing engine. Based on the data reading method, the batch / stream processing client can determine the region to be read from the batch / stream processing server and the data lake, and based on the data reading parameters, obtain the third-party business data requested by the data reading request from the region; then, it sends the third-party business data to the data processing engine.

[0139] Specifically, when determining the region to be read from the batch / stream processing server and the data lake based on the data reading method, and retrieving the third business data requested by the data read request from the region based on the data reading parameters, if the data reading method is stream reading, the batch / stream processing client can determine the batch / stream processing server as the region to be read, send a data read request to the batch / stream processing server, and obtain the third business data returned by the batch / stream processing server based on the data read request; the third business data is obtained by the batch / stream processing server from the service queue based on the data read parameters. If the data reading method is batch reading, the batch / stream processing client can determine the data lake as the region to be read, and retrieve the third business data requested by the data read request from the data lake based on the data read parameters.

[0140] Furthermore, if the entire original data is not sent to the batch stream processing server, the batch stream processing client can continue to serialize the data record information according to the structure information to obtain the next queue data, and then send the obtained queue data to the batch stream processing server until all queue data corresponding to the original data is obtained. This process can be referred to in steps S1002 to S1003, and will not be described again here. All queue data corresponding to the original data constitute the first business data; the first business data is in row-based storage format.

[0141] Of course, data is continuously generated in stream processing. Here, we take a single raw data set as an example. The batch stream processing client can sequentially send all queue data corresponding to the raw data to the batch stream processing server. In the specific implementation, the batch stream processing client can serialize the raw data, sequentially obtain and send the queue data corresponding to the raw data. When all queue data corresponding to the raw data is sent to the batch stream processing server, the batch stream processing client can obtain the next raw data set. If the next raw data set is not obtained, the batch stream processing client enters a data processing waiting state until it receives new raw data. For the new raw data set, steps S1002 to S1003 are executed. If the next raw data set is obtained, the batch stream processing client can execute steps S1002 to S1003 for the next raw data set. In other words, when there is data to be sent, the batch stream processing client serializes the data to be sent to obtain queue data and sends the queue data to the batch stream processing server. When there is no data to be sent, it enters a data processing waiting state and continues to serialize the new data when it arrives. This allows the batch stream processing client to send the acquired data to the batch stream processing server as soon as it obtains the data, thereby realizing real-time writing of the acquired data to the batch stream processing server.

[0142] In this embodiment, a batch stream processing client can be constructed, which connects the data processing engine and the batch stream processing server. When the data processing engine obtains raw data, it can send the data record information for the raw data to the batch stream processing client. The batch stream processing client generates first business data based on the structure information of the raw data and the data record information for the raw data. The first business data includes multiple queue data, and when the queue data is obtained, it sends the queue data to the batch stream processing server. The batch stream processing server can write the obtained queue data into a service queue and can call the lake synchronization service to convert the obtained queue data into second business data in columnar storage format. The second business data is then written into the data lake through the lake synchronization service. By constructing a batch-stream processing client and integrating a data lake synchronization service into the batch-stream processing server, unified batch-stream and data access is achieved. Specifically, the raw data sent by the data processing engine is synchronously stored based on a service queue and a data lake. This allows subsequent data retrieval to be performed by reading data from the service queue based on stream read requests and from the data lake based on batch read requests. This ensures real-time data stream processing based on the service queue, resolving issues such as high latency and sequentiality in data lake stream writing and reading, thus improving the efficiency and real-time performance of data stream processing. Simultaneously, the use of the data lake is retained, enabling the processing of various data types and handling of large amounts of data while ensuring data integrity. In this embodiment, both a service queue and a data lake are used for data retrieval, allowing the application to retain the characteristics of both. Data can be retrieved from either the service queue or the data lake as needed, thereby improving the efficiency and flexibility of data processing.

[0143] Further, please refer to Figure 11, which is an interactive flowchart of a data writing method provided in an embodiment of this application. As shown in Figure 11, the data writing process includes the following steps:

[0144] Step S1101, access.

[0145] In this embodiment of the application, the business object (i.e., the user) can access the data processing engine.

[0146] Step S1102: Generate raw data.

[0147] In this embodiment of the application, the data processing engine can generate raw data during operation. At this time, the data processing engine can execute step S1103.

[0148] Step S1103: Obtain the data record information of the original data.

[0149] In this embodiment of the application, the data processing engine can obtain data record information of the original data. The data record information is used to represent the complete information of the original data and can be any form of data, including but not limited to text, numbers and dates. The data record information can be a composite data type. For example, see the example of data record information based on the original data shown in Table 1. Specifically, the data record information can be seen in the relevant description in step S401 of Figure 4.

[0150] Step S1104: Send the data record information of the original data.

[0151] In this embodiment of the application, the data processing engine can send data record information of the raw data to the batch stream processing client.

[0152] Step S1105: Generate queue data based on the structure information and data record information of the original data.

[0153] In this embodiment, the batch stream processing client can generate queue data based on the structure information and data record information of the original data. Specifically, the batch stream processing client can parse the original data to construct queue structure information for the original data; based on the queue structure information, the original data is split into A sub-data, where A is a positive integer; based on the queue structure information, the A sub-data are parsed to obtain data parameters; the parameter values ​​corresponding to the data parameters are sequentially obtained from the A sub-data to obtain queue data, and the queue data corresponding to the A sub-data respectively constitute the first business data. This process can be referred to the relevant description in step S1002 of Figure 10, and will not be repeated here.

[0154] Optionally, the batch stream processing client can serialize the data record information according to the structure information of the original data to obtain a queue of data. If the batch stream processing client sends the queue of data one by one, then step S1107 is executed to send the queue of data to the batch stream processing server; if the batch stream processing client sends the queue of data in batches, then step S1106 is executed to cache the queue of data.

[0155] Optionally, if the batch processing client has finished processing the original data and has not received any new original data, the batch processing client can enter a data processing waiting state until it receives new original data. Based on the structure information and data record information of the new original data, queue data is then generated. If the batch processing client has finished processing the original data and has received new original data, the batch processing client can generate queue data based on the structure information and data record information of the new original data. If the batch processing client has not finished processing the original data, the batch processing client can generate queue data based on the structure information and data record information of the original data. The process of generating queue data for either original data or new original data can be found in the relevant description in step S1002 of Figure 10.

[0156] Step S1106: Cache queue data.

[0157] In this embodiment, the batch stream processing client can cache the queue data. Further, if the number of cached queue data is greater than or equal to the number of batch queue elements, or the sending wait time is greater than or equal to the data sending cycle, then step S1107 is executed to send the cached queue data to the batch stream processing server; if the number of cached queue data is less than the number of batch queue elements, and the sending wait time is less than the data sending cycle, then step S1105 is returned to be executed to generate the next queue data.

[0158] Step S1107: Send queue data.

[0159] In this embodiment, the batch streaming client can send queue data to the batch streaming server. Optionally, as shown in Figure 1, the batch streaming client can send queue data to the batch streaming server through the core processing unit. Optionally, the batch streaming client can directly send t queue data to the batch streaming server; or, the batch streaming client can determine the target batch streaming service partition from the N batch streaming service partitions included in the batch streaming server and send t queue data to the target batch streaming service partition. The process of determining the target batch streaming service partition can be referred to the relevant description in step S402 of Figure 4.

[0160] Optionally, when sending one queue of data (t = 1), the batch stream processing client can send t queues of data to the batch stream processing server. When sending multiple queues of data (t > 1), the batch stream processing client can merge the t queues of data into a batch and send the batch data to the batch stream processing server. Optionally, the batch stream processing client can also return to step S1105 to continue generating the next queue of data.

[0161] Optionally, the core processing unit can also be used only to generate queues. Specifically, the batch stream processing client can execute steps S1105 and S1106 through the core processing unit to obtain queue data, and then send the queue data to the batch stream processing server.

[0162] Step S1108: Convert the queue data into second business data in columnar storage format.

[0163] In this embodiment, the batch streaming server can write t queue data into a service queue and convert the t queue data into second business data in columnar storage format. Specifically, the batch streaming server can convert the t queue data into second business data in columnar storage format through a lake synchronization service.

[0164] When the batch processing server receives batch data, it can parse the batch data to obtain t queue data, write the t queue data into the service queue, and convert the t queue data into second business data in columnar storage format.

[0165] Specifically, when the batch stream processing server directly receives t queues of data, it can determine the target batch stream processing service partition from the N batch stream processing service partitions included in the server. Within the target batch stream processing service partition, the t queues of data are converted into second business data in columnar storage format. Conversely, when the batch stream processing server receives t queues of data through the target batch stream processing service partition, it can also convert the t queues of data into second business data in columnar storage format within the target batch stream processing service partition.

[0166] The above process can also be referred to in the relevant description of step S402 in Figure 4, which will not be repeated here.

[0167] Step S1109: Write the second business data into the data lake.

[0168] In this embodiment of the application, the batch processing server can write the second business data into the data lake, and the process can be referred to the relevant description in step S403 of Figure 4.

[0169] In this embodiment, the BSS client enables unified access to the BSS server and the data lake, providing a means for the data processing engine to interact with the BSS server and the data lake, thereby achieving integrated batch and stream storage of data and improving the flexibility and efficiency of data storage.

[0170] Further, please refer to Figure 12, which is an interactive flowchart of a data reading method provided in an embodiment of this application. As shown in Figure 12, the data reading process includes the following steps:

[0171] Step S1201: Send a data read request.

[0172] In this embodiment, the data processing engine can send a data read request to the batch / stream processing client. This data read request can be generated by the data processing engine or triggered by a business object. Specifically, when data is missing during runtime, the data processing engine can obtain the missing data parameters, determine these parameters as data read parameters, and send a data read request to the batch / stream processing client based on these parameters. Alternatively, the data processing engine can respond to a data read operation triggered by a business object, parse the operation to obtain the data read parameters, and send the request to the batch / stream processing client based on these parameters. Or, the data processing engine can obtain a structured query language (SQL) statement submitted by a business object, parse it to obtain the data read parameters, and send the request to the batch / stream processing client based on these parameters. This method enables the BSS client to support SQL, allowing users to directly read data using SQL, thereby simplifying data read and write operations and improving the interactive performance of the data processing engine.

[0173] Step S1202: Obtain the data reading method and data reading parameters from the data reading request.

[0174] In this embodiment of the application, the batch-stream processing client can obtain the data reading method and data reading parameters from the data reading request. Further, if the data reading method is a stream reading method, then step S1203 is executed; if the data reading method is a batch reading method, then step S1205 is executed.

[0175] Step S1203: If the data reading method is stream reading, then send a data reading request.

[0176] In this embodiment of the application, if the data reading method is streaming, the batch streaming client can send the data reading request to the batch streaming server.

[0177] Step S1204: Send a data read request for the requested third service data.

[0178] In this embodiment, the batch stream processing server can retrieve the third business data associated with the data read parameters from the service queue based on the data read request, and send the third business data to the batch stream processing client. Specifically, the batch stream processing server can determine the data positioning point based on the data read parameters, start reading from the data positioning point in the service queue, send the read third business data to the batch stream processing client, and determine the next position of the read third business data in the service queue as the data positioning point. It continues to read data from the service queue based on the data positioning point until there is no data in the service queue at the data positioning point, or a stream read termination command is received. Further, if there is no data in the service queue at the data positioning point, the batch stream processing server enters a read waiting state. When new business data is written to the service queue, the batch stream processing server can continue to read the new business data from the service queue based on the data positioning point and send the new business data to the batch stream processing client. Based on the reading status of the service queue, the data positioning point is updated. This method enables real-time data reading, thereby improving data real-time performance. Further batch processing clients can execute step S1206.

[0179] Step S1205: If the data reading method is batch reading, then based on the data reading parameters, obtain the third business data requested by the data reading request from the data lake.

[0180] In this embodiment, if the data reading method is batch reading, the batch stream processing client can obtain the third business data requested by the data reading request from the data lake based on the data reading parameters. Specifically, the batch stream processing client can determine the reading position based on the data reading parameters. This reading position indicates the data lake to which the data requested by the data reading request belongs, and the table to which it belongs in the data lake. The batch stream processing client can obtain the third business data from the reading position in the data lake. Further, the batch stream processing client can execute step S1206.

[0181] Step S1206: Send the third service data.

[0182] In this embodiment of the application, the batch processing client can send third-party business data to the data processing engine.

[0183] The embodiments shown in Figures 11 and 12 enable integrated batch and stream data access and processing, providing high-throughput and low-latency storage services. This meets the performance and timeliness requirements of data reading during business processing, while saving storage and maintenance costs. It can also solve problems related to streaming writes, streaming reads, and sequentiality in data lakes.

[0184] Further, please refer to Figure 13, which is a schematic diagram of a data processing device provided in an embodiment of this application. This data processing device can be a computer program (including program code, etc.) running on a computer device; for example, the data processing device can be an application software. The device can be used to execute the corresponding steps in the method provided in the embodiment of this application. As shown in Figure 13, the data processing device 1300 can be used in the batch-stream processing server in the embodiment corresponding to Figure 4. Specifically, the device may include: a queue acquisition module 11, a data conversion module 12, and a data import module 13.

[0185] The queue acquisition module 11 is used to acquire t queue data sent by the batch stream processing client; the batch stream processing client provides an access interface for the data processing engine; each queue data is a piece of data in the first business data generated by the batch stream processing client for the original data; t is a positive integer; the first business data is generated by the batch stream processing client based on the structure information of the original data and the data record information for the original data sent by the data processing engine; the first business data is in row storage format.

[0186] Data conversion module 12 is used to write t queue data into the service queue, call the lake synchronization service, and convert the t queue data into second business data in columnar storage format through the lake synchronization service; the service queue is used to provide data for data stream read requests initiated by batch stream processing clients;

[0187] The data lake module 13 is used to write the second business data into the data lake through the lake synchronization service; the data lake is used to provide data for data batch read requests initiated by the batch processing client.

[0188] The data conversion module 12 can be used for:

[0189] From the N batch stream processing service partitions included in the batch stream processing server, determine the target batch stream processing service partition, and write t queue data into the service queue in the target batch stream processing service partition; N is a positive integer;

[0190] Call the lake synchronization service in the target batch stream processing service partition, and convert t queue data into second business data in columnar storage format through the lake synchronization service in the target batch stream processing service partition.

[0191] Among them, the N batch stream processing service partitions correspond to different partition values; when determining the target batch stream processing service partition from the N batch stream processing service partitions included in the batch stream processing server, the data conversion module 12 can be used to:

[0192] The batch stream processing server performs round-robin processing on its N batch stream processing service partitions to obtain the target batch stream processing service partition for t queues of data; or,

[0193] Obtain the queue processing parameters, perform a hash transformation on the first business data using the queue processing parameters to obtain the business hash value, and perform a modulo operation on the business hash value using N to obtain the target partition value; the queue processing parameters are preset hash values ​​used for hash transformation;

[0194] Among the N batch stream processing service partitions, the batch stream processing service partition corresponding to the target partition value is determined as the target batch stream processing service partition.

[0195] Specifically, when calling the lake synchronization service to convert t queue data into second business data in columnar storage format, the data conversion module 12 can be used for:

[0196] When calling the lake synchronization service, if the amount of data in the first business data in t queues is greater than the data write threshold, then the t queues will be split into M business sub-data based on the data write threshold; M is a positive integer.

[0197] In the lake synchronization service, M write threads are used to perform parallel columnar storage conversion on M business sub-data to obtain second business data with columnar storage format corresponding to each of the M business sub-data.

[0198] The data landing module 13 can be used for:

[0199] The M write threads in the lake synchronization service generate commit information for one of the M second business data, and obtain the data write path corresponding to the M second business data.

[0200] Using M write threads, the commit information corresponding to each of the M write threads is submitted to the coordinating node corresponding to the data write path.

[0201] The commit thread scans the coordinating node. If commit information is found in the coordinating node, it is written into the data lake. The commit thread continues to scan the coordinating node until no commit information is found in the coordinating node.

[0202] The number of data lakes is K, where K is a positive integer; one data lake corresponds to one type of lake.

[0203] The data landing module 13 can be used for:

[0204] Obtain the lake configuration data from the lake synchronization service, parse the second business data, and determine the data storage parameters of the second business data;

[0205] Based on the lake configuration data, the target data lake corresponding to the data storage parameters is determined from K data lakes, and the second business data is written to the target data lake through the lake synchronization service.

[0206] Specifically, when writing t queue data into the service queue, the data conversion module 12 can be used for:

[0207] Obtain the business object information that generates the raw data, and determine the data storage space associated with the business object information from the data storage space included in the batch stream processing server; the batch stream processing server includes multiple mutually isolated data storage spaces;

[0208] Write the t queue data into the service queue in the data storage space associated with the business object information in sequence.

[0209] Further, please refer to Figure 14, which is a schematic diagram of another data processing device provided in an embodiment of this application. This data processing device can be a computer program (including program code, etc.) running on a computer device; for example, the data processing device can be an application software. The device can be used to execute the corresponding steps in the method provided in the embodiment of this application. As shown in Figure 14, the data processing device 1400 can be used in the batch processing client of the embodiment corresponding to Figure 10. Specifically, the device may include: a data acquisition module 21, a data construction module 22, and a queue generation module 23.

[0210] The data acquisition module 21 is used to acquire the raw data sent by the data processing engine, as well as the data record information generated by the data processing engine for the raw data;

[0211] Data construction module 22 is used to obtain the structural information of the original data and generate t queue data based on the structural information and data record information; t is a positive integer.

[0212] The queue generation module 23 is used to send t queue data to the batch stream processing server, so that the batch stream processing server writes the t queue data into the service queue, and converts the t queue data into second business data in columnar storage format through the lake synchronization service; the second business data is written to the data lake by the batch stream processing server through the lake synchronization service; the service queue is used to provide data for data stream read requests initiated by the batch stream processing client; the data lake is used to provide data for data batch read requests initiated by the batch stream processing client.

[0213] The structural information includes queue structure information and message structure information;

[0214] This data building module 22 can be used for:

[0215] The original data is parsed to construct queue structure information for the original data; the queue structure information is used to split the original data into row-based storage format data.

[0216] Message structure information is constructed based on queue structure information and data record information. Data record information is then serialized based on message structure information to obtain t queue data in sequence.

[0217] The number of data processing engines is H; H is a positive integer.

[0218] This data building module 22 can be used for:

[0219] Call the data writing function associated with the data processing engine that sends the raw data, and obtain the structural information of the raw data through the data writing function; H data processing engines include the data processing engine used to send the raw data;

[0220] The data writing function converts the structural information and data record information into t queues of data.

[0221] The device 1400 also includes:

[0222] Request parsing module 24 is used to respond to data read requests sent by the data processing engine and obtain the data read method and data read parameters from the data read request; the data read parameters are obtained by the data processing engine from parsing the structured query statement;

[0223] The data reading module 25 is used to determine the area to be read from the batch stream processing server and the data lake based on the data reading method, and to obtain the third business data requested by the data reading request from the area to be read based on the data reading parameters.

[0224] The data sending module 26 is used to send third-party business data to the data processing engine.

[0225] The data reading module 25 can be used for:

[0226] If the data reading method is streaming, then the batch stream processing server is identified as the area to be read, a data read request is sent to the batch stream processing server, and the third business data returned by the batch stream processing server based on the data read request is obtained; the third business data is obtained by the batch stream processing server from the service queue based on the data read parameters.

[0227] If the data reading method is batch reading, then the data lake is identified as the area to be read, and the third business data requested by the data reading request is obtained from the data lake based on the data reading parameters.

[0228] The device 1400 also includes:

[0229] The write processing module 27 is used to obtain write configuration parameters. If the write configuration parameter is the first write configuration parameter, the step of obtaining the structure information of the original data is executed. The write configuration parameter is used to indicate the data storage method.

[0230] The write processing module 27 is also used to write the original data into the data lake if the write configuration parameter is the second write configuration parameter.

[0231] This application provides a data processing apparatus. The apparatus shown in FIG13 can run in a batch stream processing server, and the apparatus shown in FIG14 can run in a batch stream processing client. The batch stream processing client connects the data processing engine and the batch stream processing server. When the data processing engine obtains raw data, it can send the data record information for the raw data to the batch stream processing client. The batch stream processing client generates first business data based on the structure information of the raw data and the data record information for the raw data. The first business data includes multiple queue data. When the queue data is obtained, the client sends the queue data to the batch stream processing server. The batch stream processing server can write the obtained queue data into a service queue and can call the lake synchronization service to convert the obtained queue data into second business data in columnar storage format. The second business data is then written into the data lake through the lake synchronization service. By constructing a batch-stream processing client and integrating a data lake synchronization service into the batch-stream processing server, unified batch-stream and data access is achieved. Specifically, the raw data sent by the data processing engine is synchronously stored based on a service queue and a data lake. This allows subsequent data retrieval based on stream read requests from the service queue and batch read requests from the data lake. Due to the first-in-first-out (FIFO) and read-as-you-go characteristics of the service queue, real-time data stream processing is ensured, resolving issues such as high latency and sequentiality in data lake stream writing and reading, thus improving the efficiency and real-time performance of data stream processing. Simultaneously, the use of the data lake is retained, enabling the processing of various data types and handling of large volumes of data while ensuring data integrity. In this embodiment, the simultaneous use of a service queue and a data lake for data retrieval processing allows the application to retain the characteristics of both. Data can be retrieved from either the service queue or the data lake as needed, thereby improving the efficiency and flexibility of data processing.

[0232] Figures 15a to 15d illustrate the test results of this application (i.e., BSS), Iceberg, and Paimon methods. Figure 15a is a comparative diagram of the read / write throughput (Transactions Per Second, TPS) of this application and a data lake, provided by an embodiment of this application. As shown in Figure 15a, the write TPS and read TPS of this application (i.e., integrated batch and stream reading), the Iceberg non-partitioned scheme, the Iceberg partitioned scheme, the Paimon non-partitioned scheme, and the Paimon partitioned scheme are displayed respectively. Figure 15a shows that this application achieves an increase in both write TPS and read TPS compared to the data lake.

[0233] Figure 15b shows a comparison of the streaming read latency of this application and a data lake, provided by an embodiment of this application. As shown in Figure 15b, the unit is denoted as seconds (s), which is used to represent the streaming read latency corresponding to this application (i.e., batch and streaming read integration), Iceberg non-partitioned scheme, Iceberg partitioned scheme, Paimon non-partitioned scheme and Paimon partitioned scheme, respectively. As can be seen from Figure 15b, the streaming read latency of this application is as low as 1s, which is a significant reduction compared to the data lake. In other words, this application reduces the streaming read latency.

[0234] Figure 15c shows a comparison of the system throughput of this application and a data lake in an embodiment of this application. As shown in Figure 15c, the unit is denoted as megabytes per second (MB / s), which is used to represent the throughput of this application (i.e., batch and stream reading), Iceberg, and Paimon when transmitting small messages and large messages, respectively. Here, a small message is a message with a single message size of 213 bytes (i.e., 213B), and a large message is a message with a single message size of 4 megabytes (i.e., MB). As can be seen from Figure 15c, the throughput of this application is greatly increased compared to the data lake, whether transmitting small messages or large messages.

[0235] Figure 15d is a schematic diagram comparing the throughput of the client and server during data processing according to an embodiment of this application. As shown in Figure 15d, the unit is denoted as gigabytes per second (GB / s). It represents the throughput of the BSS client when writing messages of different data volumes to the BSS server during message transmission, and the throughput of the lake synchronization service when writing messages of different data volumes to the data lake. As can be seen from Figure 15d, in this application, the throughput of the BSS client when writing data to the BSS server is higher. When using the lake synchronization service, by performing multi-threaded processing of data and single-threaded (i.e., submission thread) writing of data to the data lake, the throughput of the lake synchronization service is further improved compared to the throughput of the BSS client when writing data to the BSS server. On this basis, when the BSS server synchronously writes queue data to the data lake, it can adapt to the throughput of the BSS client writing data to the BSS server, thereby improving the efficiency of data synchronization.

[0236] This demonstrates that, compared to a data lake, this application can improve data throughput and reduce latency.

[0237] Referring to Figure 16, which is a schematic diagram of the structure of a computer device provided in an embodiment of this application, the computer device in this embodiment may include one or more processors 1601, a memory 1602, and an input / output interface 1603. The processor 1601, memory 1602, and input / output interface 1603 are connected via a bus 1604. The memory 1602 stores a computer program, which includes program instructions. The input / output interface 1603 receives and outputs data, such as for data interaction between a data processing engine and a batch / stream processing client, or between a batch / stream processing client and a batch / stream processing server, or between a batch / stream processing client and a data lake, or between a batch / stream processing server and a data lake. The processor 1601 executes the program instructions stored in the memory 1602.

[0238] When the processor 1601 is used in a batch stream processing server, it can perform the following operations:

[0239] Retrieve t queue data sent by the batch stream processing client; the batch stream processing client provides an access interface for the data processing engine; each queue data is a piece of data in the first business data generated by the batch stream processing client for the original data; t is a positive integer; the first business data is generated by the batch stream processing client based on the structure information of the original data and the data record information for the original data sent by the data processing engine; the first business data is in row storage format.

[0240] Write t queue data into the service queue, call the lake synchronization service, and convert the t queue data into second business data in columnar storage format through the lake synchronization service; the service queue is used to provide data for data stream read requests initiated by batch stream processing clients;

[0241] The second business data is written into the data lake through the lake synchronization service; the data lake is used to provide data for batch read requests initiated by batch processing clients.

[0242] When the processor 1601 is used in a batch stream processing client, it can perform the following operations:

[0243] Obtain the raw data sent by the data processing engine, as well as the data record information generated by the data processing engine based on the raw data;

[0244] Obtain the structural information of the original data, and generate t queue data based on the structural information and data record information; t is a positive integer;

[0245] t queues of data are sent to the batch stream processing server, which then writes the t queues of data into a service queue. The server then uses a lake synchronization service to convert the t queues of data into second business data in columnar storage format. This second business data is written to the data lake by the batch stream processing server via the lake synchronization service. The service queue provides data for data stream read requests initiated by the batch stream processing client. The data lake provides data for batch read requests initiated by the batch stream processing client.

[0246] Continue generating the next queue data based on the structure information and data record information, until all queue data corresponding to the original data are obtained; all queue data corresponding to the original data constitute the first business data; the first business data is data in row storage format.

[0247] In some feasible implementations, the processor 1601 may be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0248] The memory 1602 may include read-only memory and random access memory, and provides instructions and data to the processor 1601 and the input / output interface 1603. A portion of the memory 1602 may also include non-volatile random access memory. For example, the memory 1602 may also store device type information.

[0249] In practice, the computer device can execute the implementation methods provided in the steps of Figure 4 or Figure 10 through its built-in functional modules. For details, please refer to the implementation methods provided in the steps of Figure 4 or Figure 10, which will not be repeated here.

[0250] This application embodiment provides a computer device including a processor, an input / output interface, and a memory. The processor retrieves a computer program from the memory and executes the steps of the method shown in Figure 4 or Figure 10 to perform data processing operations. This application embodiment implements the construction of a batch stream processing client, which connects to a data processing engine and a batch stream processing server. When the data processing engine obtains raw data, it can send data record information for the raw data to the batch stream processing client. The batch stream processing client generates first business data based on the structure information of the raw data and the data record information for the raw data. The first business data includes multiple queue data, and when the queue data is obtained, it is sent to the batch stream processing server. The batch stream processing server can write the obtained queue data into a service queue and can call a lake synchronization service to convert the obtained queue data into second business data in columnar storage format. The second business data is then written into a data lake through the lake synchronization service. By constructing a batch-stream processing client and integrating a data lake synchronization service into the batch-stream processing server, unified batch-stream and data access is achieved. Specifically, the raw data sent by the data processing engine is synchronously stored based on a service queue and a data lake. This allows subsequent data retrieval based on stream read requests from the service queue and batch read requests from the data lake. Due to the first-in-first-out (FIFO) and read-as-you-go characteristics of the service queue, real-time data stream processing is ensured, resolving issues such as high latency and sequentiality in data lake stream writing and reading, thus improving the efficiency and real-time performance of data stream processing. Simultaneously, the use of the data lake is retained, enabling the processing of various data types and handling of large volumes of data while ensuring data integrity. In this embodiment, the simultaneous use of a service queue and a data lake for data retrieval processing allows the application to retain the characteristics of both. Data can be retrieved from either the service queue or the data lake as needed, thereby improving the efficiency and flexibility of data processing.

[0251] This application also provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the data processing method provided in the steps of FIG4 or FIG10. Specific implementations of the steps in FIG4 or FIG10 are described below. Furthermore, the beneficial effects of using the same method are also not described in detail here. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application. As an example, the computer program can be deployed to execute on a single computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed across multiple locations and interconnected via a communication network.

[0252] The computer-readable storage medium can be the data processing apparatus provided in any of the foregoing embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0253] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional modes shown in Figures 4 or 10. This achieves integrated batch and stream data access. Specifically, this application synchronously stores the raw data sent by the data processing engine based on a service queue and a data lake. This allows subsequent data retrieval based on data stream read requests from the service queue and data retrieval based on data batch read requests from the data lake. This ensures the real-time performance of data stream processing based on the service queue, solving problems such as high latency and sequentiality in data lake stream writing and reading, and improving the efficiency and real-time performance of data stream processing. Simultaneously, the use of the data lake is retained, enabling this application to process data of various data types, enabling the processing of large amounts of data and ensuring data integrity. In this embodiment, both service queues and data lakes are used for data reading and processing, which allows the application to retain the characteristics of both service queues and data lakes. When reading data, data can be read from either the service queue or the data lake as needed, thereby improving the efficiency and flexibility of data processing.

[0254] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0255] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0256] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0257] The methods and related apparatus provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowcharts and / or structural diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, create means for implementing the functions specified in one or more blocks of the flowcharts and / or one or more blocks of the structural diagrams. These computer program instructions can also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more blocks of the flowcharts and / or one or more blocks of the structural diagrams. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks in the structural diagram.

[0258] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0259] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0260] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Images