Data management method and related device

Abstract

The present application relates to the technical field of databases, and discloses a data management method and a related device. In the data management method, upon receiving ordered data and out-of-order data, a database system first stores an aggregation result of the ordered data as well as the out-of-order data, separately, and aggregates, via a background task, the out-of-order data with the aggregation result of the ordered data within an aggregation time range to which the out-of-order data belongs. Consequently, the integrity of the data aggregation result within the aggregation time range to which the out-of-order data belongs is improved when a query request arrives, thereby reducing the costs of aggregation computation during a query, and helping to improve the query efficiency of the database system.

Description

Data management methods and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202510304113.9, filed on March 12, 2025, entitled “Data Management Method and Related Equipment”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of database technology, and in particular to a data management method and related equipment. Background Technology

[0003] A distributed database is a database system that stores data across multiple physical locations. Stream computing within a distributed database is a technique for processing continuously arriving data streams. Data may originate from various sources, such as sensors, log files, or user activity. In practical applications, due to factors such as network latency and varying processing speeds, events in the data stream may occur out-of-order, meaning that data from later events arrives at the processing system before data from earlier events. This out-of-order arrival of data is called out-of-order data.

[0004] To effectively handle such out-of-order data, the following out-of-order data processing methods are usually adopted: time window-based processing methods, buffer / queue-based processing methods, replay-based processing methods, and adaptive adjustment algorithm-based processing methods.

[0005] However, current methods for processing out-of-order data all suffer from high query and recalculation costs. Summary of the Invention

[0006] This application provides a data management method and related equipment. This application reduces the recalculation cost during the query process. The technical solution provided by this application is as follows:

[0007] Firstly, this application provides a data management method. The method includes: receiving multiple sets of first data; determining first ordered data and first disordered data from the multiple sets of first data based on the time attribute of each set of first data received, wherein the first ordered data includes multiple sets of first data belonging to ordered data, and the first disordered data includes multiple sets of first data belonging to disordered data; storing the result of aggregating the first ordered data into a first ordered data file; storing the first disordered data into a first disordered data file; before receiving a first query request, aggregating the first disordered data from the first disordered data file into the first ordered data file to obtain an aggregated first ordered data file; receiving a first query request, the first query request indicating a first time range, wherein both the first disordered data and the first ordered data belong to the first time range; and determining the query result of the first query request based on the aggregated first ordered data file.

[0008] In this method, after obtaining the first ordered data and the first disordered data, the first disordered data can be aggregated into the first ordered data file before receiving the first query request, resulting in a more complete aggregation result in the aggregated first ordered data file. Upon receiving the first query request, since the first query request indicates a query for data within a first time range, if both the first disordered data and the first ordered data belong to the first time range, the query result of the first query request can be directly determined based on the more complete aggregation result in the aggregated first ordered data file. This reduces the recalculation cost of aggregation calculations during the query process and improves query efficiency. Furthermore, because this process performs aggregation calculations on the first disordered data and the first ordered data, rather than discarding the disordered data, it ensures data integrity and thus helps improve query accuracy.

[0009] In one possible implementation, before receiving the first query request, the first disordered data is aggregated into a first ordered data file to obtain an aggregated first ordered data file. This includes: before receiving the first query request, aggregating the first disordered data in the first disordered data file to obtain an aggregation result of the first disordered data; and aggregating the aggregation result of the first disordered data into the first ordered data file to obtain an aggregated first ordered data file.

[0010] In one possible implementation, the method further includes: receiving multiple sets of second data; determining second ordered data and second disordered data from the multiple sets of second data based on the time attribute of each set of second data received, wherein the second ordered data includes multiple sets of second data belonging to ordered data, and the second disordered data includes multiple sets of second data belonging to disordered data; storing the result of aggregating the second ordered data into a second ordered data file; storing the second disordered data into a second disordered data file; before receiving a second query request, aggregating the second disordered data in the second disordered data file to obtain an aggregation result of the second disordered data; receiving a second query request, the second query request indicating a second time range, wherein both the second ordered data and the second disordered data belong to the second time range; after receiving the second query request, aggregating the aggregation result of the second disordered data into the second ordered data file to obtain an aggregated second ordered data file; and determining the query result of the second query request based on the aggregated second ordered data file.

[0011] In this situation, although the second out-of-order data has not yet been aggregated into the second ordered data file, since the second out-of-order data itself has already been aggregated, compared to the second out-of-order data not being aggregated, it also reduces the recalculation cost of performing aggregation calculations after the second query request arrives, which helps to improve query efficiency.

[0012] In one possible implementation, determining the query result of the first query request based on the aggregated first ordered data file includes: determining whether first target data exists, wherein the first target data is out-of-order data belonging to a first time range and has not been aggregated into the aggregated first ordered data file; if the first target data does not exist, performing a query in the aggregated first ordered data file to obtain the query result of the first query request; if the first target data exists, aggregating the first target data into the aggregated first ordered data file and then performing a query to obtain the query result of the first query request.

[0013] During this query process, the system can select an appropriate query strategy based on the aggregation of data within the time range specified in the query request, thereby achieving intelligent querying. In other words, it can adaptively determine the query strategy when the first query request arrives, without requiring any instructions from the tenant, thus improving the user's query experience.

[0014] In one possible implementation, the method further includes: receiving multiple third data; determining third ordered data and third disordered data from the multiple third data based on the time attribute of each received third data, wherein the third ordered data includes multiple third data belonging to ordered data, the third disordered data includes multiple third data belonging to disordered data, and the multiple third data belonging to disordered data belong to the same time window and are received in adjacent order; storing the result of aggregating the third ordered data to a third ordered data file; storing the result of aggregating the third disordered data to a third disordered data file; before receiving a third query request, aggregating the aggregation result of the third disordered data from the third disordered data file into the third ordered data file to obtain an aggregated third ordered data file; receiving a third query request, the third query request indicating a third time range, wherein both the third disordered data and the third ordered data belong to the third time range; and determining the query result of the third query request based on the aggregated third ordered data file.

[0015] In this implementation, since the third out-of-order data is aggregated in a timely manner when it contains multiple out-of-order data belonging to the same time window and received in adjacent order, only the aggregation result of the third out-of-order data needs to be cached before storing the aggregation result of the third out-of-order data into the out-of-order data file. Compared with caching the larger amount of third out-of-order data, this reduces the caching cost required for the implementation method.

[0016] In one possible implementation, the method further includes receiving a data processing instruction that specifies a processing strategy for out-of-order data. This enables the database system to process the out-of-order data according to the user-indicated processing strategy.

[0017] In one possible implementation, the timestamp of ordered data belongs to the system time window in which the ordered data is received, while the timestamp of out-of-order data does not belong to the system time window in which the out-of-order data is received.

[0018] Secondly, this application provides a data management device. The device includes: a data receiving module for receiving multiple first data sets; a data discrimination module for determining first ordered data and first disordered data from the multiple first data sets based on the time attribute of each received first data set; a first storage module for storing the aggregation result of the first ordered data set into a first ordered data file; a second storage module for storing the first disordered data set into a first disordered data file; an aggregation calculation module for aggregating the first disordered data set from the first disordered data file into the first ordered data file before receiving a first query request, to obtain an aggregated first ordered data file; a request receiving module for receiving a first query request, the first query request indicating a first time range, wherein both the first disordered data set and the first ordered data set belong to the first time range; and a request response module for determining the query result of the first query request based on the aggregated first ordered data file.

[0019] In one possible implementation, before receiving the first query request, the aggregation calculation module is used to: aggregate the first disordered data in the first disordered data file to obtain an aggregated first disordered data file; and aggregate the aggregated first disordered data file into the first ordered data file to obtain an aggregated first ordered data file.

[0020] In one possible implementation, the data receiving module is further configured to receive multiple sets of second data; the data discrimination module is further configured to determine second ordered data and second disordered data from the multiple sets of second data based on the time attribute of each set of second data received; the first storage module is further configured to store the result of aggregating the second ordered data into a second ordered data file; the second storage module is further configured to store the second disordered data into a second disordered data file; the aggregation calculation module is further configured to aggregate the second disordered data in the second disordered data file to obtain an aggregation result of the second disordered data before receiving the second query request; the request receiving module is further configured to receive a second query request, the second query request indicating a second time range, wherein both the second ordered data and the second disordered data belong to the second time range; the aggregation calculation module is further configured to aggregate the aggregation result of the second disordered data into the second ordered data file after receiving the second query request to obtain an aggregated second ordered data file; and the request response module is further configured to determine the query result of the second query request based on the aggregated second ordered data file.

[0021] In one possible implementation, the request-response module is used to: determine whether first target data exists, wherein the first target data is out-of-order data belonging to a first time range and not aggregated into the first ordered data file after aggregation; if the first target data does not exist, perform a query in the first ordered data file after aggregation to obtain the query result of the first query request; if the first target data exists, aggregate the first target data into the first ordered data file after aggregation and then perform a query to obtain the query result of the first query request.

[0022] In one possible implementation, the data receiving module is further configured to receive multiple third data items; the data discrimination module is further configured to determine third ordered data and third disordered data from the multiple third data items based on the time attribute of each received third data item, wherein the third ordered data includes multiple third data items belonging to ordered data, the third disordered data includes multiple third data items belonging to disordered data, and the multiple third data items belonging to disordered data belong to the same time window and are received in adjacent order; the first storage module is further configured to store the result of aggregating the third ordered data to a third ordered data file; the second storage module is further configured to store the result of aggregating the third disordered data to a third disordered data file; the aggregation calculation module is further configured to aggregate the aggregation result of the third disordered data from the third disordered data file into the third ordered data file before receiving the third query request, to obtain the aggregated third ordered data file; the request receiving module is further configured to receive a third query request, the third query request indicating a third time range, wherein both the third disordered data and the third ordered data belong to the third time range; the request response module is further configured to determine the query result of the third query request based on the aggregated third ordered data file.

[0023] In one possible implementation, the request receiving module is also used to: receive data processing instructions, which specify the processing strategy for out-of-order data.

[0024] In one possible implementation, the timestamp of ordered data belongs to the system time window in which the ordered data is received, while the timestamp of out-of-order data does not belong to the system time window in which the out-of-order data is received.

[0025] Thirdly, this application provides a computing device including a memory and a processor, the memory storing program instructions, and the processor executing the program instructions to perform the methods provided in the first aspect of this application and any possible implementation thereof.

[0026] Fourthly, this application provides a computing device cluster, including multiple computing devices, each computing device including multiple processors and multiple memories, the multiple memories storing program instructions, and the multiple processors executing the program instructions, causing the computing device cluster to perform the methods provided in the first aspect of this application and any possible implementation thereof.

[0027] Fifthly, this application provides a computer-readable storage medium that is a non-volatile computer-readable storage medium, which includes program instructions that, when executed on a computing device, cause the computing device to perform the methods provided in the first aspect of this application and any of its possible implementations.

[0028] Sixthly, this application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods provided in the first aspect of this application and any possible implementation thereof. Attached Figure Description

[0029] Figure 1 is a structural schematic diagram of an implementation scenario involving a data management method provided in an embodiment of this application;

[0030] Figure 2 is a structural diagram of an implementation scenario involving another data management method provided in this application embodiment;

[0031] Figure 3 is a schematic diagram of the deployment of basic resources in a data center according to an embodiment of this application;

[0032] Figure 4 is a schematic diagram of a database system provided in an embodiment of this application;

[0033] Figure 5 is a flowchart of a data management method provided in an embodiment of this application;

[0034] Figure 6 is a schematic diagram of a data management method provided in an embodiment of this application;

[0035] Figure 7 is a flowchart of an embodiment of this application, which shows how to aggregate first disordered data into a first ordered data file.

[0036] Figure 8 is a flowchart of another data management method provided in an embodiment of this application;

[0037] Figure 9 is a flowchart of a query provided in an embodiment of this application;

[0038] Figure 10 is a flowchart of another data management method provided in an embodiment of this application;

[0039] Figure 11 is a flowchart of another data management method provided in an embodiment of this application;

[0040] Figure 12 is a schematic diagram of a data management device provided in an embodiment of this application;

[0041] Figure 13 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0042] Figure 14 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0043] Figure 15 is a schematic diagram of another computing device cluster provided in an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0045] To facilitate understanding, the technologies and background involved in the embodiments of this application will be introduced below.

[0046] Cloud computing is a type of distributed computing that refers to a network that centrally manages and schedules a large number of computing and storage resources to provide on-demand services to users. These computing and storage resources are provided through clusters of computing devices located in data centers. Furthermore, cloud computing can provide users with various types of services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Infrastructure as a Service provides virtual machines or other resources as a service to tenants. Platform as a Service provides a development platform as a service to tenants. Software as a Service provides applications (Apps) as a service to customers.

[0047] A distributed database is a database system that stores data across multiple physical locations, typically connected via a network. Distributed databases are designed to improve data availability, reliability, and performance, while supporting large-scale data processing and access.

[0048] Stream computing is a technology for processing continuous, real-time data streams. Unlike traditional batch processing, stream computing can process and analyze continuously flowing data in real time, making it suitable for scenarios requiring rapid response, such as real-time monitoring, online advertising, and financial transactions.

[0049] Out-of-order processing refers to the technique in stream processing systems that processes data arriving out of order according to the correct time sequence. In stream computing, data should ideally arrive at the processing system in the chronological order in which events occur. However, in practice, due to factors such as network latency, system load, and different data transmission paths, the time order in which data arrives at the processing system may not match the time order in which they occur, resulting in out-of-order arrival.

[0050] Materialized views are database objects that store the actual data retrieved from one or more base tables. Unlike regular views, materialized views are not merely logical representations of query definitions; they also pre-compute and store the query results on disk.

[0051] A watermark is a special record that represents the timestamp limit at which all events prior to the current time have been observed. In other words, once a watermark for a timetamp T is issued, it means that all events less than or equal to T have either arrived or are considered lost.

[0052] Stream computing in distributed databases is a technique for processing continuously arriving data streams. Data may originate from various sources, such as sensors, log files, or user activity. In practical applications, due to factors such as network latency and varying processing speeds, data in the stream may arrive out of order; that is, data from later events may arrive at the processing system before data from earlier events. This out-of-order arrival of data is called out-of-order data.

[0053] To effectively handle out-of-order data, several methods are typically employed, including time-window-based methods, buffer / queue-based methods, replay-based methods, and adaptive adjustment algorithms. The choice of method depends on the specific application scenario and requirements for accuracy, real-time performance, and resource consumption. In practice, a combination of techniques is often necessary to arrive at the optimal solution.

[0054] In time-window-based processing, the system pre-defines a maximum allowable delay time window (watermark). Within this window, the system waits for late-arriving data. If data arrives within the window, it can be processed correctly; once the window expires, it's assumed no earlier data will arrive, and subsequent data is processed according to the current timestamp. This method offers some flexibility in handling out-of-order data but introduces additional latency, and data arriving after the window is discarded, leading to data loss and affecting accuracy. In buffer / queue-based processing, memory or persistent storage is used as a temporary buffer to store all received data until it can be processed sequentially. This requires sufficient space to store unsorted data and effective management strategies to avoid resource exhaustion. In replay-based processing, when the system detects significant out-of-order conditions, it can choose to restart reading data from a specific point and process the entire reread data stream. While this method guarantees consistency of the final result, it can lead to higher performance overhead and complexity. In processing methods based on adaptive adjustment algorithms, the processing system dynamically adjusts the aforementioned parameters (such as the time window size) by monitoring the actual latency of the input stream, thereby optimizing the balance between the system's tolerance for out-of-order processing and performance. In general, current out-of-order data processing methods all suffer from high query and recalculation costs.

[0055] In view of this, embodiments of this application provide a data management method. In this method, after receiving ordered and disordered data, the database system first stores the aggregation results of the ordered data and the disordered data respectively. Then, a background task aggregates the disordered data with the aggregation results of the ordered data within the same aggregation time range. This improves the integrity of the aggregation results within the aggregation time range of the disordered data when a query request arrives, thereby reducing the recalculation cost of aggregation calculations during the query process and helping to improve the query efficiency of the database system. Furthermore, since this process performs aggregation calculations on the disordered and ordered data before the query, rather than discarding the disordered data, it ensures data integrity and helps improve the query accuracy of the database system.

[0056] This article provides a detailed introduction to the technical solution of this application from multiple perspectives, including implementation scenarios, methods and processes, hardware devices, and software devices.

[0057] The following are examples illustrating the implementation scenarios of the embodiments of this application.

[0058] Figure 1 is a structural diagram of an implementation scenario involving a data management method provided in this application. As shown in Figure 1, the implementation scenario includes one or more server terminals 10 and client terminals 20. The one or more server terminals 10 are used to provide database services, and the server terminal 10 is also referred to as a database server terminal 10. The client terminal 20 is capable of establishing a communication connection with one or more server terminals 10. For example, the client terminal 20 and one or more server terminals 10 can establish a communication connection through a network. Optionally, the network can be a local area network, the Internet, or other networks, and this application embodiment does not limit the scope of the network.

[0059] Client 20 is used to allow users to interact with one or more servers 10. In one implementation, client 20 is used to send a query request to server 10 according to the user's instructions. One or more servers 10 are used to query the data stored on server 10 based on the query request and return the query results to client 20.

[0060] In one possible implementation, the client 20 can be a desktop computer, laptop computer, mobile phone, smartphone, tablet computer, cloud server, portable mobile terminal, multimedia player, smart home appliance, artificial intelligence device, smart wearable device, e-reader, smart in-vehicle device, or Internet of Things device, etc.

[0061] Server 10 can be a server (such as a cloud server). When the implementation scenario includes multiple servers 10, these multiple servers 10 can be referred to as a computing device cluster. In this case, the computing device cluster is a server cluster composed of several servers, or it can be implemented as a cloud computing service center. The cloud computing service center deploys a large amount of basic resources owned by the cloud service provider. For example, the cloud computing service center deploys computing resources, storage resources, and network resources. The cloud computing service center can utilize these large amounts of basic resources to store the data involved in this application and provide data query services to the client 20.

[0062] When a computing device cluster is implemented through a cloud computing service center, the database functionality implemented by the computing device cluster can be abstracted into a cloud service, such as a database cloud service, by the cloud service provider on the cloud platform. In this case, users can access the cloud management platform through client 20, purchase the cloud service on the cloud management platform, and then use the cloud service provided by the computing device cluster through the cloud management platform. Optionally, the cloud management platform can be a central cloud cloud management platform, an edge cloud cloud management platform, or a cloud management platform including both a central cloud and an edge cloud; this application embodiment does not specifically limit its scope. Furthermore, the cloud service can be provided by the cloud management platform as a standalone cloud service, or it can be provided as an add-on cloud service to other cloud services; this application embodiment does not specifically limit its scope either.

[0063] When the computing device cluster is implemented through a cloud computing service center, as shown in Figure 2, the implementation scenario includes: cloud system 1 and client 20. Cloud system 1 and client 20 can establish a communication connection via a network. Optionally, this network can be the Internet, or other networks; this embodiment does not limit the specific network used. Tenants can interact with cloud system 1 through client 20. For example, a tenant can send query requests and other information to cloud system 1 through client 20. Cloud system 1 responds based on the information sent by client 20.

[0064] As shown in Figure 2, cloud system 1 includes a cloud management platform and infrastructure. Server 10 can be implemented through this infrastructure. The cloud management platform and infrastructure are connected via the cloud system's internal network. In another implementation, the cloud management platform may optionally be located within the infrastructure. The cloud management platform is used to manage the infrastructure. The infrastructure is used to provide public cloud services. The infrastructure includes at least one data center (DC). The cloud management platform can connect to this at least one data center. In this case, the cloud management platform is used to manage this at least one data center. The data center deploys a large number of cloud resources owned by the cloud service provider, such as computing resources, storage resources, and network resources. Computing resources can be computing devices (such as servers) capable of providing computing power. For example, as shown in Figure 2, multiple servers are deployed in the data center. Cloud services may optionally be deployed on the servers. Cloud services are implemented by running virtual instances, therefore also referred to as virtual instances deployed on servers to implement tenant services. Tenants can send cloud service requests and related information to the server through their client 20. The server can process the cloud service requests and related information and provide cloud services to the tenant based on the processed cloud service requests and related information.

[0065] The cloud management platform can be logically divided into: tenant console, compute management service, network management service, storage management service, authentication service, and image management service. The tenant console provides a user interface or application programming interface (API) for interaction with tenants. The compute management service manages servers running virtual instances and bare metal servers. The network management service manages network services (such as gateways and firewalls). The storage management service manages storage services (such as data bucket services). The authentication service manages tenant accounts and passwords. The image management service manages virtual instance images.

[0066] In the implementation scenario shown in Figure 2, a data center contains multiple servers. The servers consist of a hardware layer and a software layer. The hardware layer comprises the standard server configuration, including processors, memory, network interface cards (NICs), disks, and buses. The software layer includes the operating system installed and running on the server. The operating system relative to the virtual machine can be called the host operating system. The host operating system runs a virtual machine manager (also known as a hypervisor). The virtual machine manager's role is to implement compute virtualization, network virtualization, and storage virtualization, and to manage the virtual machines.

[0067] The virtual machine manager runs a cloud management platform client. This client receives control plane commands from the cloud management platform, creates virtual instances on the server based on these commands, and manages the virtual instances throughout their lifecycle. For example, the client can monitor the hardware resource usage of the server in real time and report it to the cloud management platform. When the cloud management platform confirms that a virtual instance needs to be created on a specific server, it sends a virtual instance creation command to the client on that server. Upon receiving the command, the client creates the virtual instance on that server. In this way, tenants can create, manage, log in to, and operate virtual instances through the cloud management platform.

[0068] Servers can run virtual machines of different specifications. Virtual machine specifications are categorized as: general-purpose computing, memory-optimized, ultra-large memory, etc., with specific specifications under each type. After a tenant selects a virtual machine specification, the cloud management platform selects a server in the data center that supports that specification and ensures sufficient idle hardware resources on that server. Then, it creates and configures the virtual machine with that specification on that server. Configuring servers through the cloud management platform allows for the analysis and planning of server hardware resources. Based on the server's hardware performance, it plans the corresponding computing products for the physical hardware, such as planning virtual machines of different specifications, to meet the diverse needs of different tenants. Furthermore, differentiated pricing strategies can be implemented based on the performance differences of virtual machines of different specifications. For example, high-performance virtual instances can be sold at a higher price, while ordinary performance virtual instances can be sold at a lower price, allowing tenants to purchase virtual instances as needed.

[0069] In one implementation, as shown in Figure 3, the location of infrastructure can be described by cloud resource deployment regions (regions) and availability zones (AZs). Tenants can choose to deploy cloud services based on resources within specific regions and AZs. Regions are defined based on geographical location and network latency. Using the same resource pool within the same region can be understood as sharing public services such as elastic computing, block storage, object storage, virtual private cloud (VPC) networks, elastic internet protocol (EIP) addresses, and images. Regions are divided into general-purpose regions and dedicated regions. General-purpose regions provide general cloud services to public tenants. Dedicated regions are dedicated regions that host the same type of business or provide business services to specific tenants. A region typically includes multiple AZs. Multiple AZs within a region are connected via high-speed fiber optic cables to meet the needs of tenants building high-availability systems across AZs. Computing, network, and storage resources within an AZ are logically divided into multiple clusters.

[0070] Tenants can send instructions to the cloud management platform through their client 20 to create, manage, log in to, and operate virtual instances on the server, and use the cloud services provided by these virtual instances. For example, the cloud management platform can provide an access interface. This interface can be provided either as a user interface or an API. Tenants can operate their client to remotely access the access interface to register a cloud account and password on the cloud management platform, and then log in using these accounts and passwords. The cloud management platform can also authenticate the cloud account and password. After successful authentication, the tenant can further select and purchase virtual instances of specific specifications (processor, memory, disk) on the cloud management platform. After the tenant successfully purchases a virtual instance, the cloud management platform provides the tenant with a remote login account and password for the purchased virtual instance. The tenant can use the remote login account and password to remotely log in to the virtual instance on their client, install and run their application within the virtual instance, and use the application to implement their business operations.

[0071] In one implementation, the data management method provided in this application embodiment can be implemented by running an executable program on a computing device in cloud system 1. When the data management method provided in this application embodiment is applied to a cloud management platform, the server used to implement the cloud management platform can implement the data management method provided in this application embodiment by running the executable program of the data management method provided in this application embodiment. Furthermore, the executable program implementing the data management method can optionally be presented in the form of an application installation package. After the server installs the application installation package, it can implement the data management method provided in this application embodiment by running the executable program therein.

[0072] As an implementable form of this application, the data management method provided can be applied to providing database services to tenants. The database service can perform management operations on data written to the database by tenants and respond to query requests sent by tenants. This method can be applied to the management of various types of data. For example, it can be applied to the process of performing stream computing on data, and also to other processing processes, such as real-time big data stream anomaly detection, and stream computing tasks in analytical databases and real-time data warehouses. The data presented by this method can also take various forms. For example, the data can be presented as a data table or a materialized view, such as online transaction processing (OLTP) or online analytical processing (OLAP).

[0073] The data management method provided in this application is implemented by a database system, but this application does not restrict the deployment form of the database system. Several possible deployment forms of the database system are described below.

[0074] In the first deployment model of the database system, the database system adopts a compute-in-store architecture. At this time, the database system includes at least one data node, and each data node serves as both a compute node and a storage node. All operations of the data management method can be performed by one or more data nodes.

[0075] In the second deployment model of the database system, a compute-storage separation architecture is adopted. In this model, the database system includes compute nodes and storage nodes. Some data management operations are performed by the compute nodes, and some by the storage nodes. Depending on different application requirements, there are various ways to implement the division of labor between compute and storage nodes.

[0076] For example, in scenarios where some operators are pushed down to storage nodes, the compute nodes are responsible for sending and receiving data and performing some processing operations on the data, while the storage nodes are responsible for performing other processing on the data processed by the compute nodes. For instance, in the data management method provided in this application, the following operations are performed by the compute nodes: receiving multiple first data from the client producing the data; determining first ordered data and first out-of-order data from the multiple first data based on the time attribute of each received first data; receiving a first query request; and determining the query result of the first query request based on the aggregated first ordered data file. The following operations are performed by the storage nodes: storing the result of aggregating the first ordered data into the first ordered data file; storing the first out-of-order data into the first out-of-order data file; and aggregating the first out-of-order data into the first ordered data file according to a trigger condition to obtain the aggregated first ordered data file.

[0077] For example, in scenarios where all operators are pushed down to the storage node, the compute node is primarily responsible for sending and receiving data, while the storage node performs all related processing on the received data. For instance, in the data management method provided in this application, the following operations are performed by the compute node: receiving multiple first data sets from the client producing the data; sending the multiple first data sets to the storage node; receiving a first query request; and determining the query result of the first query request based on the aggregated first ordered data file. The following operations are performed by the storage node: receiving multiple first data sets from the compute node, and determining first ordered data and first out-of-order data from the multiple first data sets based on the time attribute of each first data set received; storing the result of aggregating the first ordered data into the first ordered data file; storing the first out-of-order data into the first out-of-order data file; and aggregating the first out-of-order data into the first ordered data file according to a trigger condition to obtain the aggregated first ordered data file.

[0078] It should be noted that regardless of the deployment form of the database system, the database system also includes storage media, which is used to store data that has been processed by storage nodes (such as stream processing). There are various implementation forms available, such as distributed storage systems with data function virtualization (DFV), object storage services (OBS), or elastic volume services (EVS).

[0079] In one possible implementation, the database system involved in this application is a Structured Query Language (SQL) database. As shown in Figure 4, this database system includes a ts-sql component, a ts-store component, a ts-meta component, and a storage medium. The ts-sql component is responsible for interacting with the client, routing written data and query conditions, and aggregating query data. The ts-sql component includes a parser, a cache, and a coordinator. The parser decodes write and query requests from the user and encodes data sent to the client. The cache temporarily stores data that needs to be transmitted and processed by the ts-sql component. The coordinator routes data that does not require the computational processing described in this application to the ts-store for storage. The ts-store component is responsible for enabling data storage and performing partial aggregation calculations on query data. The ts-store component includes a query executor, a cache, and a storage engine. The query executor executes query requests. The cache is used to temporarily store data that needs to be transferred and processed by the ts-store component. The storage engine is used to store data in the storage medium. The ts-meta component is responsible for managing metadata. Metadata is data needed to manage the database system. For example, metadata includes sharding information for multiple shards of the storage medium, and the association between multiple ts-store components and shards. Sharding information indicates shard usage, etc. When a client needs to write data to the database system, the ts-sql component needs to obtain the current sharding information from the ts-meta component, and then distribute the write data to different ts-store components based on the sharding information, allowing the ts-store components to process the write data. The ts-meta component includes a node manager, a metadata store, a balancer, and a job manager. The node manager manages the nodes (such as compute nodes and storage nodes) in the database system. The metadata store stores metadata. The balancer performs load balancing scheduling among multiple nodes in the database system. The job manager manages the business operations executed by the database system.

[0080] As shown in Figure 4, to implement this application, the database system also includes a Pre-stream component and a Stream engine component. The Pre-stream component is responsible for data tagging and preliminary computation processing. Data tagging refers to marking the written data to indicate whether the computation processing (such as stream processing) of this application needs to be performed. For example, users may decide whether to perform the computation processing of this application on their data based on their needs, and set whether to perform the computation processing of this application on their data in the configuration information of the data source to which the written data belongs. The Pre-stream component then determines whether the computation processing of this application needs to be performed on the written data based on the configuration information of the data source to which the written data belongs, and then adds a tag to the written data, which is used to indicate whether the written data needs to perform the computation processing of this application. Preliminary computation processing refers to routing the written data that needs to perform the computation processing of this application to the component used to perform the computation processing (i.e., the Stream engine component). The Stream engine component is responsible for performing the computation processing of this application on the written data and flushing the computationally processed written data to the storage medium. For a storage-compute separation database, the Pre-stream component can be deployed in the compute node or independently of the compute node. The Stream engine component can be deployed on storage nodes or independently of storage nodes. For in-store compute databases, both the Pre-stream component and the Stream engine component are deployed on data nodes.

[0081] In addition, when the database system is implemented through a computing device cluster, the implementation form of the data nodes, computing nodes and storage nodes can all be selected as a computing device cluster, and this application embodiment does not specifically limit it.

[0082] It should be understood that the above content is an exemplary description of the implementation scenarios of the data management method provided in the embodiments of this application, and does not constitute a limitation on the implementation scenarios of the data management method. Those skilled in the art will know that as business needs change, the implementation scenarios can be adjusted according to application requirements, and the embodiments of this application do not specifically limit them. Furthermore, when the data management method provided in the embodiments of this application is applied to other scenarios, the executable program of the method can also be presented in the form of an application installation package or in other ways, and the embodiments of this application do not list them all.

[0083] The implementation process of a data management method provided in this application embodiment is illustrated below. Figure 5 is a flowchart of a data management method provided in this application embodiment. As shown in Figure 5, the implementation process of this method includes the following steps:

[0084] Step 501: Receive multiple first data.

[0085] When a tenant needs to write data to the database system, the tenant can send a write request to the database system through their client to instruct the database system to write data. A write request typically carries one or more pieces of data to be written to the database. The database system parses the write request, obtains the data carried in the write request, and stores the data in the storage medium. The client, as a data source, continuously produces data, and the database system receives multiple initial data entries from the clients that produce data.

[0086] Step 502: Based on the time attribute of each first data received, determine the first ordered data and the first disordered data from multiple first data.

[0087] In one possible implementation, the time attribute of receiving the first data is represented by the timestamp of the first data. The storage node can then determine whether the first data is ordered or out-of-order based on the timestamp. For example, a database system has a system time window. The system time window in which the database system is located during a certain time period indicates data that should arrive in the database system during that time period. For a given set of data, the database system can determine whether the data is ordered or out-of-order based on the system time window and the data's timestamp. That is, when the timestamp of a data belongs to the system time window in which it was received, the data is ordered; when the timestamp of a data does not belong to the system time window in which it was received, the data is out-of-order.

[0088] In this embodiment, when the timestamp of the first data belongs to the system time window in which the first data is received, the first data is ordered data; when the timestamp of the first data does not belong to the system time window in which the first data is received, the first data is out-of-order data. Therefore, the storage node can determine which first data is ordered data and which is out-of-order data based on the time attribute of each first data received. That is, the storage node determines the first ordered data and the first out-of-order data from multiple first data based on the time attribute of each first data received, wherein the first ordered data includes multiple ordered data, and the first out-of-order data includes multiple out-of-order data.

[0089] As the write process progresses, the generation time of the ordered data received later by the database system should lag behind the generation time of the ordered data received earlier by the compute nodes. Correspondingly, the time range of the system time window used to distinguish between ordered and out-of-order data will change accordingly; that is, the start and end times of the system time window will be shifted backward. Shifting the start and end times of the system time window is also the process of closing the old system time window and opening a new one. For example, as the write process proceeds, the storage node can close the old system time window [t0, t0+L) and open a new system time window [t0+L, t0+2×L), where the time range covered by the new system time window is L time-long later than the time range covered by the old system time window.

[0090] For ease of description, the values ​​of the data in the following text also represent the timestamps of the data. In the example shown in Figure 6, the length of the system time window is 10, and the multiple first data include data 10, 11, 12, 13, 14, 15, 16, 17, 18, and 19. The database system receives data 12, 13, 15, 16, 17, 18, and 19 within the system time window [10, 20), and data 10, 11, and 14 within the system time window [20, 30). Based on the timestamp of each first data, the database system determines that data 12, 13, 15, 16, 17, 18, and 19 are all ordered data, while data 10, 11, and 14 are all out of order. That is, the first ordered data includes data 12, 13, 15, 16, 17, 18, and 19, and the first out of order data includes data 10, 11, and 14.

[0091] Optionally, the time window-related data in this application can be managed using an array. Furthermore, to reuse the array, a circular pointer array can be used to manage the time window-related data. When the database system needs to close and open a time window, the data related to the opened time window can be managed using the array of the closed time window. However, the time range of the data managed by this array changes from the time range of the closed time window to the time range of the opened time window. The database system can perform the closing and opening of a time window simultaneously or not simultaneously. For example, when the maximum allowed latency of the database system is 0, the database system needs to close the time window and open a new time window when the opening duration equals the length of the time window. In this case, the closing and opening operations are performed simultaneously. As another example, when the maximum allowed latency of the database system is not 0 (e.g., m), the database system closes the time window only when the opening duration equals the length of the time window plus the maximum latency. However, when the opening duration of the time window equals the length of the time window, a new time window needs to be started. In this case, the closing and opening operations are not performed simultaneously.

[0092] Additionally, after receiving multiple sets of initial data, the database system can temporarily store them. In one possible implementation, these multiple sets of initial data can be stored in a source table. The creation of the source table can be achieved through the interactive design of the database system. For example, the database system may optionally have an interactive interface, through which tenants can interact with the database system. For instance, tenants can send Data Definition Language (DDL) statements to the database system through the interactive interface to achieve interaction. In one possible implementation, the storage medium includes multiple data shards, each containing multiple data tables. When creating a source table, the tenant can specify the data shard containing the source table and the name of the source table using DDL statements. For example, the DDL statement for creating a source table is:

[0093] CREATE MEASUREMENT[tableName]WITH SHARDKEY <groupkey> [, <groupkey>]

[0094] This DDL statement instructs the creation of a source table named tableName based on the data shard with the key groupkey.

[0095] Step 503: Store the result of aggregating the first ordered data into the first ordered data file.

[0096] After the database system obtains the first ordered data, it can first aggregate the first ordered data, and then store the result of the aggregation in the first ordered data file. Aggregation of the first ordered data involves performing aggregation operations on multiple ordered data items included in the first ordered data. For example, aggregation operations can include summation, average, counting, median, standard deviation, variance, finding the maximum value, finding the minimum value, or product, etc.

[0097] In one possible implementation, the first ordered data can be aggregated according to a specified time granularity. The specified time granularity indicates the length of the aggregation time range. The specified time granularity can be determined based on application requirements. For example, the specified time granularity is determined based on the minimum query time range that the database system can provide. An aggregation time range can be regarded as a data group, and the database system needs to perform aggregation calculations on data belonging to the same aggregation time range. In the embodiments of this application, the aggregation of the first ordered data by the database system includes performing aggregation operations on data belonging to the same aggregation time range within the first ordered data. When multiple ordered data of the first ordered data belong to one aggregation time range, the result of aggregating the first ordered data includes only one aggregation result for one aggregation time range; when multiple ordered data of the first ordered data belong to multiple aggregation time ranges, the result of aggregating the first ordered data includes multiple aggregation results for multiple aggregation time ranges.

[0098] For example, as shown in Figure 6, the aggregation time range is 5, and the database system has four aggregation time windows: window1, window2, window3, and window4. The aggregation time range of window1 is [0, 5), the aggregation time range of window2 is [5, 10), the aggregation time range of window3 is [10, 15), and the aggregation time range of window4 is [15, 20). In the example shown in Figure 6, the data 12 and 13 in the first ordered data belong to the aggregation time range of window3, and the data 15, 16, 17, 18, and 19 in the first ordered data belong to the aggregation time range of window4. Therefore, the aggregation result of the first ordered data includes the aggregation results of window3 and window4. For example, when the aggregation calculation is a summation, the aggregation result of window3 is the sum of data 12 and 13, which is 25, and the aggregation result of window4 is the sum of data 15, 16, 17, 18, and 19, which is 85. In one possible embodiment, the database system writes the two records "[10,15), sum=25" and "[15,20), sum=85" into the first ordered data file, thereby storing the result of aggregating the first ordered data into the first ordered data file. As shown in Figure 6, the file storing ordered data is called an ordered data file. After a period of time, the database system will store multiple ordered data files, and the first ordered data file is one of the multiple ordered data files.

[0099] To facilitate retrieval of different ordered data files, the database system can also add a time attribute to the first ordered data file to identify which time range of data it stores. In this embodiment, the time range of the first ordered data file is determined based on multiple aggregated time ranges corresponding to the first ordered data file. For example, the time attribute of the first ordered data file can be a large time range summarized from the multiple aggregated time ranges it includes.

[0100] Optionally, aggregation on the first ordered data can be performed using atomic operations. For example, when multiple concurrent threads perform aggregation calculations on multiple ordered data points within the first ordered data set, using atomic operations ensures that the aggregation calculation is completed by one thread before the next thread's calculation begins. Alternatively, this effect can be achieved using locking mechanisms. Combining atomic operations or locking mechanisms ensures the accuracy and concurrency safety of aggregation calculations on the first ordered data set. When this effect is achieved using atomic operations, its simple implementation principle allows for near-lock-free aggregation calculations. Furthermore, similar to system time windows, the time range of the aggregation time window also changes with the system time, and the aggregation time window can also be managed using an array. For its implementation, please refer to the relevant description in the system time window section; it will not be elaborated upon here.

[0101] The process of determining the first ordered data from multiple first data sets can optionally be performed in the database system's cache. After obtaining the first ordered data, the database system can write it into its memory, and then flush the first ordered data from memory to disk onto the database system's storage medium. The first ordered data is stored on the storage medium as a data file. The operation of flushing the first ordered data from memory to disk can optionally be performed when the accumulated data reaches a specified amount or when the data accumulation time reaches a specified time interval.

[0102] As described above, after receiving multiple sets of first data, the storage node can store these first data in a source table. Similarly, the first ordered data after aggregation can be stored in a destination table. For example, since the aggregation calculation performed on the first ordered data is a stream processing operation, this destination table is also called a stream table. The creation of this destination table can also be achieved through the interactive design of the database system. For example, a tenant sends a DDL statement instructing the creation of the destination table to the database system through an interactive interface. The tenant's instruction to create the destination table can optionally be executed before the storage node aggregates the first ordered data. For example, when the tenant instructs the creation of the source table via a DDL statement, they also instruct the creation of the destination table via a DDL statement. Optionally, the destination table may also record the source table on which the creation of the destination table is based. All this information that the target table needs to record can be indicated by the tenant to the database system via a DDL statement. For example, the DDL statement for creating a stream table is:

[0103] CREATE STREAM[StreamName]INTO[db.rp.measurement]AS

[0104] SELECT_clause FROM_clause[WHERE_clause]GROUP BY[<time(<time_interval>)>[,<tag_key>]]

[0105] The DDL statement is used to instruct to create a stream computing task named StreamName. The aggregate computing to be performed by the stream computing task is determined by the application requirements of the database. The computing result of the stream computing task is written into a stream table named db.rp.measurement. In the DDL statement, the "FROM_clause" field indicates the source table where the data to be operated by the stream computing task is located, the "[WHERE_clause]" field indicates the data filtering condition, the "<time(<time_interval>)>" field indicates the aggregation time range, and the field "[,<tag_key]" indicates the aggregation key. The stream computing task needs to perform grouped aggregation on data according to the aggregation time range and the aggregation key.

[0106] Step 504, storing first out-of-order data into a first out-of-order data file.

[0107] In this embodiment, the file storing out-of-order data is called an out-of-order data file. After obtaining the first out-of-order data, the database system can store the first out-of-order data in the first out-of-order data file. The first out-of-order data file can be generated after obtaining the first out-of-order data, or the aggregation result of the first ordered data can be written into a pre-generated first ordered data file after obtaining the first ordered data. In one possible embodiment, any data file also carries an identifier used to distinguish whether the data file is an ordered data file or an out-of-order data file. Optionally, in order to distinguish different first out-of-order data files, the database system can also add a time attribute to the first out-of-order data file. For example, the database system adds a time range to the first out-of-order data file. The minimum value of the time range of the first out-of-order data file can be selected as the timestamp of the earliest generated first out-of-order data stored in the first out-of-order data file; the maximum value of the time range of the first out-of-order data file can be selected as the timestamp of the latest generated first out-of-order data stored in the first out-of-order data file. Furthermore, to facilitate the distinction between the first out-of-order data file used to store the first out-of-order data and the aggregated first out-of-order data file used to store the aggregation results of the first out-of-order data, the storage node can also add file attributes to the corresponding out-of-order data files. These file attributes indicate whether the data type stored in the out-of-order data file is the first out-of-order data or the aggregation results of the first out-of-order data. Additionally, the first ordered data and the first out-of-order data belonging to the same data source can optionally be stored in data files belonging to the same data table. For example, the name prefixes of the ordered data files and the out-of-order data files storing data from the same data source can use the same data table name. In this way, since data belonging to the same data source can be recorded through the same data table, data fragmentation due to shared origins is avoided, which helps reduce recalculation costs.

[0108] The process of determining the first out-of-order data from multiple sets of first data can be performed in the database system's cache. After obtaining the first out-of-order data, the database system can write it into its memory, and then flush it from memory to disk to the database system's storage medium. The first out-of-order data is stored in the storage medium as a data file. For example, as shown in Figure 6, the database system has an out-of-order data queue in its memory. After obtaining the first out-of-order data based on a system time window, the database system can first write the first out-of-order data into the out-of-order data queue in memory, and then flush it from memory to disk to the database system's storage medium. The operation of flushing the first out-of-order data to the database system's storage medium can be performed when the accumulated data reaches a specified amount or when the data accumulation time reaches a specified time interval.

[0109] As described above, after receiving multiple sets of first data, the database system can store these first data sets in a source table. Similarly, both the first out-of-order data and the first out-of-order data after aggregation can be stored in a destination table. For example, since aggregation is a stream processing operation, this destination table is also called a stream table. The creation of this destination table can also be achieved through the interactive design of the database system. For example, a tenant can send a DDL statement instructing the database system to create a destination table through an interactive interface. The tenant's instruction to create the destination table can optionally be executed before the database system aggregates the first ordered data. For example, when the tenant instructs the creation of a source table via a DDL statement, they can also instruct the creation of a destination table via a DDL statement. Optionally, the destination table may also record the source table on which the creation of the destination table is based. When the destination table is used to record the first out-of-order data, it may also need to record strategies for processing the first out-of-order data. All this information that the destination table needs to record can be instructed by the tenant to the database system via DDL statements.

[0110] In one possible embodiment, based on tenant needs, the database system of this application can support two out-of-order data processing strategies: a discard strategy and a recalculation strategy. The discard strategy instructs the discarding of out-of-order data in the stream computing task, while the recalculation strategy instructs the recalculation of out-of-order data in the stream computing task. For example, the recalculation strategy instructs the recalculation of out-of-order data in the stream computing task through a background task without the query tenant's awareness. When the database system's out-of-order data processing strategy is configured as a recalculation strategy, step 504 and other aggregation operations related to out-of-order data will be executed. When the database system's out-of-order data processing strategy is configured as a discard strategy, the database system will discard the identified out-of-order data. For example, the DDL statement for creating the target table is:

[0111] CREATE STREAM[StreamName]INTO[db.rp.measurement]AS

[0112] SELECT_clause FROM_clause[WHERE_clause]GROUP BY[ <time(<time_interval> )>[, <tag_key]]

[0113] DELAY<delay_time> ]

[0114] [DISORDERED_POLICY<>]

[0115] The DDL statement is used to instruct creating a stream computing task named StreamName. The aggregate computation to be performed by the stream computing task is determined by the application requirements of the database. The computation result of the stream computing task is written into a stream table named db.rp.measurement. In the DDL statement, the "FROM_clause" field indicates the source table where the data to be processed by the stream computing task is located, the "[WHERE_clause]" field indicates the data filtering condition, the "<time(<time_interval>)>" field indicates the aggregation time range, the field "[,<tag_key]" indicates the aggregation key, the "[DELAY<delay_time>]" field indicates the delay time for waiting out-of-order data, and the "[DISORDERED_POLICY<>]" field is used to specify the processing policy for out-of-order data as a discard policy or a recalculation policy. The stream computing task needs to group and aggregate the data according to the aggregation time range and the aggregation key. In this example DDL statement, the "<time(<time_interval>)>" field and the "[DELAY<delay_time>]" field also jointly indicate the length of the system time window. For example, the sum of the aggregation time range and the delay time for waiting out-of-order data is selected as the length of the system time window. In other possible embodiments, a separate additional field may also be used to indicate the length of the system time window.

[0116] It should be noted that, in the present application, recording ordered data in an ordered data file and out-of-order data in an out-of-order data file is an example of separately recording ordered data and out-of-order data in the present application, which does not limit that ordered data and out-of-order data can also be recorded in other manners. For example, ordered data and out-of-order data can alternatively be recorded in the same data file, which has an ordered data area and an out-of-order data area, the ordered data area is used for recording ordered data, and the out-of-order data area is used for recording out-of-order data, so that separate recording of ordered data and out-of-order data can be achieved. For another example, ordered data and out-of-order data are recorded in the same data file, and the data file distinguishes ordered data from out-of-order data through data identifiers.

[0117] Step 505: Before receiving the first query request, aggregating the first out-of-order data from the first out-of-order data file into the first ordered data file to obtain an aggregated first ordered data file.

[0118] In this embodiment, the database system performs real-time stream processing on data received from the production data client through a foreground stream computing task, specifically steps 501-504. To reduce recalculation costs and improve query efficiency after receiving a query request, the database system initiates a separate aggregation task in the background to process out-of-order data and aggregate it into the aggregation result within the aggregation time range of the out-of-order data. Taking a system time window as an example, within the time range of a system time window, the database system continuously receives data from the production data client. For the data received within this system time window, after determining the ordered and out-of-order data based on the timestamp of each data and the current system time window, the database system first stores the aggregation result of the ordered data and the out-of-order data respectively. Then, it aggregates the out-of-order data with the aggregation result of the ordered data within the aggregation time range to which the out-of-order data belongs through a background task, thereby improving the completeness of the data aggregation result within the aggregation time range to which the out-of-order data belongs when the query arrives.

[0119] For example, in the example shown in Figure 6, data 12 and 13 within the aggregation time range [10, 15) arrive within the system time window [10, 20) and are determined to be ordered data. After the system time window [10, 20) closes, the database system performs aggregation calculations on data 12 and 13 to obtain the aggregation result within the aggregation time range [10, 15) and stores the aggregation result in the first ordered data file. Data 10, 11, and 14 within the aggregation time range [10, 15) arrive within the system time window [20, 30) and are determined to be out-of-order data. After the system time window [20, 30) closes, the database system stores the out-of-order data 10, 11, and 14 in the first out-of-order data file. Similar to the processing of ordered data within the aggregation time range [10, 15), ordered data 15, 16, 17, 18, and 19 within the aggregation time range [15, 20) are also stored in the first ordered data file, thereby realizing the storage of the first ordered data in the first ordered data file. After the system time window [20, 30) closes, the database system's front end can continue to open the system time window [30, 40) to receive new data from the production client and perform similar processing on the received new data as the previous two system time windows. Simultaneously, the database system can use a background aggregation task to aggregate the first out-of-order data (i.e., the out-of-order data 10, 11, and 14 within the aggregation time range [10, 15)) into the first ordered data file. The aggregation result within the aggregation time range [10, 15) is then stored in the first ordered data file, resulting in the aggregated first ordered data file. In other words, by executing the above process, the database system achieves the aggregation of the first out-of-order data into the first ordered data file via a background aggregation task, obtaining the aggregated first ordered data file. This will be further explained using a summation operation as an example of aggregation calculation. After executing steps 501-504, the aggregation result within the aggregation time range [10,15) stored in the first ordered data file is only the sum of data 12 and 13; however, after executing step 505, data 10, 11, and 14 within the aggregation time range [10,15) are also aggregated, and the aggregation result within the aggregation time range [10,15) stored in the first ordered data file after aggregation is the sum of data 10, 11, 12, 13, and 14, making the data aggregation result more complete.

[0120] In one possible embodiment, the background aggregation task of the database system processes out-of-order data based on a triggering mechanism. When the triggering condition is met, the database system performs a recalculation operation. The recalculation operation includes aggregating out-of-order data from the out-of-order data file into the corresponding ordered data file. The corresponding ordered data file stores the aggregation results of ordered data within the aggregation time range to which the out-of-order data belongs. Aggregating out-of-order data into the corresponding ordered data file means aggregating out-of-order data into the aggregation results of ordered data within the aggregation time range to which the out-of-order data belongs. For example, the first ordered data includes all ordered data within the aggregation time range to which the first out-of-order data belongs. Therefore, under the triggering mechanism of the background task, the database system will aggregate the first out-of-order data into the first ordered data file according to the triggering condition. This specifically means aggregating the first out-of-order data into the aggregation results of ordered data within the aggregation time range to which the first out-of-order data belongs, thereby obtaining a more complete aggregation result. The triggering condition of the background aggregation task can be any common triggering mechanism in background tasks, and this application does not limit it. For example, the operation of aggregating out-of-order data can be a scheduled task, and correspondingly, the triggering condition can be that the time interval since the last aggregation of out-of-order data reaches a preset time interval.

[0121] It should be noted that this application is not limited to the number of background aggregation tasks executed to aggregate the first out-of-order data into the first ordered data file. For example, when the first out-of-order data contains out-of-order data within a small time range, it is possible that only one background task is executed to aggregate the first out-of-order data into the first ordered data file. As another example, when the first out-of-order data contains out-of-order data within multiple aggregation time ranges, it is possible that multiple background tasks are required to aggregate all the out-of-order data from the first out-of-order data into the first ordered data file, with each background task only aggregating a portion of the first out-of-order data into the first ordered data file.

[0122] In this application, there are two ways to implement the background aggregation task.

[0123] In the first scenario, the background aggregation task consists of only one background task. This task indicates that when the trigger condition is met, it aggregates the out-of-order data from the out-of-order data file into the corresponding ordered data file. In this case, the aggregation of out-of-order data into the corresponding ordered data file is achieved by performing an aggregation calculation on the out-of-order data and the aggregation results of the ordered data within the aggregation time range to which the out-of-order data belongs, stored in the corresponding ordered data file.

[0124] In the second scenario, the background aggregation task includes two background tasks: a first background task and a second background task. The first background task, when a first trigger condition is met, aggregates out-of-order data within the same aggregation time range from the out-of-order data file. The second background task, when a second trigger condition is met, aggregates the data from the out-of-order data file into the corresponding ordered data file. In this case, the execution interval of the first background task is usually shorter than that of the second background task, and at least one first background task will be executed between two consecutive second background tasks. In this situation, out-of-order data within a certain aggregation time range may be processed by both the first and second background tasks simultaneously. In other words, after the out-of-order data within a certain aggregation time range is stored in the out-of-order data file, the database system first aggregates the out-of-order data within that aggregation time range using the first background task, and then aggregates the aggregation result of the out-of-order data within that aggregation time range with the aggregation result of the ordered data within the same aggregation time range using the second background task.

[0125] As shown in Figure 7, in one possible embodiment, the background aggregation task is implemented in the second case, and the implementation process of step 505 includes the following steps 5051 and 5052:

[0126] Step 5051: Aggregate the first disordered data in the first disordered data file to obtain the aggregation result of the first disordered data.

[0127] In this embodiment, the database system aggregates the first out-of-order data according to a first background task to obtain an aggregated result of the first out-of-order data. In other words, the database system aggregates the first out-of-order data according to a first triggering condition to obtain an aggregated result of the first out-of-order data.

[0128] The system aggregates the first out-of-order data, optionally according to a specified time granularity, and updates the first out-of-order data file with the aggregation result, resulting in an aggregated first out-of-order data file. The specified time granularity indicates the length of the aggregation time range. Since users need to query the aggregation results of ordered and out-of-order data belonging to the same aggregation time range in practical applications, the database system can aggregate the first out-of-order data according to the aggregation time range used to aggregate the first ordered data. The database system first obtains multiple out-of-order data belonging to multiple aggregation time ranges from the first out-of-order data, and then aggregates the multiple first out-of-order data belonging to each aggregation time range to obtain the aggregation result of multiple out-of-order data. Specifically, when the timestamp of out-of-order data belongs to a certain aggregation time range, it is determined that the out-of-order data belongs to that aggregation time range. Furthermore, to distinguish the aggregation results of different aggregation time ranges, the database system can also add time attributes to the aggregation results. For example, adding a time range to the aggregation results indicates the aggregation time range corresponding to the aggregation result.

[0129] For example, as shown in step 503 above, the database system has four aggregation time windows: window1, window2, window3, and window4. Aggregating the first ordered data involves aggregating the ordered data belonging to each of these four aggregation time windows separately. When aggregating the first out-of-order data, the out-of-order data belonging to each of the four aggregation time windows can be filtered first. Then, the out-of-order data belonging to window1, window2, window3, and window4 can be aggregated separately, and the time range of the corresponding aggregation time window can be added to the aggregation result of each aggregation time window. In this way, the aggregated first out-of-order data file can be obtained.

[0130] Aggregating the first disordered data involves performing aggregation operations on multiple disordered data points included in the first disordered data set. For example, aggregation operations can include summation, averaging, counting, median calculation, standard deviation calculation, variance calculation, finding the maximum value, finding the minimum value, or product calculation.

[0131] As described above, aggregating the first out-of-order data is essentially a process of performing aggregation calculations on multiple out-of-order data to reduce the amount of data in the first out-of-order data. This makes the data more clustered, thereby reducing the number of query I / O operations when querying the first out-of-order data and thus improving query efficiency.

[0132] Step 5052: Aggregate the aggregation result of the first disordered data into the first ordered data file to obtain the aggregated first ordered data file.

[0133] In this embodiment, the database system aggregates the aggregation results of the first disordered data into the first ordered data file through a second background task, resulting in the aggregated first ordered data file. In other words, the database system aggregates the aggregation results of the first disordered data into the first ordered data file according to a second triggering condition, resulting in the aggregated first ordered data file.

[0134] Aggregating the aggregation results of the first disordered data into the first ordered data file to obtain the aggregated first ordered data file means aggregating the aggregation results of the first disordered data with the aggregation results of the first ordered data, and then updating the first ordered data file with the new aggregation result. In one possible embodiment, the aggregation results of the first disordered data and the aggregation results of the first ordered data are aggregated again, including aggregating aggregation results belonging to the same aggregation time range from both sides according to the aggregation time range.

[0135] Similarly, aggregating the aggregation results of the first disordered data with the aggregation results of the first ordered data involves performing an aggregation operation on the aggregation results of the first disordered data and the aggregation results of the first ordered data. For example, aggregation operations can include summation, averaging, counting, median calculation, standard deviation calculation, variance calculation, finding the maximum value, finding the minimum value, or product calculation.

[0136] By aggregating the aggregation results of the first disordered data into the first ordered data file, the aggregated first ordered data file stores more complete aggregation results, which can further reduce the recalculation overhead during the query process, reduce the number of query I / Os during the query, and thus improve query efficiency.

[0137] Step 506: Receive the first query request, which indicates a first time range, wherein both the first out-of-order data and the first ordered data belong to the first time range.

[0138] As described in step 505, in this embodiment, after the database system stores the aggregated results of ordered data and the out-of-order data, it can aggregate the out-of-order data into the aggregated results of ordered data through a background task, thereby obtaining a more complete aggregated result. Therefore, when a user needs to query data within a specified time range, if some of the out-of-order data within the specified time range has already been aggregated into the aggregated results of ordered data and a new aggregated result has been obtained, the database system can perform query calculations based on the new aggregated result, thereby reducing the computational overhead when the query arrives and improving query efficiency.

[0139] For example, assuming that both the first ordered data and the first disordered data belong to a first time range, when the database system receives a query request for data within the first time range, it can perform query calculations based on the aggregated first ordered data file to determine the query result of the first query request. In one possible embodiment, the first query request indicates the query for the aggregated results of data within the first time range.

[0140] Step 507: Determine the query result of the first query request based on the aggregated first ordered data file.

[0141] Since both the first ordered data and the first disordered data are data within a first time range, and the first disordered data has already been aggregated into the first ordered data file, the database system determines the query result of the first query request based on the aggregated first ordered data file. In this embodiment, after receiving the first query request, the database system can directly determine the query result based on the aggregated first ordered data file without needing to perform aggregation calculations on the first disordered data and the corresponding ordered data in the first ordered data file, thereby improving the response speed of the database system to the first query request.

[0142] Furthermore, in this embodiment, to achieve intelligent querying, the database system's query engine can select an appropriate query strategy based on the aggregation status of data within the specified time range of the query request. In other words, when a query request is received, the out-of-order data within the specified time range may have already been aggregated into the corresponding ordered data file, or some out-of-order data within the specified time range may not have been aggregated into the corresponding ordered data file. Therefore, the query engine selects the corresponding query strategy for each of these two data aggregation scenarios. When all the out-of-order data within the specified time range has been aggregated into the corresponding ordered data file, the query engine can directly query from the corresponding ordered data file to obtain the query results; when some out-of-order data within the specified time range has not been aggregated into the corresponding ordered data file, the query engine needs to first aggregate this portion of out-of-order data within the specified time range into the corresponding ordered data file before it can query from the corresponding ordered data file to obtain the query results.

[0143] In one possible embodiment, as shown in FIG9, the implementation process of step 507 includes the following steps 5071-5073:

[0144] Step 5071: Determine whether there is first target data. The first target data is out-of-order data within the first time range that has not been aggregated into the first ordered data file after aggregation.

[0145] As described above, when the first query request is received, the first out-of-order data within the first time range has already been aggregated into the first ordered data file. Therefore, the database system's query engine determines whether there is still out-of-order data within the first time range that has not been aggregated into the aggregated first ordered data file.

[0146] As described above, in one possible embodiment, each data file carries time attribute information, which indicates the range of time in which the data recorded in the data file was generated. In this embodiment, upon receiving a query request, the database system first searches among the currently stored out-of-order data files for any out-of-order data file whose time range, as indicated by its time attribute information, intersects with the first time range. If no such file exists, it means that the aforementioned first target data does not exist. If some out-of-order data files have time ranges whose time attribute information intersects with the first time range, then the system further iterates through these out-of-order data files to determine whether the aforementioned first target data exists.

[0147] Step 5072: If the first target data does not exist, perform a query in the aggregated first ordered data file to obtain the query result of the first query request.

[0148] In the absence of the initial target data, when the first query request arrives, the database system already contains the latest aggregation results in the currently stored first ordered data file after aggregation. Therefore, the database system can directly query the first ordered data file after aggregation to obtain the query results for the first query request. In this case, the database system achieves the goal of eliminating the overhead of recalculating out-of-order data during the query process.

[0149] Step 5073: If the first target data exists, aggregate the first target data into the aggregated first ordered data file and then perform a query to obtain the query result of the first query request.

[0150] When the first target data exists, the first query request arrives, but the first ordered data file after aggregation currently stored in the database system is not the latest aggregation result. Before executing the query, the database system needs to aggregate the first target data into the first ordered data file after aggregation, and then perform the query in the latest ordered data file after aggregation to obtain the latest query result.

[0151] As shown in Figure 8, when the database system provides multiple out-of-order data processing strategies for tenants to choose from, the method may optionally include the following steps before step 504:

[0152] Step 508: Receive data processing instructions, which specify the processing strategy for out-of-order data.

[0153] Tenants can send instructions to the database system through an interactive interface to specify a processing strategy for out-of-order data. The processing strategy in the instruction indicates the target method for processing the out-of-order data. After receiving the data processing instruction from the tenant, the database system can parse the instruction to obtain the target method for processing the out-of-order data, and then process the out-of-order data according to that method. For example, as described in step 504 above, the target method could be to discard the out-of-order data or to recalculate it. When the target method is to recalculate the out-of-order data, the database system needs to perform the aggregation calculation described above on the out-of-order data. The implementation method of the data processing instruction can be referred to the description in step 504, and the number of steps will not be repeated here. In one possible embodiment, the data processing instruction specifies that the processing strategy for out-of-order data is to recalculate the out-of-order data. In this case, the implementation flow of the data management method provided in this application is shown in Figure 8. In this embodiment, step 504 includes: step 504a, storing the first out-of-order data into a first out-of-order data file when the out-of-order data processing strategy indicates that the out-of-order data should be recalculated; step 505 includes: step 505a, aggregating the first out-of-order data into a first ordered data file when the out-of-order data processing strategy indicates that the out-of-order data should be recalculated, to obtain an aggregated first ordered data file; step 5051 includes: step 5051a, aggregating the first out-of-order data when the out-of-order data processing strategy indicates that the out-of-order data should be recalculated, to obtain an aggregated result of the first out-of-order data; step 5052 includes: step 5052a, aggregating the aggregated result of the first out-of-order data into a first ordered data file when the out-of-order data processing strategy indicates that the out-of-order data should be recalculated, to obtain an aggregated first ordered data file. As described above, there are two ways to implement the background aggregation task in this application. When the background aggregation task is implemented in the second scenario, in one possible embodiment, before the query request arrives, the out-of-order data within the time range specified by the query request has been processed by the first background task but not yet by the second background task. In this case, the data management method further includes steps 601 to 608 as shown in Figure 10. It should be noted that the implementation process and various possible implementation methods of each step in Figure 10 can be found in the relevant descriptions of similar steps in the first data-related embodiment above, and will not be repeated here.For example, the implementation process of step 601 refers to the implementation process of step 501, the implementation process of step 602 refers to the implementation process of step 502, the implementation process of step 603 refers to the implementation process of step 503, the implementation process of step 604 refers to the implementation process of step 504, the implementation process of step 605 refers to the implementation process of step 5051, the implementation process of step 606 refers to the implementation process of step 506, the implementation process of step 607 refers to the implementation process of step 5052, and the implementation process of step 608 refers to the implementation process of step 507.

[0154] Step 601: Receive multiple second data.

[0155] Step 602: Based on the time attribute of each second data received, determine the second ordered data and the second out-of-order data from the multiple second data, wherein the second ordered data includes multiple second data belonging to the ordered data, and the second out-of-order data includes multiple second data belonging to the out-of-order data.

[0156] Step 603: Store the result of aggregating the second ordered data into the second ordered data file.

[0157] Step 604: Store the second disordered data into the second disordered data file.

[0158] Step 605: Before receiving the second query request, aggregate the second disordered data in the second disordered data file to obtain the aggregated result of the second disordered data.

[0159] Similar to the processing of the first out-of-order data, the database system aggregates the second out-of-order data through a first background task to obtain the aggregated result of the second out-of-order data. In other words, the database system aggregates the second out-of-order data according to the first triggering condition to obtain the aggregated result of the second out-of-order data.

[0160] Step 606: Receive a second query request. The second query request indicates a second time range, wherein both the second ordered data and the second out-of-order data belong to the second time range.

[0161] Step 607: After receiving the second query request, aggregate the aggregation results of the second disordered data into the second ordered data file to obtain the aggregated second ordered data file.

[0162] Unlike step 5052, step 607 is not triggered by a background aggregation task, but rather by the query strategy after receiving the second query request. As described above, the database system's query engine can select an appropriate query strategy based on the aggregation of data within the time range specified in the query request. Therefore, after receiving the second query request, the database system will first perform pre-query recalculation, that is, aggregate the aggregation results of the second out-of-order data in the second out-of-order data file into the second ordered data file.

[0163] Step 608: Determine the query result of the second query request based on the aggregated second ordered data file.

[0164] In the embodiment shown in Figure 10, although the second out-of-order data has not yet been aggregated into the second ordered data file, since the second out-of-order data itself has already been aggregated, compared to the second out-of-order data not being aggregated, it also reduces some of the aggregation calculation costs after the arrival of the second query request, which helps to speed up the database system's response to the second query request.

[0165] To reduce the storage cost of the out-of-order data queue, the database system can perform real-time processing on the out-of-order data entering the queue. As shown in Figure 11, in one possible embodiment, the data management method further includes steps 701 to 707. In this embodiment, when multiple out-of-order data belonging to the same aggregation time range are received consecutively, the database system will promptly aggregate these multiple out-of-order data and directly store the aggregation result in the out-of-order data queue. Then, the aggregation result will be stored from the out-of-order data queue into an out-of-order data file.

[0166] Step 701: Receive multiple third-party data.

[0167] Step 702: Based on the time attribute of each third data received, determine the third ordered data and the third out-of-order data from multiple third data. The third ordered data includes multiple third data belonging to the ordered data, and the third out-of-order data includes multiple third data belonging to the out-of-order data. The multiple third data belonging to the out-of-order data belong to the same time window and are received in adjacent order.

[0168] The order in which multiple data are received is adjacent, meaning that the multiple data are received sequentially and consecutively. For example, if data 1, data 2, and data 3 are received sequentially and consecutively, then data 1, data 2, and data 3 are said to be received in an adjacent order. However, if data 1, data 2, data 3, and data 4 are received sequentially and consecutively, then data 1, data 2, and data 4 cannot be said to be received in an adjacent order, nor can data 1, data 3, and data 4 be said to be received in an adjacent order.

[0169] Step 703: Store the result of aggregating the third ordered data into the third ordered data file.

[0170] Step 704: Store the result of aggregating the third disordered data into the third disordered data file.

[0171] Step 705: Before receiving the third query request, aggregate the aggregation results of the third disordered data from the third disordered data file into the third ordered data file to obtain the aggregated third ordered data file.

[0172] Similar to the processing of the first out-of-order data, the database system aggregates the aggregation results of the third out-of-order data into the third ordered data file through a background aggregation task.

[0173] Step 706: Receive a third query request. The third query request indicates a third time range, wherein both the third out-of-order data and the third ordered data belong to the third time range.

[0174] Step 707: Determine the query result of the third query request based on the aggregated third ordered data file.

[0175] It should be noted that the implementation process and various possible implementation methods of each step in Figure 11 are described in the corresponding similar steps in the embodiments related to the first data mentioned above, and will not be repeated here. For example, the implementation process of step 701 refers to the implementation process of step 501, the implementation process of step 702 refers to the implementation process of step 502, the implementation process of step 703 refers to the implementation process of step 503, the implementation process of step 705 refers to the implementation process of step 505, the implementation process of step 706 refers to the implementation process of step 506, and the implementation process of step 707 refers to the implementation process of step 507.

[0176] In summary, in the data management method provided in this application embodiment, after obtaining the first ordered data and the first disordered data, the first disordered data can be aggregated with the first ordered data in the first ordered data file according to the triggering condition to obtain the aggregated first ordered data file, thus realizing the aggregation calculation of the first disordered data and the first ordered data. When a first query request is received, and the first query request indicates that data belonging to a first time range is queried, if both the first disordered data and the first ordered data belong to the first time range, the query result of the first query request can be determined based on the data in the aggregated first ordered data file. At this time, since the aggregated first ordered data file stores the aggregation result of the first ordered data and the first disordered data, the query recalculation during the process of obtaining the query result can be reduced or even eliminated, thereby improving query efficiency. Furthermore, since this process realizes the aggregation calculation of the first disordered data and the first ordered data, rather than discarding the disordered data, the integrity of the data can be guaranteed, which helps to improve the accuracy of the query result.

[0177] Meanwhile, in this application, the database system can adaptively determine a query strategy that matches the aggregation of the data to be queried based on the query request, thereby achieving intelligent querying without the user's awareness.

[0178] It should be noted that the order of steps in the data management method provided in this application that have no logical dependencies can be appropriately adjusted, and steps can also be added or removed as appropriate. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and therefore will not be elaborated further.

[0179] The data management method of this application embodiment has been described above. Corresponding to the above method, this application embodiment also provides a data management device. Figure 12 is a schematic diagram of the structure of a data management device provided in this application embodiment. Based on the multiple modules shown in Figure 12, the data management device 130 can perform all or part of the operations shown in Figures 5, 8, 10 and 11. It should be understood that the data management device may include more additional components or combine some of the shown components than the modules shown, and this application embodiment does not limit this. Optionally, the data management device can be applied to a cloud management platform. The cloud management platform is used to manage the infrastructure that provides at least one cloud service. The infrastructure includes at least one cloud data center. Each cloud data center is provided with multiple servers. One or any combination of at least one cloud service is deployed in at least one server of the infrastructure. As shown in Figure 12, the data management device 130 includes:

[0180] The data receiving module 1301 is used to receive multiple first data.

[0181] The data discrimination module 1302 is used to determine the first ordered data and the first disordered data from multiple first data based on the time attribute of each first data received.

[0182] The first storage module 1303 is used to store the result of aggregating the first ordered data into the first ordered data file.

[0183] The second storage module 1304 is used to store the first out-of-order data into the first out-of-order data file.

[0184] The aggregation calculation module 1305 is used to aggregate the first disordered data from the first disordered data file into the first ordered data file before receiving the first query request, so as to obtain the aggregated first ordered data file.

[0185] The request receiving module 1306 is used to receive a first query request, which indicates a first time range, wherein both the first out-of-order data and the first ordered data belong to the first time range.

[0186] The request response module 1307 is used to determine the query result of the first query request based on the aggregated first ordered data file.

[0187] In one possible implementation, before receiving the first query request, the aggregation calculation module 1305 is used to: aggregate the first disordered data in the first disordered data file to obtain the aggregation result of the first disordered data; and aggregate the aggregation result of the first disordered data into the first ordered data file to obtain the aggregated first ordered data file.

[0188] In one possible implementation, the data receiving module 1301 is also used to receive a plurality of second data.

[0189] The data discrimination module 1302 is also used to determine the second ordered data and the second disordered data from multiple second data based on the time attribute of each second data received.

[0190] The first storage module 1303 is also used to store the result of aggregating the second ordered data into the second ordered data file.

[0191] The second storage module 1304 is also used to store the second out-of-order data into a second out-of-order data file.

[0192] The aggregation calculation module 1305 is also used to aggregate the second disordered data in the second disordered data file to obtain the aggregation result of the second disordered data before receiving the second query request.

[0193] The request receiving module 1306 is further configured to receive a second query request, the second query request indicating a second time range, wherein both the second ordered data and the second out-of-order data belong to the second time range.

[0194] The aggregation calculation module 1305 is also used to, after receiving the second query request, aggregate the aggregation results of the second disordered data into the second ordered data file to obtain the aggregated second ordered data file.

[0195] The request response module 1307 is also used to determine the query result of the second query request based on the aggregated second ordered data file.

[0196] In one possible implementation, the request response module 1307 is used to: determine whether first target data exists, wherein the first target data is out-of-order data belonging to a first time range and not aggregated into the aggregated first ordered data file; if the first target data does not exist, perform a query in the aggregated first ordered data file to obtain the query result of the first query request; if the first target data exists, aggregate the first target data into the aggregated first ordered data file and then perform a query to obtain the query result of the first query request.

[0197] In one possible implementation, the data receiving module 1301 is also used to receive multiple third data.

[0198] The data discrimination module 1302 is further configured to determine third ordered data and third disordered data from multiple third data based on the time attribute of each third data received, wherein the third ordered data includes multiple third data belonging to ordered data, the third disordered data includes multiple third data belonging to disordered data, and the multiple third data belonging to disordered data belong to the same time window and are received in adjacent order.

[0199] The first storage module 1303 is also used to store the result of aggregating the third ordered data into the third ordered data file.

[0200] The second storage module 1304 is also used to store the result of aggregating the third out-of-order data into the third out-of-order data file.

[0201] The aggregation calculation module 1305 is also used to aggregate the aggregation results of the third disordered data from the third disordered data file into the third ordered data file before receiving the third query request, so as to obtain the aggregated third ordered data file.

[0202] The request receiving module 1306 is also used to receive a third query request, the third query request indicating a third time range, wherein both the third out-of-order data and the third ordered data belong to the third time range.

[0203] The request response module 1307 is also used to determine the query result of the third query request based on the aggregated third ordered data file.

[0204] In one possible implementation, the request receiving module 1306 is further configured to: receive a data processing instruction, which specifies a processing strategy for out-of-order data.

[0205] In one possible implementation, the timestamp of ordered data belongs to the system time window in which the ordered data is received, while the timestamp of out-of-order data does not belong to the system time window in which the out-of-order data is received.

[0206] Here, the detailed working process of the data receiving module 1301, data discrimination module 1302, first storage module 1303, second storage module 1304, aggregation calculation module 1305, request receiving module 1306, and request response module 1307 is described in the preceding method embodiments. For example, the data receiving module 1301 receives multiple first data using the aforementioned step 501. The data discrimination module 1302 determines first ordered data and first out-of-order data from the multiple first data based on the time attribute of each received first data using the aforementioned step 502. The first storage module 1303 stores the aggregation result of the first ordered data into a first ordered data file using the aforementioned step 503. The second storage module 1304 stores the first out-of-order data into a first out-of-order data file using the aforementioned step 504. The aggregation calculation module 1305 aggregates the first out-of-order data from the first out-of-order data file into the first ordered data file using the aforementioned step 505, obtaining the aggregated first ordered data file. The request receiving module 1306 receives the first query request using the aforementioned step 506. The first query request indicates a first time range, wherein both the first out-of-order data and the first ordered data belong to the first time range. The request response module 1307 determines the query result of the first query request based on the aggregated first ordered data file using the aforementioned step 507. The embodiments of this application will not be described again here.

[0207] The data receiving module 1301, data discrimination module 1302, first storage module 1303, second storage module 1304, aggregation calculation module 1305, request receiving module 1306, and request response module 1307 can all be implemented in software or in hardware. For example, the implementation of the data receiving module 1301 will be described below. Similarly, the implementation of the data discrimination module 1302, first storage module 1303, second storage module 1304, aggregation calculation module 1305, request receiving module 1306, and request response module 1307 can refer to the implementation of the receiving module 1301.

[0208] As an example of a software functional unit, the receiving module 1301 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the receiving module 1301 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one cloud data center or multiple geographically proximate cloud data centers. Typically, a region may include multiple AZs.

[0209] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0210] As an example of a hardware functional unit, the receiving module 1301 may include at least one computing device, such as a server. Alternatively, the receiving module 1301 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0211] The multiple computing devices included in the receiving module 1301 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the receiving module 1301 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the receiving module 1301 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0212] It should be noted that, in other embodiments, any one of the data receiving module 1301, data discrimination module 1302, first storage module 1303, second storage module 1304, aggregation calculation module 1305, request receiving module 1306, and request response module 1307 can be used to execute any step in the data management method. The steps implemented by the receiving module 1301, data discrimination module 1302, first storage module 1303, second storage module 1304, aggregation calculation module 1305, request receiving module 1306, and request response module 1307 can be specified as needed. Different steps in the data management method are implemented by the receiving module 1301, data discrimination module 1302, first storage module 1303, second storage module 1304, aggregation calculation module 1305, request receiving module 1306, and request response module 1307 to achieve all the functions of the data management device.

[0213] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of each component described above can be referred to the corresponding content in the foregoing method embodiments, and will not be repeated here.

[0214] The following provides examples illustrating the basic hardware structures involved in the embodiments of this application.

[0215] This application also provides a computing device 1400. As shown in FIG13, the computing device 1400 includes: a bus 1402, a processor 1404, a memory 1406, and a communication interface 1408. The processor 1404, the memory 1406, and the communication interface 1408 communicate with each other via the bus 1402. The computing device 1400 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1400.

[0216] Bus 1402 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 14, but this does not imply that there is only one bus or one type of bus. Bus 1402 can include pathways for transmitting information between various components of computing device 1400 (e.g., memory 1406, processor 1404, communication interface 1408).

[0217] The processor 1404 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0218] The memory 1406 may include volatile memory, such as random access memory (RAM). The processor 1404 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0219] The memory 1406 stores executable program code, and the processor 1404 executes this executable program code to implement the functions of the aforementioned receiving module 1301, data discrimination module 1302, first storage module 1303, second storage module 1304, aggregation calculation module 1305, request receiving module 1306, and request response module 1307, thereby realizing the data management method. That is, the memory 1406 stores instructions for executing the data management method.

[0220] The communication interface 1408 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1400 and other devices or communication networks.

[0221] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0222] As shown in Figure 14, the computing device cluster includes at least one computing device 1400. The memory 1406 of one or more computing devices 1400 in the computing device cluster may store the same instructions for executing data management methods.

[0223] In some possible implementations, the memory 1406 of one or more computing devices 1400 in the computing device cluster may also store partial instructions for executing data management methods. In other words, a combination of one or more computing devices 1400 can jointly execute instructions for executing data management methods.

[0224] It should be noted that the memory 1406 in different computing devices 1400 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data management device. That is, the instructions stored in the memory 1406 of different computing devices 1400 can implement the functions of one or more modules among the data receiving module 1301, data discrimination module 1302, first storage module 1303, second storage module 1304, aggregation computing module 1305, request receiving module 1306, and request response module 1307.

[0225] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 15 illustrates one possible implementation. As shown in Figure 15, two computing devices 1400A and 1400B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 1406 in computing device 1400A stores instructions for executing the functions of the receiving module 1301. Simultaneously, the memory 1406 in computing device 1400B stores instructions for executing the functions of the data discrimination module 1302, the first storage module 1303, the second storage module 1304, the aggregation computing module 1305, the request receiving module 1306, and the request response module 1307.

[0226] It should be understood that the functions of computing device 1400A shown in Figure 15 can also be performed by multiple computing devices 1400. Similarly, the functions of computing device 1400B can also be performed by multiple computing devices 1400.

[0227] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device clusters in Figures 14 and 15. The difference is that the memory 1406 of one or more computing devices 1400 in this computing device cluster can store the same instructions for executing data management methods.

[0228] In some possible implementations, the memory 1406 of one or more computing devices 1400 in the computing device cluster may also store partial instructions for executing data management methods. In other words, a combination of one or more computing devices 1400 can jointly execute instructions for executing data management methods.

[0229] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a data management method.

[0230] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a data management method, or instruct the computing device to perform a data management method.

[0231] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0232] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the raw data and executable code involved in this application were obtained with full authorization.

[0233] In the embodiments of this application, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "at least one" refers to one or more, and the term "multiple" refers to two or more, unless otherwise expressly defined.

[0234] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0235] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.< / groupkey> < / groupkey>

Claims

1. A data management method, characterized in that, The method includes: Receive multiple first data; Based on the time attribute of each first data received, first ordered data and first disordered data are determined from the plurality of first data, wherein the first ordered data includes a plurality of first data belonging to ordered data, and the first disordered data includes a plurality of first data belonging to disordered data; The result of aggregating the first ordered data is stored in the first ordered data file; The first disordered data is stored in the first disordered data file; Before receiving the first query request, the first disordered data is aggregated from the first disordered data file into the first ordered data file to obtain the aggregated first ordered data file; Receive the first query request, the first query request indicating a first time range, wherein both the first disordered data and the first ordered data belong to the first time range; The query result of the first query request is determined based on the aggregated first ordered data file.

2. The method according to claim 1, characterized in that, Before receiving the first query request, the first disordered data is aggregated into the first ordered data file to obtain the aggregated first ordered data file, including: The first disordered data in the first disordered data file is aggregated to obtain the aggregated result of the first disordered data. The aggregation result of the first disordered data is aggregated into the first ordered data file to obtain the aggregated first ordered data file.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Receive multiple second data; Based on the time attribute of each second data received, second ordered data and second out-of-order data are determined from the plurality of second data, wherein the second ordered data includes a plurality of second data belonging to ordered data, and the second out-of-order data includes a plurality of second data belonging to out-of-order data; The result of aggregating the second ordered data is stored in the second ordered data file; The second disordered data is stored in the second disordered data file; Before receiving the second query request, the second disordered data in the second disordered data file is aggregated to obtain the aggregated result of the second disordered data; Receive the second query request, which indicates a second time range, wherein both the second ordered data and the second out-of-order data belong to the second time range; After receiving the second query request, the aggregation result of the second disordered data is aggregated into the second ordered data file to obtain the aggregated second ordered data file; The query result of the second query request is determined based on the aggregated second ordered data file.

4. The method according to any one of claims 1 to 3, characterized in that, Determining the query result of the first query request based on the aggregated first ordered data file includes: Determine whether there is first target data, where the first target data is out-of-order data belonging to the first time range and has not been aggregated into the first ordered data file after aggregation; If the first target data does not exist, a query is performed in the aggregated first ordered data file to obtain the query result of the first query request; If the first target data exists, the first target data is aggregated into the aggregated first ordered data file and then queried to obtain the query result of the first query request.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Receive multiple third-party data; Based on the time attribute of each third data received, third ordered data and third out-of-order data are determined from the plurality of third data, wherein the third ordered data includes a plurality of third data belonging to ordered data, the third out-of-order data includes a plurality of third data belonging to out-of-order data, and the plurality of third data belonging to out-of-order data belong to the same time window and are received in adjacent order. The result of aggregating the third ordered data is stored in the third ordered data file; The result of aggregating the third disordered data is stored in the third disordered data file; Before receiving the third query request, the aggregation result of the third disordered data is aggregated from the third disordered data file into the third ordered data file to obtain the aggregated third ordered data file; Receive the third query request, which indicates a third time range, wherein both the third disordered data and the third ordered data belong to the third time range; The query result of the third query request is determined based on the aggregated third ordered data file.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Receive a data processing instruction, which specifies a processing strategy for out-of-order data.

7. The method according to any one of claims 1 to 6, characterized in that, The timestamp of the ordered data belongs to the system time window in which the ordered data is received, while the timestamp of the out-of-order data does not belong to the system time window in which the out-of-order data is received.

8. A data management device, characterized in that, The device includes: The data receiving module is used to receive multiple first data. The data discrimination module is used to determine first ordered data and first disordered data from the plurality of first data based on the time attribute of each first data received, wherein the first ordered data includes a plurality of first data belonging to ordered data, and the first disordered data includes a plurality of first data belonging to disordered data; The first storage module is used to store the result of aggregating the first ordered data into the first ordered data file; The second storage module is used to store the first disordered data into the first disordered data file; The aggregation calculation module is used to aggregate the first disordered data from the first disordered data file into the first ordered data file before receiving the first query request, so as to obtain the aggregated first ordered data file. A request receiving module is used to receive the first query request, wherein the first query request indicates a first time range, and both the first disordered data and the first ordered data belong to the first time range; The request-response module is used to determine the query result of the first query request based on the aggregated first ordered data file.

9. The apparatus according to claim 8, characterized in that, Before receiving the first query request, the aggregation calculation module is used to: The first disordered data in the first disordered data file is aggregated to obtain the aggregated result of the first disordered data. The aggregated result of the first disordered data is then aggregated into the first ordered data file to obtain the aggregated first ordered data file.

10. The apparatus according to claim 8 or 9, characterized in that, The data receiving module is also used to receive multiple second data; The data discrimination module is further configured to determine second ordered data and second disordered data from the plurality of second data based on the time attribute of each second data received, wherein the second ordered data includes a plurality of second data belonging to ordered data, and the second disordered data includes a plurality of second data belonging to disordered data; The first storage module is further configured to store the result of aggregating the second ordered data into a second ordered data file; The second storage module is further configured to store the second out-of-order data into a second out-of-order data file; The aggregation calculation module is further configured to aggregate the second disordered data in the second disordered data file to obtain the aggregation result of the second disordered data before receiving the second query request. The request receiving module is further configured to receive the second query request, the second query request indicating a second time range, wherein both the second ordered data and the second disordered data belong to the second time range; The aggregation calculation module is further configured to, after receiving the second query request, aggregate the aggregation result of the second disordered data into the second ordered data file to obtain the aggregated second ordered data file; The request response module is further configured to determine the query result of the second query request based on the aggregated second ordered data file.

11. The apparatus according to any one of claims 8 to 10, characterized in that, The request-response module is used for: Determine whether there is first target data, where the first target data is out-of-order data belonging to the first time range and has not been aggregated into the first ordered data file after aggregation; If the first target data does not exist, a query is performed in the aggregated first ordered data file to obtain the query result of the first query request; If the first target data exists, the first target data is aggregated into the aggregated first ordered data file and then queried to obtain the query result of the first query request.

12. The apparatus according to any one of claims 8 to 11, characterized in that, The data receiving module is also used to receive multiple third-party data; The data discrimination module is further configured to determine third ordered data and third disordered data from the plurality of third data based on the time attribute of each third data received, wherein the third ordered data includes a plurality of third data belonging to ordered data, the third disordered data includes a plurality of third data belonging to disordered data, and the plurality of third data belonging to disordered data belong to the same time window and are received in adjacent order. The first storage module is further configured to store the result of aggregating the third ordered data into a third ordered data file; The second storage module is further configured to store the result of aggregating the third disordered data into a third disordered data file; The aggregation calculation module is further configured to, before receiving the third query request, aggregate the aggregation result of the third disordered data from the third disordered data file into the third ordered data file to obtain the aggregated third ordered data file; The request receiving module is further configured to receive the third query request, the third query request indicating a third time range, wherein both the third disordered data and the third ordered data belong to the third time range; The request response module is further configured to determine the query result of the third query request based on the aggregated third ordered data file.

13. The apparatus according to any one of claims 8-12, characterized in that, The request receiving module is also used for: Receive a data processing instruction, which specifies a processing strategy for out-of-order data.

14. The apparatus according to any one of claims 8 to 13, characterized in that, The timestamp of the ordered data belongs to the system time window in which the ordered data is received, while the timestamp of the out-of-order data does not belong to the system time window in which the out-of-order data is received.

15. A computing device cluster, characterized in that, The system includes multiple computing devices, each comprising multiple processors and multiple memories, the multiple memories storing program instructions, and the multiple processors executing the program instructions to cause the cluster of computing devices to perform the method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, Includes program instructions that, when executed on a computing device, cause the computing device to perform the method as described in any one of claims 1 to 7.

17. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 7.

Images