Data processing method and related device
Patent Information
- Application Number
- PCT/CN2026/073320
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-03-21
- Filing Date
- 2026-01-19
- Publication Date
- 2026-09-24
Smart Images
Figure CN2026073320_24092026_PF_FP_ABST
Abstract
Description
A data processing method and related equipment
[0001] This application claims priority to Russian Patent Application No. RU2025106793, filed on March 21, 2025, entitled "A Data Processing Method and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of communications, and more particularly to a data processing method and related equipment. Background Technology
[0003] A consensus protocol is an algorithm or protocol used in a distributed system to reach agreement on a value or state among multiple nodes. Based on consensus protocols, multi-replica consistency can be achieved. Multi-replica consistency refers to ensuring that data remains consistent across multiple nodes or replicas in a distributed system, thereby improving system reliability.
[0004] In one data processing method, the leader node processes all client requests based on a consensus protocol and replicates the requests to other follower nodes. Specifically, the leader node replicates client requests based on fixed packaging rules. Since the amount of data indicated by these fixed client rules is constant, it means that even with fluctuations in system load, the leader node replicates data according to these fixed rules, limiting data processing performance. Summary of the Invention
[0005] This application provides a data processing method and related equipment for improving data processing performance.
[0006] In a first aspect, this application provides a data processing method, which is applied to a first network device, and the method includes:
[0007] The system detects the amount of requested data in the message queue, which stores requests from terminal devices. Based on this amount, it adjusts the data size of a window, which indicates the amount of data the first network device replicates for each request. Then, based on the adjusted window size, it replicates requests retrieved from the message queue, ensuring the adjusted window size is greater than or equal to the replicated data size. Finally, it sends a replication request to the second network device, achieving consistency between the replicas of the first and second network devices.
[0008] In this application, the first network device is provided with a window, which defines the amount of data requested by the first network device in each copy. The amount of data in the window is adjusted based on the amount of data requested in the message queue, and is not constant. This allows the amount of data processed by the first network device in each batch to vary, better matching the actual communication state of the system and thus improving data processing performance.
[0009] In one possible implementation of the first aspect, the first network device can adjust the window data size based on the requested data size in several ways. Optionally, in a scenario where the requested data size exceeds a data size threshold, indicating a data backlog in the message queue, the first network device increases the window data size. Optionally, in a scenario where the requested data size is less than the data size threshold, indicating that the current system throughput can easily cover the load demand, the first network device reduces the window data size.
[0010] In this application, when the requested data volume exceeds a data volume threshold, the system is under high load due to data backlog in the message queue. The first network device increases the window's data volume to meet the high load demand and achieve high throughput. Conversely, when the requested data volume is less than the data volume threshold, the system is under low load. The first network device reduces the window's data volume, thus reducing the data volume requested for each copy and consequently reducing data processing latency. In summary, based on different adjustment methods, the first network device optimizes data processing performance under different load conditions.
[0011] In one possible implementation of the first aspect, after increasing the data size of the window, the first network device also acquires the data throughput between the first network device and the second network device. If the data throughput begins to decrease, the first network device no longer increases the data size of the window, but keeps the data size of the window unchanged.
[0012] In this application, the first network device may not continuously increase the data volume of the window. In addition to detecting the amount of data requested in the message queue, it also obtains the data throughput between the first and second network devices to comprehensively evaluate whether it is necessary to continue increasing the data volume of the window. In the scheme where the data throughput begins to decline, keeping the data volume of the window constant, and comprehensively evaluating the balance between system load and device throughput, is conducive to maintaining the stability of data processing.
[0013] In one possible implementation of the first aspect, the first network device increases the amount of data in the window linearly. Linear increases improve data processing stability and prevent instability caused by sudden changes in the amount of data in the window.
[0014] In one possible implementation of the first aspect, the first network device reduces the amount of data in the window linearly. Linear reduction also improves data processing stability, avoiding instability caused by sudden changes in the amount of data in the window.
[0015] Secondly, this application provides a network device, including:
[0016] The processing unit is used to detect the amount of data requested in the message queue, which is used to store requests from terminal devices.
[0017] The processing unit is also configured to adjust the data size of the window based on the requested data size, wherein the data size of the window indicates the requested data size copied by the first network device each time.
[0018] The processing unit is also used to copy requests retrieved from the message queue based on the adjusted window's data volume, wherein the adjusted window's data volume is greater than or equal to the copied data volume of the requests retrieved from the message queue.
[0019] The sending unit is used to send a copy request to the second network device.
[0020] The network device provided in the second aspect is regarded as the first network device, used to implement the method shown in the first aspect or any possible implementation of the first aspect, as detailed above, and will not be repeated here.
[0021] Thirdly, this application provides a computer device including a processor and a memory, wherein the processor stores instructions that, when executed on the processor, implement the method shown in the first aspect or any possible implementation of the first aspect.
[0022] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a processor, implement the method shown in the first aspect or any possible implementation of the first aspect.
[0023] Fifthly, this application provides a computer program product that, when executed on a processor, implements the method shown in the first aspect or any possible implementation of the first aspect.
[0024] In a sixth aspect, embodiments of this application provide a chip including at least one processor and a communication interface, the communication interface and at least one processor being interconnected via a circuit, the at least one processor being used to run computer programs or instructions to perform the methods shown in the first aspect or any possible implementation of the first aspect.
[0025] The communication interface in the chip can be an input / output interface, pins, or circuits.
[0026] In one possible implementation of the sixth aspect, the chip further includes at least one memory storing computer programs or instructions. This memory can be an internal storage unit within the chip, such as a register or cache.
[0027] The beneficial effects shown in any of the second to sixth aspects are similar to those of the first aspect or any possible implementation of the first aspect, and will not be repeated here. Attached Figure Description
[0028] Figure 1 is a schematic diagram of the system architecture provided in an embodiment of this application;
[0029] Figure 2 is a flowchart illustrating a data processing method provided in an embodiment of this application.
[0030] Figure 3 is another flowchart illustrating the data processing method provided in an embodiment of this application;
[0031] Figure 4 is a schematic diagram of a network device provided in an embodiment of this application;
[0032] Figure 5 is another structural schematic diagram of the network device provided in an embodiment of this application. Detailed Implementation
[0033] This application provides a data processing method and related equipment to improve data processing performance.
[0034] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.
[0035] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses. Additionally, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can be expressed as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0036] First, the proper nouns and related concepts that may be involved in the embodiments of this application will be explained.
[0037] 1) Message queue.
[0038] A message queue can be understood as a container for storing messages, such as requests from terminal devices. The two parties involved in message passing are called producers and consumers. Producers are responsible for sending messages to the message queue, and consumers are used to read messages from the message queue.
[0039] Message queues employ a first-in, first-out (FIFO) data structure, meaning that messages that enter the queue first are processed by the consumer first, ensuring data ordering. Furthermore, message queues continuously store messages, preventing message congestion due to slow consumer read speeds, thus ensuring non-blocking message delivery. Based on message queues, even in high-concurrency scenarios, each message can be reliably received by the consumer.
[0040] 2) Consensus Protocol.
[0041] In layman's terms, a consensus protocol is an agreement among multiple participants to reach consensus on a specific matter. Specifically, in a distributed system, network instability, node malfunctions, or malicious behavior can lead to discrepancies in the data across different nodes. A consensus protocol ensures that all honest nodes in the system can reach agreement based on the same data and rules.
[0042] In practical applications, there are many possible implementations of consensus protocols, such as the Raft consensus algorithm, the Paxos consensus algorithm, and the Byzantine fault tolerance (BFT) algorithm, etc., which are not limited here.
[0043] 3) Multi-replica consistency.
[0044] Multi-replica consistency refers to ensuring a consistent state among data replicas when data is copied to multiple nodes or replicas in a distributed system. This is to improve system availability, reliability, and performance, because even if some nodes fail, other nodes can still provide services, and users can get the same view of data regardless of which node they access. Consensus protocols can be used to achieve multi-replica consistency.
[0045] Please refer to Figure 1 below, which is a schematic diagram of the system architecture provided in the embodiment of this application.
[0046] As shown in Figure 1, multiple terminal devices act as producers in a message queue, sending requests to the message queue to request services provided by a first network device. The first network device acts as a consumer in the message queue, consuming requests sequentially according to their entry order. Request consumption can be understood as retrieving a request from the message queue, or the message queue actively sending a request to the first network device. After successfully retrieving a request, the first network device removes the request from the message queue.
[0047] The first network device communicates with at least one second network device based on a consensus protocol, ensuring multi-replica consistency among these network devices. In the embodiment shown in Figure 1, network devices 1 to N are all second network devices. Furthermore, after the first network device retrieves a request from the message queue, it replicates the retrieved request and sends the replicated request to each second network device, ensuring that both the first network device and each second network device receive the same request, thereby achieving multi-replica consistency.
[0048] Optionally, the request sent by the first network device to each second network device can be a single request or a batch of requests. The specific implementation process will be explained in detail later and will not be elaborated here.
[0049] Furthermore, the first network device and multiple second network devices can be understood as a consensus cluster. The first network device can be considered the master node, or leader node, of this cluster. The second network devices can be considered the follower nodes, or backup nodes, of this cluster. If the first network device becomes unavailable, any second network device can replace the first network device, that is, any second network device can replace the first network device and become the new first network device.
[0050] In some alternative implementations, the first network device and the second network device can be in the form of a server or other devices that use application layer protocols, and no specific limitation is made here.
[0051] In some alternative implementations, consensus cluster deployment can also be applied in cloud computing scenarios. The network device can be a cloud server or a business instance (including virtual machines, containers, dedicated hosts, etc.) running on a cloud server, and there are no specific limitations here.
[0052] It should be noted that the terminal device mentioned in the embodiments of this application can be a device with wireless transceiver function, or a terminal. Specifically, it can refer to user equipment (UE), access terminal, subscriber unit, user station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless communication equipment, user agent, or user device. Terminal devices can also be satellite phones, cellular phones, smartphones, wireless data cards, wireless modems, machine-type communication devices, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices, communication devices mounted on high-altitude aircraft, wearable devices, drones, robots, terminals in device-to-device (D2D) communication, terminals in vehicle-to-everything (V2X) communication, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, mixed reality (MR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical care, wireless terminals in smart grids, wireless terminals in transportation safety, and wireless terminals in smart cities. This application does not limit the scope of wireless terminals, such as those in a city, smart homes, or future communication networks.
[0053] In addition, in the embodiments of this application, there may be more or fewer terminal devices or network devices in the system. The number and type of terminal devices are determined according to actual needs, and are not limited here.
[0054] It should also be noted that the embodiment shown in Figure 1 is based on a single consensus cluster. In practical applications, the communication system can include more consensus clusters. The size of each consensus cluster, i.e., the number of network devices included, can be the same or different, and is not limited here. Each consensus cluster can apply the data processing method provided in the embodiments of this application, as detailed below.
[0055] Please refer to Figure 2 below, which is a flowchart illustrating the data processing method provided in the embodiments of this application.
[0056] 201. Detect the amount of data requested in the message queue, which is used to store requests from terminal devices.
[0057] Terminal devices, acting as producers in the message queue, send requests to the message queue, which then stores these requests. Network devices, acting as consumers, retrieve requests from the message queue. Therefore, the amount of requested data in the message queue is not constant, and it can reflect the system's load status. Furthermore, the requests stored in the message queue can also be understood as log data stored within it.
[0058] In this embodiment, a first network device detects the amount of requested data in the message queue to assess the system's load status. Specifically, the amount of requested data in the message queue refers to the amount of data currently stored in the message queue that has not yet been consumed by the first network device. These requests will be consumed by the first network device during subsequent communication, and consumed requests will be removed from the message queue. For example, the first network device can periodically or intermittently send detection instructions to the message queue, indicating the current amount of requested data in the message queue.
[0059] In practical applications, the relationship between the network device deploying the message queue and the first network device can be varied, enriching the implementation methods and application scenarios of the technical solution in this application. Optionally, the message queue can be deployed on the first network device. In this solution, the first device can more conveniently detect the message queue, and the latency is also lower. Optionally, the message queue can be deployed on other network devices besides the first network device. In this solution, the message queue and the first network device are independent of each other, and the amount of data stored in the message queue does not occupy the storage space of the first network device, which is beneficial to improving the performance of the first network device.
[0060] 202. Adjust the window size based on the requested data size, whereby the window size indicates the requested data size copied by the first network device each time.
[0061] The first network device detects the amount of requested data in the message queue to determine whether the data size of the current time window needs to be adjusted. The window refers to the amount of data requested for one replication operation set for the first network device. Specifically, the first and second network devices maintain replica consistency between themselves and among the second network devices based on a consensus protocol. This means that the first network device replicates the requests retrieved from the message queue and sends the replication requests to each second network device, ensuring that both the first and second network devices store the same requests. Therefore, the amount of requested data indicated by the window refers to the amount of data sent by the first network device to each second network device at a time.
[0062] The initial window size is set based on the needs of the actual application and is not limited here. For example, the granularity of the data size can be megabytes (MB), gigabytes (GB), etc.
[0063] The first network device adjusts the data volume of the window based on the amount of data requested in the message queue. This can also be understood as adjusting the size of the window, or adjusting its scale. In summary, the first network device determines the data volume adjustment strategy by comparing the requested data volume with a data volume threshold.
[0064] The data volume threshold represents the critical point at which data backlog occurs in the message queue. The specific value of the data volume threshold can be determined based on the needs of the actual application. Optionally, the data volume threshold is related to at least one of the following: the storage space of the network device deploying the message queue, or the current network state. For example, the larger the storage space of the network device deploying the message queue, the larger the data volume threshold can be. The better the current network state, the larger the data volume threshold can also be. The network state is determined based on parameters such as bandwidth, latency, and throughput of communication between the message queue and the first network device; specific details are not limited here.
[0065] In some alternative implementations, if the amount of requested data detected by the first network device exceeds a data volume threshold, it indicates a data backlog in the message queue. This means the current communication system is under high load, and the current system throughput cannot meet the current load. In this solution, the first network device increases the data volume in the window to improve system throughput.
[0066] The amount and method by which the first network device increases the data volume of the window each time can be determined based on the actual application needs, and are not specifically limited here. Optionally, the increase amount is related to at least one of the following: the storage space of the network device deploying the message queue, the current network status, or the current business. For example, the larger the storage space of the network device deploying the message queue, the smaller the increase amount can be. The better the current network status, the larger the increase amount can be. The more complex the current business, the larger the increase amount can be. The first network device can increase the data volume of the window linearly or non-linearly.
[0067] For example, in a scheme where the window data size increases linearly, the update of the window data size satisfies the formula: window_size = window_size × α. Here, α is an increment factor greater than 1, used to accelerate the increase in the window data size. The value of α is determined based on the actual application scenario and is not limited here.
[0068] It is understandable that linearly increasing the window size is beneficial to the stability of data processing, avoiding instability caused by sudden changes in the window size.
[0069] For example, the update of the window data size can also satisfy the formula: window_size = window_size + β. Here, β is an increment factor greater than 1, and the value of β is determined based on the actual application scenario; no specific limit is imposed here.
[0070] In some alternative implementations, if the amount of data requested detected by the first network device is less than a data volume threshold, it indicates that there is no data backlog in the message queue, the current communication system is under low load, and the current system throughput can easily cover the current load. In this approach, the first network device reduces the amount of data in the window, thus reducing the amount of data in a single copy request, thereby shortening the data copying time and improving latency.
[0071] The amount and method by which the first network device reduces the data volume of the window each time can be determined based on the actual application needs, and are not limited here. Optionally, the reduction amount is related to at least one of the following: the storage space of the network device deploying the message queue, the current network status, or the current business. For example, the larger the storage space of the network device deploying the message queue, the larger the reduction amount can be. The better the current network status, the smaller the reduction amount can be. The more complex the current business, the smaller the reduction amount can be. The first network device can also reduce the data volume of the window linearly or non-linearly.
[0072] For example, in a scheme where the window data size increases linearly, the update of the window data size satisfies the formula: window_size = window_size × A. Here, A is a reduction factor less than 1, used to accelerate the reduction speed of the window data size. The value of A is determined based on the actual application scenario and is not limited here.
[0073] Understandably, linearly reducing the amount of data in the window also helps to improve the stability of data processing and avoid instability caused by sudden changes in the amount of data in the window.
[0074] For example, the update of window data size can also satisfy the formula: window_size = window_size - B. Here, B is a reduction factor greater than 1, and the value of B is determined based on the actual application scenario; no limitation is made here.
[0075] Based on the foregoing explanation of step 202, in this embodiment, when the requested data volume exceeds a data volume threshold, the system is under high load due to data backlog in the message queue. The first network device increases the window's data volume to meet the high load demand and achieve high throughput. Conversely, when the requested data volume is less than the data volume threshold, the system is under low load. The first network device reduces the window's data volume, thus reducing the data volume requested for each copy and consequently reducing data processing latency. In summary, based on different adjustment methods, the first network device optimizes data processing performance under different load conditions.
[0076] In some alternative implementations, the amount of data requested detected by the first network device may also be equal to a data volume threshold. In this scheme, there are several possibilities for the first network device to adjust the data volume of the window.
[0077] Optionally, this approach can be understood as follows: the system throughput just meets the current system load, and the first network device keeps the amount of data in the current window unchanged.
[0078] Optionally, this approach can also be understood as follows: when the system throughput just reaches the current system load, in order to avoid the reduction in data processing performance due to subsequent load increases, the first network device increases the data size of the window.
[0079] 203. Based on the adjusted window data size, the requests retrieved from the message queue are copied, where the adjusted window data size is greater than or equal to the copied request data size from the message queue.
[0080] After the first network device adjusts the data size of the window, it replicates requests retrieved from the message queue based on the adjusted window data size. The replication strategy must satisfy the condition that the adjusted window data size is greater than or equal to the data size of the replication request. A specific example will illustrate this below.
[0081] For example, assume the adjusted window size is 50MB, and the first network device communicates with two second network devices based on a consensus protocol. If the first network device has already retrieved request 1 (20MB), request 2 (10MB), request 3 (5MB), request 4 (5MB), and request 5 (8MB) from the message queue, the sum of the data sizes of these five requests is 20 + 10 + 5 + 5 + 8 = 48MB. The difference between this and the adjusted window size is 50 - 48 = 2MB.
[0082] Based on the previous example, if the data size of the next request (i.e., request 6) retrieved by the first network device from the message queue is greater than 2MB, and this, combined with the data size of the first 5 requests, exceeds the data size of the adjusted window, then the first network device will copy only the first 5 requests in a single copy operation. Specifically, the first 5 requests will be copied twice, ensuring that each second network device receives one copy of the request. In this scheme, although the first network device retrieves 6 requests from the message queue, it only copies the first 5 requests, with request 6 awaiting the next copy. For this copy operation, the data size of the adjusted window is greater than the data size of the copied request retrieved from the message queue.
[0083] Based on the previous example, if the data size of the next request (i.e., request 6) retrieved by the first network device from the message queue is equal to 2MB, and this, combined with the data size of the first five requests, exactly equals the adjusted window size, then the first network device will copy all six requests in a single copy operation. Specifically, it will copy these six requests twice, ensuring that each second network device receives one copy of the requests. In this scheme, the first network device copies all six requests retrieved from the message queue, and the adjusted window size is greater than or equal to the data size of the copied requests retrieved from the message queue.
[0084] Based on the previous example, if the data size of the next request (i.e., request 6) retrieved by the first network device from the message queue is less than 2MB, even combined with the data size of the first five requests, it still does not reach the adjusted window size. The first network device can continue to retrieve the next request (i.e., request 7) until the total data size of the requests retrieved from the message queue reaches the adjusted window size. In this scheme, the adjusted window size may be greater than or equal to the copied data size of the requests retrieved from the message queue, similar to the previous example. In other words, the adjusted window size can be understood as the maximum data size of a request that the first network device can copy at one time.
[0085] In some alternative implementations, for a stable communication system, the data size of the window set for the first device can reflect the number of requests acquired. For example, a window data size of 100MB reflects that the first network device copies 10 requests at a time. In this scheme, the window data size indicates the number of requests copied from the message queue. A stable communication system includes stable network conditions between the first network device and the message queue, between the second network device, or stable service conditions corresponding to the requests acquired by the first network device, etc., which are not specifically limited here.
[0086] 204. Send a copy request to the second network device.
[0087] Based on the adjusted window data volume, the first network device copies the requests obtained from the message queue and sends the copy requests to each second network device with which it communicates based on the consensus protocol, thereby achieving replica consistency between the first network device and at least one second network device.
[0088] In summary, in this embodiment, the first network device is provided with a window, which defines the amount of data requested by the first network device in each copy. The amount of data in the window is adjusted based on the amount of data requested in the message queue, and is not constant. This allows the amount of data processed by the first network device in each batch to vary, better matching the actual communication state of the system and thus improving data processing performance.
[0089] In some alternative implementations, step 202 described above introduces methods for increasing the data volume of the window. In practical applications, to ensure that the system maintains optimal throughput performance during dynamic window size adjustment, the first network device can further adjust the data volume of the window based on feedback from actual throughput and data backlog. The possible solutions are described below:
[0090] In some alternative implementations, when the first network device detects that the window data size needs to be increased to cope with high load, it will gradually increase the window data size. Furthermore, the first network device continuously acquires data throughput data, specifically the data throughput between the first and second network devices, to detect changes in throughput performance. If a decrease in data throughput is detected during the increase of the window data size, the first network device will no longer increase the window data size but will instead maintain the window data size unchanged.
[0091] There are several possibilities for the first network device to obtain the data throughput between itself and the second network device. It can detect it itself, or the second network device or other devices can detect it and send it to the first network device. The specific method is not limited here. A decrease in data throughput means that the currently obtained data throughput is less than the previously obtained data throughput.
[0092] In this embodiment, the first network device may not continuously increase the data volume of the window. In addition to detecting the amount of data requested in the message queue, it also obtains the data throughput between the first and second network devices to comprehensively evaluate whether it is necessary to continue increasing the data volume of the window. In the scheme where the data throughput begins to decline, keeping the data volume of the window constant, and comprehensively evaluating the balance between system load and device throughput, is beneficial to maintaining the stability of data processing.
[0093] In some alternative implementations, in a scheme where the first network device reduces the amount of data in the window to reduce processing latency, if data backlog is detected in the message queue during the reduction of the amount of data in the window, the first network device can stop further reducing the amount of data in the window and restore the amount of data in the window to its previous size.
[0094] In the foregoing embodiments, the data processing method provided by the embodiments of this application is described with the first network device as the execution subject. The following description further elaborates on the method from the perspective of a communication system. Please refer to Figure 3, which is a flowchart illustrating the data processing method provided by the embodiments of this application.
[0095] In the embodiment shown in Figure 3, a network device with the message queue deployed outside the first network device is taken as an example. As shown in Figure 3, the terminal sends a request to the message queue to use the message storage service provided by the message queue. The message queue receives the terminal's request and maintains a request queue, including arranging requests sequentially according to the order in which they were received. The process of interaction between the terminal and the message queue can be called the request receiving phase.
[0096] In this embodiment, the first network device and the second network device need to complete a consensus phase between nodes. The first network device sends a consensus request to the second network device, and the second network device receives and responds to the consensus request, thereby establishing a consensus between the first and second network devices. The first network device retrieves requests from the message queue through the consumption interface provided by the message queue. After packaging the retrieved requests into batch requests, it performs replication and sends the replicated batch requests to the second network device.
[0097] During the consensus phase between nodes, the first network device acquires system load and throughput performance to determine the data volume of the current window, which is also the data volume of each batch replication request. In other words, system load detection, throughput performance detection, and window parameter adjustment can also be performed during the consensus process between nodes.
[0098] System load detection can be calculated and determined by the message queue based on received client requests, or it can be performed by the first network device. Throughput performance is determined by the first network device, the second network device, or other devices based on consensus, specifically based on the throughput between the first and second network devices. The specific implementation process for window parameter adjustment is as described above and will not be repeated here.
[0099] After determining the data size of the window, the first network device consumes requests from the message queue based on that data size and packages the consumed requests into batch requests. A batch request is essentially the request that the first network device will copy all at once. The data size of the batch request is less than or equal to the data size of the window. After copying is complete, the first network device sends the copied batch request to each second network device. Each second network device receives the batch request and participates in the consensus process.
[0100] It should be noted that the foregoing embodiments all use a single consensus cluster as an example. In practical applications, the communication system may include multiple consensus clusters. In this scenario, each consensus cluster can apply the data processing method provided in the embodiments of this application, and the implementation principle is similar to that described above, so it will not be repeated here. However, the specific implementation details may differ slightly: if different consensus clusters need to communicate, the communication messages can also be stored in the message queue used by the receiver. That is, the message queue is not only used to store requests from terminal devices, but also to store messages from the sender. For example, a third network device sends a message to a first network device, and this message is stored in the message queue used by the first network device. The third network device is a network device outside the consensus cluster where the first network device is located.
[0101] Please refer to Figure 4 below, which is a schematic diagram of the structure of a network device provided in an embodiment of this application. As shown in Figure 4, the network device 400 includes a processing unit 401 and a sending unit 402. Optionally, it also includes an acquisition unit 403.
[0102] In some optional implementations, processing unit 401 is configured to: detect the amount of requested data in a message queue, the message queue being used to store requests from terminal devices; adjust the data size of a window based on the amount of requested data, the data size of the window indicating the amount of requested data copied by the network device each time; and copy requests retrieved from the message queue based on the adjusted data size of the window, wherein the adjusted data size of the window is greater than or equal to the amount of requested data copied from the message queue.
[0103] The sending unit 402 is used to send a copy request to the second network device.
[0104] In some optional implementations, processing unit 401 is specifically configured to: increase the data volume of the window if the requested data volume is greater than a data volume threshold; and decrease the data volume of the window if the requested data volume is less than the data volume threshold.
[0105] In some optional implementations, network device 400 further includes an acquisition unit 403 for acquiring the data throughput between the network device and the second network device.
[0106] The processing unit 401 is also configured to keep the amount of data in the window constant if the data throughput decreases.
[0107] In some alternative implementations, processing unit 401 is specifically configured to linearly increase the amount of data in the window.
[0108] In some alternative implementations, processing unit 401 is specifically used to linearly reduce the amount of data in the window.
[0109] The processing unit 401, sending unit 402, and acquiring unit 403 can all be implemented in software or in hardware. For example, the implementation of processing unit 401 will be described below. Similarly, the implementation of sending unit 402 and acquiring unit 403 can refer to the implementation of processing unit 401.
[0110] As an example of a software functional unit, processing unit 401 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, processing unit 401 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.
[0111] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.
[0112] As an example of a hardware functional unit, processing unit 401 may include at least one computing device, such as a server. Alternatively, processing unit 401 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.
[0113] The processing unit 401 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the processing unit 401 includes multiple computing devices that can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the processing unit 401 includes multiple computing devices that can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0114] It should be noted that the network device 400 shown in Figure 4 is regarded as the first network device. The processing unit 401, the sending unit 402, and the acquisition unit 403 respectively implement different steps in the data processing method to realize all the functions of the network device 400, so as to realize the data processing method provided in the embodiment of this application. It will not be described again here.
[0115] Please refer to Figure 5 below, which is another structural schematic diagram of the network device provided in the embodiment of this application.
[0116] As shown in Figure 5, the network device 500 includes a processor 501, a memory 502, a communication interface 503, and a bus 504. The processor 501, memory 502, and communication interface 503 communicate via the bus 504, or they can communicate through other means such as wireless transmission.
[0117] The memory 502 stores program code, and the processor 501 can call the program code stored in the memory 502 to execute the operations performed by the first network device in the foregoing embodiment, so as to implement the data processing method provided in the embodiments of this application, which will not be described in detail here.
[0118] It should be understood that in the embodiments of this application, the processor 501 may be a CPU, or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.
[0119] The memory 502 may include read-only memory and random access memory, and provides instructions and data to the processor 501. The memory 502 may also include non-volatile random access memory. For example, the memory 502 may also store device type information.
[0120] The memory 502 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).
[0121] In addition to the data bus, bus 504 can also include a power bus, control bus, and status signal bus. However, for clarity, all buses are labeled as bus 504 in the diagram. Bus 504 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. Bus 504 can be divided into address bus, data bus, control bus, etc.
[0122] Network device 500 may also include one or more communication interfaces and one or more operating systems, such as Windows Server. TM Mac OS X TMUnix TM Linux TM FreeBSD TM wait.
[0123] In some optional implementations, embodiments of this application also provide a chip, including: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits.
[0124] The processing unit can execute computer execution instructions stored in the storage unit to cause the chip to perform the data processing method described in the foregoing embodiments. Optionally, the storage unit is an internal storage unit within the chip, such as a register or cache. Optionally, the storage unit can also be an external storage unit located within the wireless access device, such as a read-only memory or other types of static storage devices capable of storing static information and instructions, such as random access memory.
[0125] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computer device, it causes the at least one computer device to perform the above-described data processing method.
[0126] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the aforementioned data processing method.
[0127] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0128] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.
[0129] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0131] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
Claims
1. A data processing method, characterized in that, The method is applied to a first network device, and the method includes: The amount of data requested in the message queue is detected, and the message queue is used to store requests from terminal devices. Based on the requested data volume, the data volume of the window is adjusted, wherein the data volume of the window indicates the requested data volume copied by the first network device each time; Based on the adjusted window data volume, requests obtained from the message queue are copied, wherein the adjusted window data volume is greater than or equal to the copied request data volume obtained from the message queue. Send a copy request to the second network device.
2. The method according to claim 1, characterized in that, Adjusting the window's data volume based on the requested data volume includes: If the requested data volume exceeds the data volume threshold, then increase the data volume of the window. If the requested data volume is less than the data volume threshold, then the data volume of the window is reduced.
3. The method according to claim 2, characterized in that, After increasing the amount of data in the window, the method further includes: Obtain the data throughput between the first network device and the second network device; If the data throughput decreases, the amount of data in the window remains unchanged.
4. The method according to claim 2 or 3, characterized in that, Increasing the amount of data in the window includes: linearly increasing the amount of data in the window.
5. The method according to any one of claims 2 to 4, characterized in that, The reduction of the data volume of the window includes: linearly reducing the data volume of the window.
6. A network device, characterized in that, include: A processing unit is used to detect the amount of data requested in a message queue, the message queue being used to store requests from terminal devices; The processing unit is further configured to adjust the data volume of the window based on the data volume of the request, wherein the data volume of the window indicates the data volume of the request copied by the network device each time; The processing unit is further configured to copy requests obtained from the message queue based on the adjusted window data volume, wherein the adjusted window data volume is greater than or equal to the copied data volume of requests obtained from the message queue. The sending unit is used to send a copy request to the second network device.
7. The network device according to claim 6, characterized in that, The processing unit is specifically used for: If the requested data volume exceeds the data volume threshold, then increase the data volume of the window. If the requested data volume is less than the data volume threshold, then the data volume of the window is reduced.
8. The network device according to claim 7, characterized in that, The network device also includes: The acquisition unit is used to acquire the data throughput between the network device and the second network device; The processing unit is further configured to keep the amount of data in the window unchanged if the data throughput decreases.
9. The network device according to claim 7 or 8, characterized in that, The processing unit is specifically used to linearly increase the amount of data in the window.
10. The network device according to any one of claims 7 to 9, characterized in that, The processing unit is specifically used to linearly reduce the amount of data in the window.
11. A network device, characterized in that, Includes a processor, which is coupled to a memory; The memory stores instructions that, when executed on the processor, cause the communication device to perform the method of any one of claims 1 to 5.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a processor, cause the method of any one of claims 1 to 5 to be implemented.
13. A computer program product, characterized in that, When the computer program product is executed on a computer, the method of any one of claims 1 to 5 is implemented.