Data synchronization method and apparatus, and device, storage medium and program product

Abstract

Disclosed in the present application are a data synchronization method and apparatus, and a device, a storage medium and a program product. The method comprises: acquiring target data to be sent to a data receiving end; detecting a network transmission state between an electronic device and the data receiving end, and on the basis of the network transmission state, determining whether a data synchronization mode corresponding to the data receiving end is a real-time synchronization mode or a snapshot synchronization mode; when it is determined that the data synchronization mode is the real-time synchronization mode, immediately sending said target data to the data receiving end; and when it is determined that the data synchronization mode is the snapshot synchronization mode, querying pre-stored service processing logic of the data receiving end, processing said target data on the basis of the service processing logic, and sending snapshot data of a processing result to the data receiving end.

Description

Data synchronization methods, devices, equipment, storage media and software products

[0001] This application claims priority to Chinese Patent Application No. 2024116219474, filed on November 14, 2024, entitled “Data Synchronization Method, Apparatus, Device and Storage Medium”. Technical Field

[0002] This application relates to the field of data processing technology, and in particular to data synchronization methods, apparatus, devices, storage media, and program products.

[0003] Background of the Invention

[0004] Currently, with the rapid development of information technology, related applications have gradually integrated into people's lives, providing a wide variety of services. These applications may involve various types of data transmission and processing. In some scenarios, different data receiving ends need to maintain data consistency. If the received data is inconsistent, it may affect the performance of the application. At this time, data synchronization technology is needed to synchronously distribute the same data to different data receiving ends.

[0005] In related technologies, real-time synchronization is generally used to improve the efficiency of data transmission and processing during data synchronization. For example, data is placed in a message queue, and different data receiving ends can read and process data from the message queue. Although all data receiving ends read data from the same message queue, ensuring data consistency, some data receiving ends may fail to obtain data in a timely manner when network transmission conditions are poor, resulting in low utilization of network transmission resources and affecting data processing efficiency. Summary of the Invention

[0006] This application provides a data synchronization method, apparatus, device, storage medium, and program product, which can improve the utilization rate of network transmission resources and improve the efficiency of data transmission and processing.

[0007] On one hand, embodiments of this application provide a data synchronization method applied to electronic devices, including:

[0008] Obtain the target data to be sent to the data receiver;

[0009] The network transmission status between the electronic device and the data receiving end is detected, and based on the network transmission status, it is determined whether the data synchronization mode corresponding to the data receiving end is real-time synchronization mode or snapshot synchronization mode.

[0010] When the data synchronization mode is determined to be the real-time synchronization mode, the target data is immediately sent to the data receiving end; and,

[0011] When the data synchronization mode is determined to be the snapshot synchronization mode, the pre-stored business processing logic of the data receiving end is queried, the target data is processed according to the business processing logic, and the snapshot data of the processing result is sent to the data receiving end.

[0012] On the other hand, embodiments of this application provide a data synchronization device, including:

[0013] The acquisition unit is used to acquire the target data to be sent to the data receiving end.

[0014] The detection unit is used to detect the network transmission status between the electronic device and the data receiving end, and determine whether the data synchronization mode corresponding to the data receiving end is real-time synchronization mode or snapshot synchronization mode based on the network transmission status.

[0015] The first processing unit is configured to immediately send the target data to the data receiving end when the data synchronization mode is determined to be the real-time synchronization mode; and,

[0016] The second processing unit is used to query the pre-stored business processing logic of the data receiving end when the data synchronization mode is determined to be the snapshot synchronization mode, process the target data according to the business processing logic, and send the snapshot data of the processing result to the data receiving end.

[0017] On the other hand, embodiments of this application provide an electronic device, including a processor and a memory;

[0018] The memory is used to store computer programs;

[0019] The processor executes the computer program to implement the aforementioned data synchronization method.

[0020] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program that is executed by a processor to implement the aforementioned data synchronization method.

[0021] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the aforementioned data synchronization method.

[0022] Brief description of the attached figures

[0023] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0024] Figure 1 is a schematic diagram of a data synchronization method used in related technologies;

[0025] Figure 2 is a system architecture diagram of a data synchronization method provided in the embodiments of this application;

[0026] Figure 3 is a schematic diagram of the system architecture of a streaming media service provided in an embodiment of this application;

[0027] Figure 4 is a schematic diagram illustrating the relationship between a data sending end and a data receiving end provided in an embodiment of this application;

[0028] Figure 5 is a flowchart illustrating a data synchronization method provided in an embodiment of this application;

[0029] Figure 6 is a schematic diagram of converting target data into log data according to an embodiment of this application;

[0030] Figure 7 is a schematic diagram of extracting data to be synchronized according to an embodiment of this application;

[0031] Figure 8 is a schematic diagram illustrating the application principle of an RSS system provided in an embodiment of this application;

[0032] Figure 9 is a schematic diagram of various data structures for transmission provided in an embodiment of this application;

[0033] Figure 10 is a schematic diagram illustrating the principle of data synchronization provided in an embodiment of this application;

[0034] Figure 11 is a schematic diagram illustrating the principle of a distributed source data system crash rollback provided in an embodiment of this application;

[0035] Figure 12 is a structural block diagram of a data synchronization device provided in an embodiment of this application;

[0036] Figure 13 is a structural block diagram of an electronic device provided in an embodiment of this application.

[0037] Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0039] It is understood that the terms “first,” “second,” etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another.

[0040] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0041] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows:

[0042] 1) ETL (Extract, Transform, Load) tools are a key technology for data integration and data warehouse construction. The application process of ETL tools involves three main steps: extraction, transformation, and loading. ETL tools can help extract data from multiple data sources, perform necessary transformations and cleaning, and then load the data into the target data warehouse or data lake.

[0043] 2) Kafka, a distributed stream processing platform, is mainly used to build real-time data pipelines and streaming applications. Its design aims to process large amounts of data with high throughput and low latency.

[0044] 3) Pulsar, a distributed message queuing system used to handle large-scale message passing and stream processing tasks.

[0045] 4) GDEF (General Data Exchange Format) aims to provide a standardized way to represent and exchange various types of data. This format is commonly used for data transfer between different systems to ensure data consistency and readability.

[0046] 5) Raft, a distributed consensus algorithm for managing replicated logs. It aims to solve the complexity of understanding and implementation of earlier consensus algorithms, with the goal of providing greater understandability and better practical applicability.

[0047] Currently, with the rapid development of information technology, related applications have gradually integrated into people's lives, providing a wide variety of services. These applications may involve various types of data transmission and processing. In some scenarios, different data receiving ends need to maintain data consistency; if the received data is inconsistent, it may affect the performance of the application.

[0048] For example, in financial transactions such as bank transfers and stock trading, data consistency must be maintained between various terminal devices involved in the transaction (e.g., the terminal devices used by traders and the terminal devices of the banking system). Otherwise, financial losses or abnormal transaction records may occur. Similarly, in medical information systems, data such as medical records, diagnoses, and treatment plans need to be consistent across the hospital's information system, the terminal devices used by doctors, and patients. Otherwise, misdiagnosis or failure to promptly understand the patient's condition may occur, impacting the medical experience. Furthermore, in social networking services, data such as account information, friend relationships, and interactive messages need to maintain consistency across the terminal devices of different users. Otherwise, asymmetrical or incorrect information may be disseminated, affecting the user experience.

[0049] Therefore, in related technologies, data synchronization techniques are often used for scenarios requiring data consistency. For example, some solutions use ETL tools for data synchronization. ETL tools can extract data from a data source, perform necessary transformations and cleaning, and then load the data into a target data warehouse or data lake. For the scenario in this application where data consistency needs to be maintained across multiple data receivers, the data to be synchronized can be directly obtained from the data source, transformed and cleaned, and then loaded onto multiple data receivers. However, this approach synchronizes data in batches, sending it to the data receivers only when a certain amount of transformed and cleaned data is available. This results in poor real-time performance, potentially affecting the efficiency of data processing at the data receivers, and also hinders the efficient utilization of network transmission resources.

[0050] To improve the efficiency of data transmission and processing, some solutions employ real-time synchronization strategies. For example, data can be placed in a message queue, such as a Kafka queue or a Pulsar queue. For instance, please refer to Figure 1, which illustrates a data synchronization method used in related technologies. As shown in Figure 1, in related technologies, data can be placed in a message queue during synchronization, allowing different data receivers to directly read and process data from the message queue. Since all data receivers read data from the same message queue, data consistency is guaranteed. However, in practical applications, it has been found that some data receivers may experience poor network transmission conditions, preventing them from acquiring and processing data in a timely manner, resulting in low utilization of network transmission resources and impacting data processing efficiency.

[0051] In view of this, embodiments of this application provide a data synchronization method, apparatus, device, storage medium, and program product. The method of this application can be applied to a data sending end, which is used for synchronous data transmission to multiple data receiving ends and pre-stores the business processing logic of each data receiving end. The method acquires target data, and for each data receiving end, detects the network transmission status between the data sending end and the data receiving end to determine whether the data receiving end uses a real-time synchronization mode or a snapshot synchronization mode. For data receiving ends using real-time synchronization mode, the target data is sent immediately, improving the efficiency of target data transmission to the data receiving end; for data receiving ends using snapshot synchronization mode, the target data is pre-processed using the pre-stored business processing logic corresponding to the data receiving end, and then a snapshot of the processing result is sent to the data receiving end, improving the processing efficiency of the target data. This application can adjust the data synchronization mode based on the network transmission status corresponding to the data receiving end, synchronizing data to heterogeneous systems in an orderly, real-time, and reliable manner, and providing two synchronization modes to adapt to various scenarios, improving the utilization rate of network transmission resources and improving the efficiency of data transmission and processing.

[0052] System architecture and scenario description used in the embodiments of this application

[0053] Figure 2 is a system architecture diagram of a data synchronization method provided in an embodiment of this application, which includes a terminal device 240, an Internet 230, a gateway 220, a backend server 210, etc.

[0054] In this embodiment, the terminal device 240 can install and run related applications, such as video playback applications and shopping applications. The terminal device 240 can include various forms such as desktop computers, laptops, PDAs (personal digital assistants), mobile phones, in-vehicle terminals, home theater terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. The terminal device 240 can communicate with the Internet 230 via wired or wireless means to exchange data.

[0055] A backend server 210 refers to a computer system that can provide certain services to terminal devices 240. Compared to ordinary terminal devices 240, backend servers 210 have higher requirements in terms of stability, security, and performance. A backend server 210 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).

[0056] Gateway 220, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal device 240 to backend server 210 are forwarded to the corresponding backend server 210 via gateway 220. Messages sent from backend server 210 to terminal device 240 are also forwarded to the corresponding terminal device 240 via gateway 220.

[0057] The data synchronization method provided in this application embodiment can be executed locally on the terminal device 240 or the backend server 210, or it can be executed interactively between the terminal device 240 and the backend server 210. The backend server 210 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0058] The data synchronization method provided in this application can be executed in various scenarios, and the following is an exemplary description of it.

[0059] (I) Shopping Scenarios

[0060] The method provided in this application embodiment can be applied in shopping applications (APPs).

[0061] Shopping apps need to ensure that product inventory data is consistent across multiple servers to avoid overselling; or ensure that account information (such as address and payment information) is consistent across multiple servers to prevent order anomalies.

[0062] Using this shopping app on different devices can better ensure smooth transactions and improve the shopping experience.

[0063] (II) Streaming Media Scenarios

[0064] The method provided in this application embodiment can be applied to streaming media apps.

[0065] Streaming media is a technology that continuously transmits audio and video content over a network, allowing end devices to begin playing media files before downloading all the data. Streaming scenarios typically involve real-time or near-real-time content delivery, which can include online video, live audio broadcasts, IPTV, distance learning courses, live game streaming, video conferencing, and more.

[0066] Figure 3 illustrates a system architecture diagram of a streaming media service provided in an embodiment of this application. As shown in Figure 3, the streaming media content provider 301 can use a content delivery network (CDN) 302 to distribute streaming media content, that is, cache relevant business data on various nodes of the CDN 302. When a user wants to obtain relevant streaming media resources, the nearest CDN node will respond to the user's request and feed back the streaming media resources to the user's terminal device 3031 or 3032 for playback and display.

[0067] To ensure that streaming media content in the content delivery network is updated synchronously in real time, the data synchronization method in this application embodiment can be used to improve the efficiency of data transmission and processing as much as possible and ensure the browsing experience of streaming media services.

[0068] The data synchronization method in this application embodiment can be applied to a data sending end, which can perform synchronized data transmission to multiple data receiving ends. Here, multiple data receiving ends can refer to any number of data receiving ends, either two or more, and this application does not impose any limitation on this.

[0069] In this embodiment, after receiving the target data to be synchronized, each data receiving end can perform corresponding business processing based on the target data. For example, in an e-commerce scenario, there are multiple sellers on the e-commerce platform. Each seller can upload their product information to the platform system, which can then display the information for each product. When a shopper logs into the e-commerce platform to shop, the platform system generates transaction data after placing an order. This transaction data is synchronized to the terminal devices used by both the shopper and the seller. At this time, the platform system can act as the data sending end, and the terminal devices used by the shopper and the seller can act as the data receiving end. When the transaction data is synchronized to the shopper's terminal device, further processing yields order data. Similarly, when the transaction data is synchronized to the seller's terminal device, further processing yields revenue data, etc. Therefore, each data receiving end generally has its own business processing logic for the target data, and the business processing logic used by different data receiving ends is often different.

[0070] In this embodiment, the business processing logic for the target data in the data receiving end can be integrated into the data sending end. This means the business processing logic of the data receiving end is moved forward, allowing the data sending end to preprocess the target data according to the specific business processing logic of the data receiving end. When network transmission is poor, the data sending end can preprocess the target data or a portion of the target data, thereby improving data processing efficiency.

[0071] For example, please refer to Figure 4, which shows a schematic diagram of the relationship between a data transmitter and a data receiver provided in an embodiment of this application. In Figure 4, for the data transmitter 402, there are three corresponding data receivers, denoted as the first data receiver 4031, the second data receiver 4032, and the third data receiver 4033, respectively. The first data receiver 4031, the second data receiver 4032, and the third data receiver 4033 can be any type of device, and their device types can be the same or different.

[0072] Specifically, the underlying systems of the first data receiver 4031, the second data receiver 4032, and the third data receiver 4033 can be the same or different. When the underlying systems of the first data receiver 4031, the second data receiver 4032, and the third data receiver 4033 are different, the three belong to heterogeneous systems.

[0073] At the data sending end 402, the service processing logic of the first data receiving end, the service processing logic of the second data receiving end, and the service processing logic of the third data receiving end are pre-stored. In this embodiment of the application, the pre-stored service processing logic of the first data receiving end is denoted as the first service processing logic, the pre-stored service processing logic of the second data receiving end is denoted as the second service processing logic, and the pre-stored service processing logic of the third data receiving end is denoted as the third service processing logic.

[0074] It should be noted that, in this embodiment, the business processing logic of the data receiving end pre-stored at the data sending end may only be a portion of the data receiving end's business processing logic. For example, after receiving the target data, the data receiving end will perform a multi-step business processing flow. In this embodiment, the business processing logic of the first few steps can be encapsulated and pre-stored at the data sending end. In this way, it can support the data sending end to handle the tasks on behalf of the data receiving end without making the data sending end bloated with too much pre-stored program code.

[0075] As described above, in this embodiment, the number of data receivers can be any number, including two or more, and is not fixed as shown in Figure 4. Furthermore, in this embodiment, the number of data receivers corresponding to a data sender can be flexible and variable. For example, in some scenarios, one (or more) data receivers may no longer need to receive and process target data. In this case, these data receivers can be removed from the current data synchronization task, reducing the number of data receivers corresponding to the data sender. In other scenarios, one (or more) terminals may not originally be data receivers, but currently also need to synchronously receive and process target data. In this case, these terminals can be added to the current data synchronization task as data receivers corresponding to the data sender, increasing the number of data receivers. This application does not impose any restrictions on this.

[0076] It should be noted that, in the embodiments of this application, the data sending end can be a single device or a distributed system, including multiple devices, each set up in a dispersed manner. The same applies to the data receiving end, and this application does not impose any restrictions on this.

[0077] Please refer to Figure 5, which shows a flowchart of a data synchronization method provided in an embodiment of this application. This method is applied to an electronic device, which serves as the aforementioned data sending end. As shown in Figure 5, the data synchronization method according to an embodiment of this application includes, but is not limited to, the following steps:

[0078] Step 510: Obtain the target data to be sent to the data receiving end;

[0079] Step 520: Detect the network transmission status between the electronic device and the data receiving end, and determine whether the data synchronization mode corresponding to the data receiving end is real-time synchronization mode or snapshot synchronization mode based on the network transmission status.

[0080] Step 530: When the data synchronization mode is determined to be real-time synchronization mode, immediately send the target data to the data receiving end;

[0081] Step 540: When the data synchronization mode is determined to be snapshot synchronization mode, query the pre-stored business processing logic of the data receiving end, process the target data according to the business processing logic, and send the snapshot data of the processing result to the data receiving end.

[0082] In this application embodiment, a data synchronization method is provided. This method can adjust the data synchronization mode according to the network transmission status of the data receiving end, so as to maximize the utilization of network transmission resources and improve the efficiency of data transmission and processing.

[0083] In step 510, the target data refers to the data that needs to be synchronized at each data receiving end. In this embodiment of the application, the data type of the target data is not limited and can be flexibly set according to the specific application.

[0084] For example, in some embodiments, the target data can be structured data, such as numerical data, including integers, floating-point numbers, etc., which can be used to represent information such as age, price, or quantity. Another example is text data, that is, a string of characters, such as name, address, or description.

[0085] In some embodiments, the target data may be unstructured data, such as image data, like photos, video frames, etc., or audio data, such as voice data, music data, etc.

[0086] In some embodiments, the target data may also be semi-structured data or multimodal data, which will not be elaborated here.

[0087] In this embodiment of the application, the specific content of the target data can be set according to the actual application scenario of the data synchronization method.

[0088] For example, in some embodiments, when the data synchronization method of this application is applied in an e-commerce scenario, the target data may include shopper behavior data (such as browsing history, purchase history and search information, etc.), product data (such as inventory quantity, supplier information and price data, etc.) or order data (such as order status, delivery information, payment records, etc.).

[0089] In some embodiments, when the data synchronization method in this application is applied to a scenario in the financial industry, the target data may include transaction data (such as transaction records of financial products such as stocks, bonds, and foreign exchange), account data (such as identity information, credit scores, account balances, etc.), and market data (such as market indices, interest rate changes, economic indicators, etc.).

[0090] In this embodiment of the application, the above description is only for illustrative purposes and does not imply any limitation on the actual application of the application.

[0091] In step 510, when acquiring target data, the specific acquisition method can be selected according to actual needs and the location of the target data.

[0092] In some embodiments, target data can be obtained through database queries, such as by retrieving target data from a database using a specific query statement (e.g., an SQL statement) or an application programming interface.

[0093] In some embodiments, the target data can be obtained by reading relevant files, such as from a log file.

[0094] In some embodiments, target data can also be obtained from websites by using web crawlers or third-party data services, and this application does not limit this.

[0095] In this embodiment, the action of acquiring target data can be performed continuously or intermittently. For example, a real-time bidirectional communication protocol can be used to continuously acquire target data from the data source, or a query can be sent periodically to acquire target data from the data source. To maximize the efficiency of data transmission synchronization, a continuous execution and real-time acquisition method can be used to obtain the target data.

[0096] It should be noted that in the embodiments of this application, the target data will be synchronously transmitted to multiple data receiving ends. The underlying systems used by these data receiving ends may be heterogeneous systems, and the data formats supported by heterogeneous systems may differ. Therefore, in some embodiments, when multiple data receiving ends use heterogeneous systems, the target data is preprocessed to obtain general data that can be used by various heterogeneous systems.

[0097] In step 520, there are several ways to detect the network transmission status between the data sender and the data receiver.

[0098] For example, in some embodiments, relevant commands can be used to query certain metrics of network transmission between the data sender and the data receiver. For instance, the Ping command can be used to view the response time and packet loss rate at the data receiver. Specifically, lower response time and packet loss rate indicate better network transmission; conversely, higher response time and packet loss rate indicate worse network transmission.

[0099] In some embodiments, network monitoring tools or network packet capture tools can be used to capture data packets transmitted between the data sender and the data receiver, and then analyze multiple indicator data, such as network bandwidth, traffic, throughput over a predetermined time period, latency, etc., and determine the network transmission status between the data sender and the data receiver based on the specific values ​​of these indicators.

[0100] In the embodiments of this application, the specific methods and related indicator data for judging network transmission status are not limited to the above-described situations. Those skilled in the art can flexibly select the judgment strategy according to actual needs, and this application does not impose any restrictions on this.

[0101] In this embodiment of the application, when detecting network transmission status, one or more detected indicator data can be used to divide the detection results into multiple levels. For example, based on the detected indicator data, n thresholds can be set, resulting in n+1 levels. When there is only one threshold, it can be divided into two levels: "good" and "poor".

[0102] For example, the response time of the data receiving end can be detected, and a threshold can be set. If the response time exceeds the set threshold, the detection result of the network transmission status of the data receiving end can be determined as "poor"; if the response time does not exceed the set threshold, the detection result of the network transmission status of the data receiving end can be determined as "good".

[0103] In some embodiments, multiple indicators can be used for judgment, and the detection result of each indicator can be divided into two levels: "good" and "poor". Similar to the judgment logic for "response time" described above, by setting corresponding thresholds, the detection result of each indicator is determined to be "good" or "poor". If a certain number of indicators (e.g., more than half) are determined to be "good", then the final network transmission status detection result is determined to be "good"; conversely, if only a portion of the indicators (e.g., less than half) are determined to be "good", then the final network transmission status detection result is determined to be "poor". This application does not impose any limitation on the number of indicators.

[0104] In step 520, based on the determined network transmission status, the data synchronization mode corresponding to the data receiving end can be further determined. In this embodiment of the application, the data synchronization modes that all data receiving ends can use include real-time synchronization mode and snapshot synchronization mode.

[0105] Real-time synchronization mode refers to transmitting the target data to the data receiver immediately after the data sending end receives the target data. This mode can quickly provide feedback to the data receiver when the target data arrives at the data sending end, ensuring the immediacy of data transmission.

[0106] Snapshot synchronization mode refers to a mode where, after the data sending end obtains the target data, it does not directly transmit the target data to the data receiving end. Instead, it transmits snapshot data of the target data at pre-set intervals. In this mode, the target data needs to wait for a period of time after arriving at the data sending end before it can be synchronized to the data receiving end.

[0107] In this embodiment, the target data is transmitted as soon as it arrives in the real-time synchronization mode, which typically requires significant network transmission resources. That is, when the network transmission between the data sender and receiver is good, the target data can be quickly sent to the receiver using the real-time synchronization mode. However, if the network transmission between the data sender and receiver is poor, even using the real-time synchronization mode, the receiver will struggle to receive the target data quickly due to network limitations after the sender transmits it. This results in a waste of network transmission resources for the sender.

[0108] Therefore, in this embodiment, when determining the data synchronization mode corresponding to the data receiving end, if the network transmission status between the data sending end and the data receiving end is good, for example, if the previous network transmission status detection result is determined to be "good", then the corresponding data synchronization mode can be determined to be real-time synchronization mode. Conversely, if the network transmission status between the data sending end and the data receiving end is poor, for example, if the previous network transmission status detection result is determined to be "poor", even if real-time synchronization mode is used, it may not be able to achieve real-time synchronization, and instead consume the network transmission resources of the data sending end, resulting in low utilization of network transmission resources. In this case, the data synchronization mode corresponding to the data receiving end is determined to be snapshot synchronization mode. Compared to real-time synchronization mode, snapshot synchronization mode has lower requirements for network transmission resources, higher utilization, and can achieve data synchronization by transmitting processed snapshot data, making it more suitable for data receiving ends with poor network transmission status.

[0109] In step 530, if it is determined that the data synchronization mode corresponding to the data receiving end is real-time synchronization mode, the target data can be sent to the data receiving end immediately.

[0110] Specifically, in this embodiment, the data transmission protocol used by the data receiver can be determined in advance based on the device type of the data receiver. Then, when the data synchronization mode of the data receiver is real-time synchronization mode, the target data is immediately sent to the data receiver using the corresponding data transmission protocol.

[0111] In step 540, if the data synchronization mode corresponding to the data receiving end is determined to be snapshot synchronization mode, then the target data is pre-processed at the data sending end, the processed result is temporarily stored, and then sent to the data receiving end as a snapshot. In this way, the data receiving end can directly use the processed result without repeated processing. Pre-processing refers to performing corresponding calculations on the target data according to the business processing logic of the data receiving end. The business processing logic can specifically be an execution script, including various calculation steps performed on the target data according to business requirements.

[0112] Specifically, the pre-stored business processing logic of the data receiving end can be queried. The target data is processed through the business processing logic, and the processing results are cached. When the set sending conditions are met, the cached processing results are retrieved, and according to the snapshot data structure, they are packaged into snapshot data of a specific format and sent to the data receiving end.

[0113] In this embodiment, the data sending end integrates the business processing logic for each data receiving end, with a one-to-one correspondence between the business processing logic and the data receiving end. In this embodiment, a unique identifier, denoted as a system identifier, can be pre-assigned to each data receiving end. Each data receiving end has a different system identifier, which can include at least one of numbers or characters. When pre-storing the business processing logic, the business processing logic corresponding to each data receiving end can be bound to the system identifier. Thus, during data synchronization, if the data synchronization mode for a certain data receiving end is snapshot synchronization mode, the corresponding business processing logic can be queried and determined based on the system identifier of that data receiving end. Then, this business processing logic is used to process the target data.

[0114] In another embodiment, preprocessing includes parsing the data format according to the business processing logic of the data receiving end, and performing corresponding calculations on the parsed data. The parsing process refers to restoring the aforementioned GDEF dataset into structured data defined by the data receiving end in sequence. In this case, the execution script includes a parsing script and a calculation script. The processing result refers to the result after performing calculations on the structured data defined by the data receiving end.

[0115] In step 540, a snapshot of the processing result is sent to the data receiving end, which can be sent periodically. The specific period refers to the time frame, which can be set according to actual needs.

[0116] For example, in some embodiments, a fixed transmission period can be set, such as sending once every 5 seconds. In this embodiment, the length of the transmission period is not limited.

[0117] In some embodiments, the transmission period can also be correlated with the network transmission status corresponding to the data receiver. For example, when the network transmission status corresponding to the data receiver is poor, its corresponding data synchronization mode is snapshot synchronization mode, and the corresponding transmission period can be further determined based on the network transmission status-related indicator data. Specifically, a functional relationship between the network transmission status-related indicator data and the transmission period is preset. When the data synchronization mode is determined to be snapshot synchronization mode based on the indicator data, the transmission period used in snapshot synchronization mode is determined according to this functional relationship. The worse the network transmission status corresponding to the data receiver, the longer the transmission period can be.

[0118] In some embodiments, the transmission time can be determined based on the amount of snapshot data. Specifically, when the amount of snapshot data reaches a preset data volume threshold, the snapshot data is transmitted to the data receiving end.

[0119] It should be noted that when executing the data synchronization method in the embodiments of this application, step 520 can be executed continuously and in real time. That is, for each data receiving end, its corresponding network transmission status can be continuously monitored, thereby realizing the switching of data synchronization mode with network transmission status.

[0120] For example, in some cases, a data receiver may have been in real-time synchronization mode, but its network transmission status has changed and deteriorated from "good" to "poor" according to a threshold. In this case, the data transmission mode of the data receiver can be switched to snapshot synchronization mode.

[0121] In other cases, a data receiver may have been in snapshot synchronization mode, but if its network transmission status changes from "poor" to "good", the data transmission mode of that data receiver can be switched to real-time synchronization mode.

[0122] In this way, it can better adapt to changes in the network transmission status of each data receiving end, which is conducive to improving the efficiency of data synchronization transmission and the utilization rate of network transmission resources.

[0123] In this embodiment, step 520 can also be performed intermittently, for example, by checking the network transmission status of each data receiving end at regular intervals to update their corresponding data synchronization modes. This application does not impose any restrictions on this.

[0124] Specifically, in this embodiment, when switching the data transmission mode of the data receiving end from real-time synchronization mode to snapshot synchronization mode, if the data receiving end experiences fluctuations within a short period, causing the network transmission status to be temporarily poor, the impact on real-time synchronization transmission is minimal due to the short duration, and switching is unnecessary. Therefore, in this embodiment, if the network transmission status detection result of the data receiving end deteriorates from "excellent" to "poor," the duration of the "poor" network transmission status can be monitored before switching. If this duration reaches a certain threshold, the data transmission mode of the data receiving end is switched to snapshot synchronization mode; otherwise, real-time synchronization mode can be maintained.

[0125] Furthermore, in this embodiment of the application, when switching the data transmission mode corresponding to the data receiving end, in order to prevent data corruption, the current synchronization task of the previous data transmission mode can be completed before switching.

[0126] The data synchronization method provided in this application embodiment can be applied to a data sending end, which is used to synchronously transmit data to multiple data receiving ends and pre-stores the business processing logic of each data receiving end. This method acquires target data, and for each data receiving end, detects the network transmission status between it and the data sending end to determine whether the data receiving end uses real-time synchronization mode or snapshot synchronization mode. For data receiving ends using real-time synchronization mode, the target data is sent immediately, improving the efficiency of target data transmission to the data receiving end. For data receiving ends using snapshot synchronization mode, the target data is pre-processed using the pre-stored business processing logic corresponding to the data receiving end, and then a snapshot of the processing result is sent to the data receiving end, improving the processing efficiency of the target data by the data receiving end. This application can adjust the data synchronization mode based on the network transmission status corresponding to the data receiving end to maximize the utilization of network transmission resources and improve the efficiency of data transmission and processing.

[0127] Specifically, in some embodiments, after acquiring the target data, the method in this application embodiment further includes:

[0128] The target data is converted into multiple log data entries, and the log index corresponding to each log data entry is determined.

[0129] In this embodiment, the target data may include a large amount of data content. To facilitate the orderly and complete transmission of the target data to the data receiving end, in this embodiment, the target data can be converted into log data according to a linked list data structure.

[0130] In this embodiment, the target data may include multiple business-related records. For example, the target data is used to record interactive operations, and each record may include at least one of the following fields: identifier, timestamp, operating account, operation type, details, etc. Each record can be converted into a log data entry, which includes all the contents of the record; it can be a simple line of text data. Thus, the target data can be split and converted into multiple log data entries, and this application does not limit the specific number of log data entries.

[0131] In this embodiment, when converting target data into log data, the conversion can be performed using a linked list data structure. A linked list is a linear data structure comprising multiple elements (nodes), each containing a link to the next element in the list. Each element consists of two parts: a data field and a pointer field. The data field stores the actual log data, while the pointer field points to the next log data. When organizing the log data according to the linked list data structure, the log index corresponding to each log data entry can be determined based on the order of the records within the target data.

[0132] In this embodiment, the log index is a unique identifier assigned to each log data, and different log data correspond to different log indexes. For example, the log index may include at least one of numbers or characters.

[0133] In this embodiment, each record in the target data includes a timestamp. Therefore, when determining the log index corresponding to each log data based on the order of each record in the target data, the log data corresponding to each record can be sorted according to the size of the timestamp, and then the log index for each log data can be generated by incrementing the value from 0.

[0134] For example, please refer to Figure 6, which illustrates a schematic diagram of converting target data into log data according to an embodiment of this application. In this embodiment, the target data includes multiple records. The log data obtained from each record can be arranged according to the order of the record in the target data, and a log index can be assigned to each log data. As shown in Figure 6, the log indexes corresponding to the log data can be numbered sequentially from 1. For example, the log indices corresponding to the log data in Figure 6 are 1, 2, 3...N (N is a positive integer). When the number of records contained in the target data is large, the log indices corresponding to the log data can be sorted one by one according to the above numbering. This application does not limit the specific value of its log indexes.

[0135] In this embodiment of the application, by converting the records in the target data into log data one by one, the log data can be conveniently indexed and marked according to the record sorting. When it is subsequently transmitted to the data receiving end, the sorting of the data can be determined according to the log index, which makes it convenient to synchronize the target data to the data receiving end for processing in an orderly and complete manner.

[0136] Specifically, in some embodiments, when the data synchronization mode is determined to be real-time synchronization mode, the target data is immediately sent to the data receiving end, including:

[0137] Obtain the system identifier of the data receiver;

[0138] Based on the log index, extract the first number of log data as the data to be synchronized in the current batch, and query the first log index corresponding to the first log data at the end of the first number of log data;

[0139] Send the system identifier, the data to be synchronized in the current batch, and the first log index to the data receiving end.

[0140] In this embodiment, "real-time" in the real-time synchronization mode refers to immediate transmission, while the synchronized data can be divided into multiple batches. Specifically, a portion of log data can be extracted from multiple consecutive log data entries based on the log index, serving as the data to be synchronized in the current batch.

[0141] Here, when extracting data to be synchronized, in order to ensure that the data can be transmitted to the data receiving end completely and in an orderly manner, the log data can be extracted according to the continuous log index.

[0142] For example, please refer to Figure 7, which shows a schematic diagram of extracting data to be synchronized according to an embodiment of this application. As shown in Figure 7, the log index corresponding to the log data includes a portion of the log index with values ​​from 187 to 192. Assuming that data to be synchronized needs to be extracted, the log index corresponding to the first log data to be extracted can be determined first. For example, extraction can start from the log data with a corresponding log index of 188. Then, the log data with a corresponding log index of 189, the log data with a corresponding log index of 190, and so on, can be extracted until the total number of log data extracted reaches the number of log data corresponding to the current batch. In this way, these extracted log data are used as the data to be synchronized in the current batch. As shown in Figure 7, if it is preset that 4 log data are extracted in the current batch to constitute the data to be synchronized, then the extracted log data are the log data with corresponding log indices of 188, 189, 190, and 191.

[0143] In this embodiment of the application, after extracting the data to be synchronized, the log index corresponding to the log data at the end of the data to be synchronized can be queried and recorded as the first log index. For example, referring to Figure 7, in the data to be synchronized in the current batch extracted in Figure 7, the log index corresponding to the log data at the end is 191. Therefore, the first log index corresponding to the current batch is 191.

[0144] Furthermore, in this embodiment of the application, the system identifier of the data receiving end (i.e., the identification information of the aforementioned data receiving end) can also be obtained, and then the system identifier, the data to be synchronized, and the first log index can be sent to the data receiving end.

[0145] After receiving the system identifier, the data to be synchronized, and the first log index, the data receiving end can perform data synchronization based on this data. Specifically, the data receiving end can verify whether the received data is correct, i.e., whether it corresponds to the data of this data receiving end, based on the system identifier. Specifically, it can match the received system identifier with the locally stored system identifier. If they match, it indicates that the received data is correct, and synchronization can continue; if they do not match, it means that the data receiving end received incorrectly sent data, and in this case, it can refuse to synchronize the currently received data to be synchronized.

[0146] During data synchronization, the data receiving end can also record a log index in real time. This log index corresponds to the last log data entry in a batch of data to be synchronized, and is designated as the second log index. The data receiving end can update the second log index based on the data in each batch that is synchronized. Furthermore, after each update, the data receiving end can send the updated second log index back to the data sending end, allowing the sending end to monitor the data synchronization progress.

[0147] In this embodiment, after receiving the system identifier, the data to be synchronized, and the first log index, the data receiving end can also compare the first log index and the second log index. If the first log index is greater than the second log index, it indicates that the currently received data to be synchronized is new data and has not yet been synchronized, and the data to be synchronized can be processed normally. Conversely, if the first log index is less than or equal to the second log index, it indicates that the currently received data to be synchronized has already been synchronized previously, and in this case, synchronization of the currently received data to be synchronized can be refused.

[0148] Specifically, in some embodiments, based on the log index, several log data entries are extracted as the data to be synchronized in the current batch, including:

[0149] Query the second log index of the most recent feedback from the data receiver. The second log index indicates the second log data at the end of a batch of log data that the data receiver has completed synchronization.

[0150] Starting from the next log entry after the second log entry, extract the first number of log entries as the data to be synchronized in the current batch.

[0151] In this embodiment, during data synchronization, the data receiving end can record a second log index in real time. The second log index is the log index corresponding to the last log data in a batch of data synchronized by the data receiving end. This log index can be used to record the data synchronization progress of the data receiving end. After each batch of data synchronization is completed, the data receiving end can update the currently recorded second log index and send it back to the data receiving end. This also facilitates the data receiving end in determining the data to be synchronized in each batch.

[0152] Specifically, in this embodiment, when the data receiving end extracts the data to be synchronized in the current batch, it can query the second log index most recently reported by the data receiving end to understand which log data the data receiving end has recently synchronized. Then, starting from the next log data corresponding to the second log data of the most recently reported second log index, a first number of log data can be extracted as the data to be synchronized in the current batch.

[0153] For example, suppose the log index corresponding to the log data obtained from the conversion of the target data includes a portion of the log index with values ​​between 187 and 220. When the data sender determines the data to be synchronized in the current batch corresponding to a certain data receiver, it finds that the second log index most recently fed back by the data receiver is 196. Then, it can start from the log data with the corresponding log index 197 and extract the first number of log data as the data to be synchronized in the current batch corresponding to that data receiver.

[0154] In this embodiment of the application, the first quantity can be a pre-set fixed value, such as 4. Then, the extracted data to be synchronized includes log data with log indices of 197, 198, 199, and 200.

[0155] It should be noted that in this embodiment, the synchronization efficiency of each data receiving end may differ. Therefore, the second log index fed back by each data receiving end may be different. In this embodiment, the data receiving end can record and maintain the second log index for feeding back to the data sending end. When the data sending end receives the second log index fed back by the data receiving end, it updates the previous second log index according to its corresponding system identifier; when sending data to be synchronized to the data receiving end, it also extracts the data to be synchronized according to its corresponding second log index.

[0156] In this embodiment of the application, the data to be synchronized is extracted by using the second log index, which can better combine the synchronization process of the data receiving end to perform data synchronization, reduce the probability of data being sent repeatedly, improve the utilization rate of network transmission resources, and help improve the efficiency of data transmission and processing.

[0157] In some embodiments, the first quantity can also be determined based on the current data synchronization task load at the data sender. Specifically, the first quantity is determined through the following steps:

[0158] The system detects a second number of data receivers currently performing data synchronization, and a third number of data synchronization requests received from the data source during a first preset time period prior to the current time.

[0159] Determine the first quantity based on the second and third quantities;

[0160] Among them, the first and second quantities are negatively correlated, and the first and third quantities are negatively correlated.

[0161] If the current data sending end is under less pressure to perform data synchronization tasks, then the amount of data to be synchronized in each batch can be less, and the number of transmissions can be more, which can facilitate the rapid synchronization of the target data to the data receiving end. Conversely, if the current data sending end is under greater pressure to perform data synchronization tasks, then the amount of data to be synchronized in each batch can be more, and the number of transmissions can be less, which can alleviate the data sending pressure on the data sending end.

[0162] When assessing the pressure of data synchronization tasks at the data sending end, the number of data receiving ends currently performing data synchronization can be detected and recorded as a second quantity. This second quantity can be used to measure the pressure at the current moment. Simultaneously, in this embodiment, the pressure of data synchronization tasks over a recent period can also be considered. For example, the number of data synchronization requests received within a first preset time period prior to the current moment can be detected and recorded as a third quantity. These data synchronization requests can be transmitted from upstream data sources. Whenever new target data is sent to the data sending end, a data synchronization request is triggered, requesting the data sending end to perform a data synchronization task for that target data. The length of the first preset time period set in this embodiment can be determined according to actual needs; this application does not impose any restrictions.

[0163] In this embodiment of the application, the larger the second quantity, the more data receiving ends the current data sending end needs to connect to, and the greater the pressure on the data synchronization task; the larger the third quantity, the more target data the data sending end needs to send in the recent period, and similarly, the greater the pressure on the data synchronization task.

[0164] Therefore, in this embodiment, the size of the first quantity can be determined based on the second and third quantities. The larger the second and third quantities are, the smaller the determined first quantity is; conversely, the smaller the second and third quantities are, the larger the determined first quantity is. That is, the first and second quantities are negatively correlated, and the third quantity is also negatively correlated. The first quantity can be 1 or any integer greater than 1. This application does not impose any restrictions on the specific functional relationship between them.

[0165] In this embodiment, the first quantity is flexibly determined based on the second and third quantities, which can facilitate the adjustment of the data to be synchronized in each batch of transmission. When the data synchronization task pressure at the data sending end is low, the data transmission efficiency is improved; when the data synchronization task pressure at the data sending end is high, the transmission frequency is appropriately reduced, thereby alleviating the data transmission pressure at the data sending end.

[0166] Specifically, in some embodiments, the method in this application further includes:

[0167] Record the first moment when the current batch of data to be synchronized is sent to the data receiver;

[0168] When the second log index corresponding to the current batch is received from the data receiver within the second preset time period after the first moment, the next batch of data to be synchronized is extracted according to the second log index corresponding to the current batch.

[0169] If the second log index corresponding to the current batch is not received from the data receiver within the second preset time period after the first moment, the data to be synchronized for the current batch will be resent to the data receiver.

[0170] As described above, in this embodiment of the application, the data receiving end can record the second log index in real time during the data synchronization process. The second log index is the log index corresponding to the log data at the end of a batch of data synchronized by the data receiving end. This log index can be used to record the data synchronization process of the data receiving end.

[0171] In this embodiment, for each batch of data to be synchronized, after the data sender transmits the data to the receiver, the transmission time of the current batch of data to be synchronized can be recorded. In this embodiment, this is recorded as the first moment. Then, it is possible to monitor in real time whether a second log index corresponding to the current batch is received. Here, when monitoring the receipt of a second log index corresponding to the current batch, the received second log index can be matched with the first log index of the current batch. If the received second log index is found to be the same as the first log index of the current batch, it indicates that the received second log index corresponds to the current batch.

[0172] In this embodiment, after the current batch of data to be synchronized is sent to the data receiving end, under normal circumstances, the data receiving end will quickly receive and complete the data synchronization of the current batch of data to be synchronized. Therefore, the data sending end can receive the second log index corresponding to the current batch in a short time. If the data sending end does not receive the second log index corresponding to the current batch, it may be due to a problem in the data transmission process, and the current batch of data to be synchronized was not sent to the data receiving end normally; or it may be due to a fault in the data receiving end, which cannot complete the data synchronization task normally. In this embodiment, a preset time period can be set, denoted as the second preset time period. The length of the second preset time period can be slightly shorter, and this application does not limit its specific length.

[0173] For the set second preset time period, the duration can be accumulated starting from the first moment. If the data sending end receives the second log index corresponding to the current batch after the first moment, before the accumulated duration reaches the second preset time period, it means that the data receiving end has successfully received and completed the data synchronization of the current batch of data to be synchronized. At this time, the next batch of data to be synchronized can be extracted according to the second log index of the current batch. The specific extraction logic has been described in the previous embodiment and will not be repeated here.

[0174] Conversely, if the accumulated time has reached the second preset time period, but the second log index corresponding to the current batch has not yet been received, it indicates that there may be a problem with the data transmission or data synchronization of the data to be synchronized in the current batch. In this case, the data to be synchronized in the current batch can be resent to the data receiving end to ensure the reliability and stability of data transmission.

[0175] It should be noted that in this embodiment, in real-time synchronization mode, the maximum number of retransmissions can be preset. For each batch of data to be synchronized, if the corresponding second log index is not received after multiple retransmissions, it indicates that the network transmission status between the data receiver and the receiver may be abnormal, or the receiver may be faulty. In this case, to reduce the waste of network transmission resources, the data synchronization mode corresponding to the receiver can be adjusted to snapshot synchronization mode. In this embodiment, the maximum number of retransmissions can be flexibly set according to actual needs.

[0176] Specifically, in some embodiments, the target data is processed according to business processing logic, and a snapshot of the processing result is sent to the data receiving end, including:

[0177] Obtain the system identifier of the data receiver;

[0178] The log data is processed according to the business processing logic to obtain the processing result;

[0179] When the amount of unsent processing results data reaches a preset capacity threshold, or when the time since the last snapshot data transmission reaches a preset duration threshold,

[0180] Query the third log index corresponding to the last log record currently being processed;

[0181] Based on a preset snapshot data structure, the unsent processing results are packaged into snapshot data to be sent;

[0182] The system identifier, the snapshot data to be sent, and the third log index are sent to the data receiving end.

[0183] In this embodiment, for the snapshot synchronization mode, the target data can also be transmitted in the form of log data. For each log data, the business processing logic of the data receiving end can preprocess it to obtain the corresponding processing result and cache it. Then, the snapshot data to be sent can be transmitted in a periodic or quantitative manner. For example, a time threshold can be preset. If the time since the last snapshot data transmission reaches the preset time threshold, the snapshot data can be sent to the data receiving end. Alternatively, a capacity threshold can be preset. When the amount of data of the processing results to be sent (corresponding to one or more log data) accumulates to reach the preset capacity threshold, the snapshot data is sent to the data receiving end.

[0184] In this embodiment, the system identifier of the data receiver can also be obtained, facilitating the data receiver to verify the received data before synchronization, thus improving the reliability of data synchronization. When sending snapshot data, the third log index corresponding to the last log data currently being processed can be queried, and the snapshot data to be sent, the system identifier, and the third log index can be sent together to the data receiver. The snapshot data to be sent corresponds to one or more log data entries.

[0185] In this embodiment, for the snapshot synchronization mode, the data receiver can also maintain an index, denoted as the fourth log index. The fourth log index represents the log index corresponding to the last log data in a batch of snapshot data synchronized by the data receiver. Furthermore, the fourth log index can be fed back to the data sender. Thus, the data sender can determine which log data processing results (which could be all unsynchronized processing results) to send subsequently based on the fourth log index fed back by the data receiver. Additionally, the data receiver can compare the relationship between the third log index and the fourth log index. When the third log index is less than or equal to the fourth log index, it indicates that the received snapshot data has already been synchronized, and synchronization can be skipped using the currently received snapshot data.

[0186] In this embodiment, when sending processing results, the processing results corresponding to multiple log data can be packaged into snapshot data of a specific format to improve data transmission efficiency. Specifically, the data sending end obtains a preset snapshot data structure based on the system identifier of the data receiving end, and packages the multiple unsent processing results according to this snapshot data structure to obtain the snapshot data sent to the data receiving end.

[0187] In this embodiment, the processing result is generated according to the order of the log data. Therefore, the obtained snapshot data is complete and can be easily used by the data receiving end, which is beneficial to improving the data synchronization and processing efficiency.

[0188] Specifically, in some embodiments, detecting the network transmission status between the electronic device and the data receiver includes:

[0189] Detect network environment parameters between electronic devices and data receivers;

[0190] Based on the network transmission status, determine whether the data synchronization mode at the data receiving end is real-time synchronization mode or snapshot synchronization mode, including:

[0191] Determine the network quality score corresponding to the data receiving end based on network environment parameters;

[0192] When the network quality score is greater than the preset score threshold, the data synchronization mode corresponding to the data receiving end is determined to be the real-time synchronization mode.

[0193] When the network quality score is less than or equal to the preset score threshold, the data synchronization mode corresponding to the data receiving end is determined to be snapshot synchronization mode.

[0194] In this embodiment, network environment parameters may include, but are not limited to, parameters such as the retransmission rate (retran_ratio), the number of timeout retransmission packets (retransmit_timeout_packets), the number of transaction layer packet (TLP) retransmissions, the round-trip time (RTT), the maximum window size (max_cwnd), the packet loss rate (loss_rate), and the throughput (goodput). The round-trip time may include types such as minimum round-trip time (min_rtt), smoothed round-trip time (srtt), and maximum round-trip time (max_rtt). These network environment parameters can be values ​​at the current moment or determined based on values ​​within a predetermined time period prior to the current moment; this application does not impose any restrictions on this.

[0195] In this embodiment, the network quality score is positively correlated with the stability of the network environment of the session connection between the data sender and receiver. If the network environment of the session connection is highly volatile and the transmission effect is poor, the network quality score corresponding to the data receiver will be low; conversely, if the network environment of the session connection is low volatile and the transmission effect is good, the network quality score corresponding to the data receiver will be high. In this embodiment, the specific functional relationship between network environment parameters and the network quality score is not limited; various types of parameters can be combined and weighted to determine the network quality score corresponding to the data receiver.

[0196] After obtaining the network quality score of the data receiving end, the data synchronization mode adopted can be determined based on the network quality score. For example, a score threshold can be preset. If the network quality score is greater than the preset score threshold, it indicates that the network environment is good, and the data synchronization mode of the data receiving end can be determined to be real-time synchronization mode. Conversely, if the network quality score is less than or equal to the preset score threshold, it indicates that the network environment is poor, and the data synchronization mode of the data receiving end can be determined to be snapshot synchronization mode.

[0197] In this embodiment, the data synchronization mode corresponding to the data receiver is determined by comprehensively considering the network environment parameters between the data sender and the data receiver. This can improve the utilization rate of network transmission resources, make more efficient use of the limited data transmission capacity of the data sender, and the data synchronization mode is well applicable to the data receiver.

[0198] Specifically, in some embodiments, the data sending end is a distributed source data system, which includes a master node and multiple slave nodes. The master node is used to transmit data synchronously to multiple data receiving ends.

[0199] Obtain the target data to be sent to the data receiver, including:

[0200] The master node is used to obtain raw data, which is then transformed to obtain standard data that conforms to a common data exchange format.

[0201] The master node sends standard data to each slave node, and the slave nodes reach a consensus on the standard data, which is then used as the target data.

[0202] In some scenarios, the data sending end can adopt a distributed source data system, which can include multiple nodes, specifically a master node and multiple slave nodes. The master node can be used to transmit data synchronously to multiple data receiving ends, while the slave nodes can serve as backup nodes. When the master node fails, the system can switch to the slave nodes to run, thereby improving the reliability and stability of data synchronization.

[0203] For the aforementioned data sending end employing a distributed source data system implementation, the relevant data needs to be backed up at the slave nodes. For the target data, it can first obtain the raw data through the master node; the data type of the raw data can be arbitrary. Then, the master node can perform format conversion on the raw data to obtain standard data belonging to a universal data exchange format. This standard data is applicable to various systems, facilitating the assurance of data consistency and readability.

[0204] In this embodiment, after obtaining the standard data, it can be distributed to each slave node for backup and consensus. Once the slave nodes have reached a consensus on the standard data, it can be used as the target data and synchronously transmitted to each data receiving end.

[0205] Specifically, in some embodiments, standard data is sent from the master node to each slave node, and consensus on the standard data is achieved through the slave nodes, including:

[0206] The master node performs hash processing on the standard data to obtain the first hash value; the standard data and the first hash value are then sent to each slave node.

[0207] By performing hash processing on the standard data from the node, a second hash value is obtained, and the consistency between the first hash value and the second hash value is verified to obtain the consistency verification result corresponding to the standard data.

[0208] When more than half of the slave nodes pass the consistency check, the standard data is considered to have passed the consensus.

[0209] In this embodiment of the application, when reaching consensus on standard data, the master node can perform hash processing on the standard data to calculate a hash value, which is recorded as the first hash value. Then, the master node can send the standard data and the first hash value to each slave node, and the slave nodes can act as consensus nodes to verify the consistency of the standard data.

[0210] Specifically, when a slave node performs consistency verification on standard data, it can also hash the standard data to calculate a hash value, which is then recorded as the second hash value. The first and second hash values ​​can then be compared; if they match, the consistency verification passes, and the slave node generates and returns the corresponding consensus signature. The master node can receive and accumulate the number of consensus signatures returned by each slave node. When the total number reaches a certain threshold (generally more than half the number of slave nodes), it can be determined that the consensus on the standard data has been achieved. In this embodiment, the specific type of hash algorithm used is not limited.

[0211] In this embodiment, a distributed source data system is used to handle data synchronization tasks. A consensus mechanism can be used to back up data, preventing single points of failure. This improves the reliability and stability of data synchronization.

[0212] Specifically, in some embodiments, the data sending end is a distributed source data system, which includes a master node and multiple slave nodes. The master node is used to perform synchronous data transmission to multiple data receiving ends. The method in this application embodiment also includes:

[0213] Synchronize the data in the master node to each slave node;

[0214] Detect the operational status of the distributed source data system;

[0215] When it is determined from the operating status that the distributed source data system needs to switch master nodes, a new master node is determined from the slave nodes;

[0216] Check if there are any incomplete data synchronization tasks in the old master node; among them, incomplete data synchronization tasks are indicated by log indexes that have sent data to be synchronized but have not received feedback from the data receiver afterward.

[0217] When there are unfinished data synchronization tasks on the old master node, stop sending new data to be synchronized through the old master node, and wait until the log index is received from the data receiver or the timeout occurs, then switch from the old master node to the new master node.

[0218] In this embodiment, the data sending end adopts a distributed source data system implementation, which can synchronize data from the master node to each slave node, such as target data (i.e., the standard data after consensus) and log indexes fed back by the data receiving end. During the execution of the data synchronization task, the running status of the distributed source data system can be detected. If the current master node is found to be faulty, or if the master node needs to be switched due to factors such as load balancing, one of the slave nodes can be selected as the new master node.

[0219] It should be noted that since the master node switchover of a distributed source data system can occur at any time, in order to better coordinate data synchronization tasks during the switchover process, in this embodiment, after the new master node is determined, if the old master node is running normally, it can be further checked whether there are any incomplete data synchronization tasks on the old master node. For real-time synchronization mode, incomplete data synchronization tasks may include data synchronization tasks that have sent data to be synchronized but have not received feedback from the data receiver regarding the second log index. For snapshot synchronization mode, incomplete data synchronization tasks may include data synchronization tasks that have sent snapshot data but have not received feedback from the data receiver regarding the fourth log index. If such tasks exist, the master node switchover can be temporarily suspended, but the sending of new data to be synchronized (and snapshot data) through the old master node can be stopped. The switchover can be performed after receiving feedback from the data receiver regarding the log index or after a timeout. After the switchover, the new master node can be used normally for data synchronization and transmission. In this way, the simultaneous sending of data to be synchronized (and snapshot data) by both the old and new master nodes can be avoided, reducing the probability of data transmission chaos and improving the utilization of network transmission resources.

[0220] In this embodiment of the application, if the old master node cannot operate normally due to faults or other factors, or if it does not have any unfinished data synchronization tasks, it can be directly switched to a new master node for data synchronization and transmission.

[0221] The following describes and explains a data synchronization method provided in this application embodiment, with reference to specific application examples.

[0222] The data synchronization method in this application embodiment can be applied to a distributed source data system built on the Raft algorithm. In this application embodiment, the distributed source data system can be referred to as an RSS system (Raft Source System).

[0223] For example, please refer to Figure 8, which illustrates the application principle of an RSS system provided in an embodiment of this application. In this embodiment, the distributed source data system 802 may include one master node and several backup nodes (i.e., slave nodes). The number of backup nodes in Figure 8 is two, but it can also be one or more of any number; this application does not impose any restrictions on this. The Raft protocol can be run between the master node and the backup nodes. The master node is mainly responsible for receiving and synchronizing data, while the backup nodes can perform data consensus and backup, and switch to be used as the new master node when appropriate.

[0224] In Figure 8, the distributed source data system 802 acts as the data sender, and the data receivers are some downstream systems, which can be referred to as target systems in this embodiment. Figure 8 includes three data receivers: target system A, target system B, and target system C. In this embodiment, for each data receiver, different synchronization modes are provided based on the network transmission status between them and the distributed source data system 802, namely, real-time synchronization mode and snapshot synchronization mode.

[0225] Each target system, upon receiving data from the distributed source data system 802, checks the data's validity. This includes verifying whether the system identifier matches its own and whether the push progress aligns with the current synchronization process. If valid, the system receives and processes the data.

[0226] Taking real-time synchronization mode as an example, when data from upstream system 801 is transmitted to distributed source data system 802, the data is first received by the master node. The master node can perform format conversion processing to obtain standard data, such as the GDEF format mentioned above. Then, the standard data is sent to each backup node for consensus. Based on the Raft protocol, it can be ensured that more than half of the backup nodes put the standard data into their own logs. The master node and backup nodes store the data in log format. After the standard data passes consensus, it can be used as target data for real-time synchronization mode transmission and sent to the target system. The target system can parse the target data into its own system-defined format and then process it. As shown in Figure 8, target system A and target system B use real-time synchronization mode.

[0227] In snapshot synchronization mode, the business processing logic corresponding to each target system can be encapsulated into execution scripts and injected into the master and backup nodes of the distributed source data system 802. Data sent by the upstream system 801 can be pre-processed on the master node through business processing logic without needing to be converted into standard data. The data processed by the master node can be packaged and stored periodically or quantitatively, and sent to the target system as snapshot data. As shown in Figure 8, target system C uses snapshot synchronization mode.

[0228] In this embodiment, the target system can report its synchronization progress to the distributed source data system. As in the previous embodiments, the second log index and the fourth log index are used to inform the current synchronization progress. In this embodiment, the distributed source data system can maintain a synchronization progress table to store these indexes. Once feedback is received from a target system, the synchronization progress table is updated, and the information is sent to the backup node. The synchronization progress of each target system is independent, and all data after the minimum progress level can be retained in the distributed source data system.

[0229] Referring to Figure 9, which illustrates various data structures transmitted in the embodiments of this application, the data structure transmitted in real-time synchronization mode is denoted as synchronization data structure 901, the data structure transmitted in snapshot synchronization mode is denoted as snapshot structure 902, and the data structure fed back from the target system to the distributed source data system is denoted as reply data structure 903. These will be described in detail below.

[0230] Referring to Figure 9, in real-time synchronization mode, each time the master node in the distributed source data system sends data to be synchronized to the target system, its overall data structure may include a target system identifier (Target-ID), a general dataset (GDEF-Set), a log index (Last-Index), and a source node identifier (From-RSS-ID). The target system identifier allows the target system to query whether the sent data matches its local identifier; if not, it can reject the data. The general dataset (i.e., the data to be synchronized) can be extracted and merged from log data and can be used for data synchronization in the target system. The log index (i.e., the first log index) identifies the index of the last log data in the general dataset and can represent the push progress of the general dataset. After receiving the data, the target system can check if it is greater than the log index in the previous batch of sent data; if not, it can reject the data and simultaneously reply to the distributed source data system with the largest log index it has received. The source node identifier identifies the node in the distributed source data system that sent the data.

[0231] In snapshot synchronization mode, each time the master node in the distributed source data system sends snapshot data to the target system, its overall data structure may include a target system identifier (Target-ID), snapshot data (Snap), log execution index (Exec-Index), and source node identifier (From-RSS-ID). The meanings of the target system identifier and source node identifier are the same as described above and will not be repeated here. Snapshot data is obtained by pre-processing log data and formatting the generated processing results into a snapshot format. The log execution index (i.e., the third log index) is used to identify the index corresponding to the last log data in the snapshot data and can represent the push progress of the snapshot data.

[0232] The response data structure 903 from the target system to the distributed source data system includes a master node identifier (To-RSS-ID) and a synchronization progress (Finish-Index). The master node identifier is used to respond to the specified node; if multiple pieces of data to be synchronized or snapshot data are received, multiple responses are required. The master node identifier may change during master node switching. The synchronization progress is used to respond to the index corresponding to the currently synchronized log data or snapshot data. When the target system uses real-time synchronization mode, it responds with the index of the synchronized log data (i.e., the second log index); when the target system uses snapshot synchronization mode, it responds with the index corresponding to the synchronized snapshot data (i.e., the fourth log index). For the distributed source data system, when it receives the synchronization progress, it can use the largest index as the updated synchronization progress.

[0233] Referring to Figure 10, which illustrates a schematic diagram of data synchronization provided in an embodiment of this application, as shown in Figure 10, after data is sent to the distributed source data system via the upstream system, corresponding log data is generated. This log data is stored in a linked list structure that is continuously appended to the end, expanding backward from the starting index position. In this embodiment, a consensus log index (Commit-Index) can be marked, which can record the position of the last log data that has completed consensus in the linked list structure.

[0234] In this embodiment of the application, under real-time synchronization mode, the master node maintains two indexes for each target system: a first log index (the aforementioned log index (Last-Index)) and a second log index (the synchronized progress (Finish-Index) in the aforementioned real-time synchronization mode). The initial value for both is the starting index 0. The generation process of the synchronization data structure 901 is as follows:

[0235] Step 1a): Move the first log index corresponding to the target system to a certain threshold. This threshold can be an index that is several positions after the second log index, and it needs to be less than or equal to the consensus log index.

[0236] Step 2a): Take all log data within the interval [second log index, first log index] (where left-open and right-closed), and read them one by one to obtain the general dataset.

[0237] Step 3a): Assemble the target system identifier, general dataset, log index and source node identifier to obtain the synchronization data structure 901.

[0238] As shown in Figure 10, for target system A, after determining the position of its current second log index, the position of the first log index is determined according to a set number. Then, starting from the next log data corresponding to the second log index and ending at the position corresponding to the first log index, these log data are extracted as the data to be synchronized in the current batch and sent to target system A. The set number can be determined based on the data transmission pressure of the master node. It should be noted that different target systems may be in different synchronization processes; therefore, a piece of log data may generate several different synchronization data structures 901. This application does not impose any restrictions on this.

[0239] Step 4a): Update the old second log index based on the second log index replied by the target system.

[0240] Under normal circumstances, the new second log index will be the first log index of the previous batch of synchronized data structure 901. As shown in Figure 10, when the master node receives the new second log index, it will update its position and continue to generate synchronized data structure 901 based on the new second log index.

[0241] As can be seen, in this embodiment of the application, in the real-time synchronization mode, the data of the target system is processed through a process of continuously advancing the synchronization progress. Step 2a) above can ensure that the data retrieved from the log data by each target system is ordered. In step 4a), when the second log index is updated, it will not cross the first log index of the previous batch of synchronized data structure 901, and there will be no data gaps, ensuring the integrity of the retrieved data. In the real-time synchronization mode, only the order of the data is concerned, not the content of the data. After the target system receives the data, it still needs to parse and process it.

[0242] In this embodiment of the application, under snapshot synchronization mode, the master node maintains two indexes for each target system: a third log index (the aforementioned log execution index (Exec-Index)) and a fourth log index (the synchronized progress (Finish-Index) in the aforementioned snapshot synchronization mode). The initial value for both is the starting index 0. The generation process of snapshot structure 902 is as follows:

[0243] Step 1b): The target system first injects the execution script and synchronization configuration into the distributed source data system. The synchronization configuration specifies the time and size for transmitting snapshot data, and the execution script is responsible for processing log data in advance on behalf of the target system.

[0244] Step 2b): When new log data passes consensus, the consensus log index increases, and the execution script immediately processes the log data (including parsing and execution of business logic). The execution result is temporarily stored in the master node (and the backup node), while the third log index moves forward. In this embodiment, the business processing logic (i.e., the execution script) is pre-stored in the distributed source data system, which will replace the target system to complete the data processing. Finally, it is only necessary to synchronize the execution result to the target system through a snapshot, which can improve the efficiency of data processing in the target system.

[0245] Step 3b): When a configured threshold (such as a preset capacity threshold or a preset duration threshold) is triggered, the temporarily stored processing results are extracted, packaged into snapshot data in a specific format, and the above snapshot structure 902 is generated. This snapshot data is equivalent to the data set within the interval [fourth log index, third log index] (where left-open and right-closed).

[0246] Step 4b): The synchronization progress table processing logic after the target system responds is similar to step 4a) in the real-time synchronization mode. The fourth log index will be updated to the position of the third log index, and then the above step 2b will continue.

[0247] As shown in Figure 10, for the target system C, the master node can generate snapshot data through the above process and send it to the target system C.

[0248] In this embodiment of the application, the snapshot synchronization mode is a process of advancing the synchronization progress in stages. The advancement of the fourth log index has a large span, but the advancement of the third log index in step 2b) above is real-time and orderly. Therefore, the snapshot synchronization mode will not have data gaps, the data has integrity, and it can improve the efficiency of the target system in processing data.

[0249] In this embodiment, a distributed source data system is employed, which can maintain data consistency even under abnormal circumstances. Specifically, abnormal scenarios mainly include:

[0250] I. Network failure:

[0251] Data synchronized by the distributed source data system fails to reach the target system, or the response from the target system fails to reach the distributed source data system. For each target system, the distributed source data system starts a timer when sending synchronized data. If a response is received from the target system before the timer expires, normal execution continues; otherwise, the distributed source data system repeats the sending steps, and the relevant indexes remain unchanged until the retry succeeds.

[0252] II. Master Node Switching:

[0253] In a distributed source data system, due to network issues or load balancing, the master node may switch to another node. This switch can occur at any time, potentially causing two nodes to simultaneously send synchronization data to the target system. Therefore, the new master node can wait for a period to allow the old master node to complete its ongoing synchronization. When the old master node detects a new master node, it stops sending new synchronization data and only completes its current synchronization task. Upon completion, it does not modify the indexes of each target system and notifies the new master node of the latest synchronization progress. The new master node then continues data synchronization based on the updated indexes. However, due to network failures or other reasons, the old master node may be unable to complete its ongoing synchronization task. In this case, the old master node will not retry and will directly notify the new master node.

[0254] III. Target system crash:

[0255] If the target system crashes, it cannot recover the progress of the synchronization from the distributed source data system. At this time, the target system may have only synchronized a portion of the data. The solution in this case is the same as for the network failure described above: the distributed source data system will retry periodically after not receiving a response. When the target system restarts, it will still receive the data that was not processed in the previous attempt, and the integrity and consistency of the data will not be affected.

[0256] IV. Distributed source data system downtime:

[0257] In a distributed source data system, each node periodically persists a synchronization progress table. If the distributed source data system recovers, it reads the relevant indexes of each target system recorded on disk and continues to execute the data synchronization task.

[0258] Taking real-time synchronization mode as an example, referring to Figure 11, Figure 11 illustrates the principle of a distributed source data system crash rollback provided in this embodiment. In this embodiment, since persistent storage is executed periodically, when a crash occurs, the persistent second log index that may be recovered from the disk will be slightly smaller than the second log index before the crash. At this time, the second log index will be rolled back to the persistent second log index. The target system may receive a piece of already processed synchronization data; however, due to the ordered nature of the log data, although the data may be duplicated, there will be no gaps, and the target system can eventually synchronize to complete and consistent data.

[0259] In summary, based on the various embodiments of this application, the following technical effects can be obtained:

[0260] First, it can synchronize data with heterogeneous systems via the Raft protocol. Data export is ordered, real-time, and reliable. The asynchronous transmission process does not affect the original performance of the system, and heterogeneous systems do not affect each other.

[0261] Secondly, the technical solution is highly adaptable, providing both real-time synchronization and snapshot synchronization modes for different scenarios. Among these,

[0262] Real-time synchronization is suitable for scenarios with high requirements for data sequence, timeliness, and integrity. It can ensure the orderliness and integrity of data, and duplicate transmission will only occur in some abnormal scenarios. At the same time, synchronizing GDEF data ensures that the processing logic of the application layer can also be synchronized.

[0263] Regular snapshot synchronization moves the target system's processing logic forward to the RSS, making it suitable for scenarios where data timing and timeliness requirements are not high or where the target system's machine performance is relatively weak. The execution calculation process is injected into the RSS side, and the synchronization process can be triggered once at relatively long intervals. After the target system receives the snapshot, it does not need to process it again, effectively reducing the pressure on the target system. At the same time, the data transmitted through snapshots also has integrity.

[0264] Both modes meet the data consistency requirements.

[0265] Finally, in scenarios where data synchronization is required but network conditions are poor and machines are prone to failure, the target system may experience data loss due to network partitions or system crashes. This technical solution possesses anomaly recovery capabilities. Through replay and retries, data synchronization maintains reliability, fault tolerance, and consistency under various abnormal conditions. Specifically, RSS's Raft protocol ensures that consensus-reached data is not lost, and maintaining synchronization progress guarantees data integrity and consistency. Synchronization progress is updated only when the target system successfully responds. If an anomaly occurs during this process, RSS attempts to replay the unprocessed data. Due to the ordered nature of Raft logs, RSS can always find where to start replaying data. Furthermore, the replay position of each target system is recorded independently, ensuring that the data processed by the target system is always reliable. The synchronization process is fault-tolerant, guaranteeing eventual data consistency even in the event of severe anomalies.

[0266] Referring to Figure 12, in this embodiment of the application, a data synchronization device is also provided, which can be used at a data sending end. The data synchronization device includes:

[0267] Acquisition unit 1210 is used to acquire target data to be sent to the data receiving end;

[0268] The detection unit 1220 is used to detect the network transmission status between the electronic device and the data receiving end, and determine whether the data synchronization mode corresponding to the data receiving end is real-time synchronization mode or snapshot synchronization mode based on the network transmission status.

[0269] The first processing unit 1230 is configured to immediately send the target data to the data receiving end when the data synchronization mode is determined to be the real-time synchronization mode; and,

[0270] The second processing unit 1240 is used to query the pre-stored business processing logic of the data receiving end when the data synchronization mode is determined to be the snapshot synchronization mode, process the target data according to the business processing logic, and send the snapshot data of the processing result to the data receiving end.

[0271] Optionally, in some embodiments, the apparatus in this application further includes a conversion unit, which is specifically used for:

[0272] The target data is converted into multiple log data entries, and the log index corresponding to each log data entry is determined.

[0273] Optionally, in some embodiments, the first processing unit is specifically used for:

[0274] Obtain the system identifier of the data receiving end;

[0275] Based on the log index, extract a first number of log data as the data to be synchronized in the current batch, and query the first log index corresponding to the first log data at the end of the first number of log data;

[0276] The system identifier, the data to be synchronized in the current batch, and the first log index are sent to the data receiving end.

[0277] Optionally, in some embodiments, the first processing unit is specifically used for:

[0278] Query the second log index most recently fed back by the data receiving end. The second log index indicates the second log data at the end of a batch of log data that the data receiving end has completed synchronization.

[0279] Starting from the next log entry of the second log data, extract the first number of log data entries as the data to be synchronized in the current batch.

[0280] Optionally, in some embodiments, the first processing unit is specifically used for:

[0281] The system detects a second number of data receivers currently performing data synchronization, and a third number of data synchronization requests received from the data source during a first preset time period prior to the current time.

[0282] The first quantity is determined based on the second quantity and the third quantity;

[0283] The first quantity and the second quantity are negatively correlated, and the first quantity and the third quantity are negatively correlated.

[0284] Optionally, in some embodiments, the first processing unit is further configured to:

[0285] Record the first moment when the current batch of data to be synchronized is sent to the data receiving end;

[0286] When the second log index corresponding to the current batch is received from the data receiving end within the second preset time period after the first moment, the next batch of data to be synchronized is extracted according to the second log index corresponding to the current batch.

[0287] If the second log index corresponding to the current batch is not received from the data receiving end within a second preset time period after the first moment, the data to be synchronized for the current batch is resent to the data receiving end.

[0288] Optionally, in some embodiments, the second processing unit is specifically used for:

[0289] Obtain the system identifier of the data receiving end;

[0290] The log data is processed according to the business processing logic to obtain the processing result;

[0291] When the amount of unsent processing results data reaches a preset capacity threshold, or when the time elapsed since the last snapshot data transmission reaches a preset time threshold,

[0292] Query the third log index corresponding to the last log record currently being processed;

[0293] Based on a preset snapshot data structure, the unsent processing results are packaged into snapshot data to be sent;

[0294] The system identifier, the snapshot data to be sent, and the third log index are sent to the data receiving end.

[0295] Optionally, in some embodiments, the detection unit is specifically used for:

[0296] Detect the network environment parameters between the electronic device and the data receiving end;

[0297] The step of determining whether the data synchronization mode corresponding to the data receiving end is real-time synchronization mode or snapshot synchronization mode based on the network transmission status includes:

[0298] Based on the network environment parameters, determine the network quality score corresponding to the data receiving end;

[0299] When the network quality score is greater than a preset score threshold, the data synchronization mode is determined to be the real-time synchronization mode;

[0300] When the network quality score is less than or equal to the preset score threshold, the data synchronization mode is determined to be the snapshot synchronization mode.

[0301] Optionally, in some embodiments, the data sending end is a distributed source data system, which includes a master node and multiple slave nodes. The master node is used for synchronous data transmission to multiple data receiving ends; the acquisition unit is specifically used for:

[0302] The master node is used to obtain raw data, which is then converted to obtain standard data that conforms to a common data exchange format.

[0303] The master node sends the standard data to each slave node, and the slave nodes reach a consensus on the standard data, determining the consensus-agreed standard data as the target data.

[0304] Optionally, in some embodiments, the acquisition unit is specifically used for:

[0305] The master node performs hash processing on the standard data to obtain a first hash value; the standard data and the first hash value are then sent to each of the slave nodes.

[0306] The slave node performs hash processing on the standard data to obtain a second hash value, and verifies the consistency between the first hash value and the second hash value to obtain the consistency verification result corresponding to the standard data.

[0307] When more than half of the slave nodes pass the consistency check, the standard data is determined to have passed consensus.

[0308] Optionally, in some embodiments, the data sending end is a distributed source data system, which includes a master node and multiple slave nodes. The master node is used to synchronously transmit data to multiple data receiving ends. The apparatus in this embodiment further includes a switching unit, which is specifically used for:

[0309] Synchronize the data in the master node to each of the slave nodes;

[0310] Detect the operating status of the distributed source data system;

[0311] When it is determined from the operating status that the distributed source data system needs to switch master nodes, a new master node is determined from each of the slave nodes;

[0312] Detect whether there are any incomplete data synchronization tasks in the old master node; wherein, the incomplete data synchronization task indicates: a log index that has been sent to synchronize data but has not been received from the data receiving end;

[0313] When there are unfinished data synchronization tasks in the old master node, the sending of new data to be synchronized through the old master node is stopped. The system waits until the log index is received from the data receiving end or the timeout occurs, and then switches from the old master node to the new master node.

[0314] In this application embodiment, the content of the data synchronization method embodiment shown in FIG5 is applicable to the data synchronization device embodiment. The specific functions implemented by the data synchronization device embodiment are the same as those of the data synchronization method embodiment shown in FIG5, and the beneficial effects achieved are also the same as those achieved by the data synchronization method embodiment shown in FIG5.

[0315] This application also discloses an electronic device, including:

[0316] At least one processor;

[0317] At least one memory for storing at least one program;

[0318] When at least one program is executed by at least one processor, the at least one processor implements the data synchronization method embodiment shown in FIG5.

[0319] The electronic device in the embodiments of this application may be a terminal device, a computer device, or a server device.

[0320] For example, referring to Figure 13, which is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Taking the electronic device as a terminal device as an example, in Figure 13, the electronic device 1300 may include an RF (Radio Frequency) circuit 1310, a memory 1320 including one or more computer-readable storage media, an input unit 1330, a display unit 1340, a sensor 1350, an audio circuit 1360, a short-range wireless transmission module 1370, a processor 1380 including one or more processing cores, and a power supply 1390, etc. Those skilled in the art will understand that the device structure shown in Figure 13 does not constitute a limitation on the terminal device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0321] RF circuit 1310 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and hands it over to one or more processors 1380 for processing; additionally, it transmits uplink data to the base station. Typically, RF circuit 1310 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a SIM card, a transceiver, a coupler, an LNA (Low Noise Amplifier), a duplexer, etc. Furthermore, RF circuit 1310 can also communicate wirelessly with networks and other devices. Wireless communication can use any communication standard or protocol, including but not limited to GSM (Global System for Mobile communication), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, SMS (Short Messaging Service), etc.

[0322] The memory 1320 can be used to store software programs and modules (or units). The processor 1380 executes various functional applications and data processing by running the software programs and modules (or units) stored in the memory 1320. The memory 1320 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 1300 (such as audio data, telephone book, etc.). In addition, the memory 1320 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1320 may also include a memory controller to provide access to the memory 1320 for the processor 1380 and the input unit 1330. Although FIG13 shows the RF circuit 1310, it is not an essential component of the electronic device 1300 in the embodiments of this application and can be omitted as needed without changing the essence of the invention.

[0323] The input unit 1330 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to object settings and function control. Specifically, the input unit 1330 may include a touch-sensitive surface 1331 and other input devices 1332. The touch-sensitive surface 1331, also known as a touch display screen or touchpad, can collect touch operations on or near the object (such as operations performed by the object using a finger, stylus, or any suitable object or accessory on or near the touch-sensitive surface 1331), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface 1331 may include two parts: a touch detection device and a touch controller. The touch detection device detects the touch position of the object and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 1380, and can receive and execute instructions from the processor 1380. In addition, the touch-sensitive surface 1331 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. Besides the touch-sensitive surface 1331, the input unit 1330 may also include other input devices 1332. Specifically, other input devices 1332 may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.

[0324] Display unit 1340 can be used to display information input by an object or information provided to an object, as well as various graphical object interfaces for controlling electronic device 1300. These graphical object interfaces can be composed of graphics, text, icons, video, and any combination thereof. Display unit 1340 may include display panel 1341, optionally configured as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or similar forms. Further, touch-sensitive surface 1331 may cover display panel 1341. When touch-sensitive surface 1331 detects a touch operation on or near it, it transmits the information to processor 1380 to determine the type of touch event. Subsequently, processor 1380 provides corresponding visual output on display panel 1341 according to the type of touch event. Although in FIG. 13, touch-sensitive surface 1331 and display panel 1341 are implemented as two separate components to realize input and output functions, in some embodiments, touch-sensitive surface 1331 and display panel 1341 can be integrated to realize input and output functions.

[0325] The electronic device 1300 may also include at least one sensor 1350, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1341 according to the ambient light level, and the proximity sensor can turn off the display panel 1341 or the backlight when the electronic device 1300 is moved to the ear. As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometers, taps), etc. Other sensors that the electronic device 1300 may be equipped with, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0326] Audio circuitry 1360, speaker 1361, and microphone 1362 provide an audio interface between the device and electronic device 1300. Audio circuitry 1360 converts received audio data into electrical signals and transmits them to speaker 1361, where speaker 1361 converts them into sound signals for output. Conversely, microphone 1362 converts collected sound signals into electrical signals, which are then received by audio circuitry 1360, converted back into audio data, processed by processor 1380, and transmitted via RF circuitry 1310 to another electronic device, or output to memory 1320 for further processing. Audio circuitry 1360 may also include an earphone jack to facilitate communication between external headphones and electronic device 1300.

[0327] The short-range wireless transmission module 1370 can be a WIFI (wireless fidelity) module, Bluetooth module, or infrared module, etc. The electronic device 1300 can transmit information with wireless transmission modules on other devices via the short-range wireless transmission module 1370.

[0328] Processor 1380 is the control center of electronic device 1300. It connects various parts of the device via various interfaces and lines, and performs various functions and processes data of electronic device 1300 by running or executing software programs or modules stored in memory 1320 and calling data stored in memory 1320, thereby providing overall control of the device. Optionally, processor 1380 may include one or more processing cores; optionally, processor 1380 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. In this embodiment, the modem processor may not be integrated into processor 1380.

[0329] Electronic device 1300 also includes a power supply 1390 (such as a battery) for supplying power to various components. Optionally, the power supply 1390 can be logically connected to the processor 1380 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. The power supply 1390 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0330] Although not shown, the electronic device 1300 may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0331] This application also discloses a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to implement the data synchronization method embodiment shown in FIG5.

[0332] In this application embodiment, the content of the data synchronization method embodiment shown in FIG5 is applicable to the computer-readable storage medium embodiment. The specific functions implemented by the computer-readable storage medium embodiment are the same as those of the data synchronization method embodiment shown in FIG5, and the beneficial effects achieved are also the same as those achieved by the data synchronization method embodiment shown in FIG5.

[0333] This application also discloses a computer program product or computer program, which includes computer instructions stored in the aforementioned computer-readable storage medium. The processor of the electronic device shown in FIG13 can read the computer instructions from the aforementioned computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the data synchronization method embodiment shown in FIG5.

[0334] In this application embodiment, the content of the data synchronization method embodiment shown in FIG5 is applicable to this computer program product or computer program embodiment. The specific functions implemented by this computer program product or computer program embodiment are the same as those of the data synchronization method embodiment shown in FIG5, and the beneficial effects achieved are also the same as those achieved by the data synchronization method embodiment shown in FIG5.

[0335] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0336] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. In addition to the embodiments of this application, a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, the actual implementation of a module will be understood within the conventional skills of an engineer, taking into account the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. In addition to the embodiments of this application, the specific concepts disclosed are merely illustrative and are not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0337] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0338] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0339] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0340] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0341] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0342] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0343] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A data synchronization method, performed by an electronic device, the method comprising: Obtain the target data to be sent to the data receiver; The network transmission status between the electronic device and the data receiving end is detected, and based on the network transmission status, it is determined whether the data synchronization mode corresponding to the data receiving end is real-time synchronization mode or snapshot synchronization mode. When the data synchronization mode is determined to be the real-time synchronization mode, the target data is immediately sent to the data receiving end; and, When the data synchronization mode is determined to be the snapshot synchronization mode, the pre-stored business processing logic of the data receiving end is queried, the target data is processed according to the business processing logic, and the snapshot data of the processing result is sent to the data receiving end.

2. The data synchronization method according to claim 1 further includes: The target data is converted into multiple log data entries, and the log index corresponding to each log data entry is determined.

3. The data synchronization method of claim 2, wherein, When the data synchronization mode is determined to be the real-time synchronization mode, the target data is immediately sent to the data receiving end, including: Obtain the system identifier of the data receiving end; Based on the log index, extract a first number of log data as the data to be synchronized in the current batch, and query the first log index corresponding to the first log data at the end of the first number of log data; The system identifier, the data to be synchronized in the current batch, and the first log index are sent to the data receiving end.

4. The data synchronization method of claim 3, wherein, The step of extracting a first number of log data as the data to be synchronized in the current batch according to the log index includes: Query the second log index most recently fed back by the data receiving end. The second log index indicates the second log data at the end of a batch of log data that the data receiving end has completed synchronization. Starting from the next log entry of the second log data, extract the first number of log data entries as the data to be synchronized in the current batch.

5. The data synchronization method according to claim 3 or 4, wherein, The first quantity is determined by the following steps: The system detects a second number of data receivers currently performing data synchronization, and a third number of data synchronization requests received from the data source during a first preset time period prior to the current time. The first quantity is determined based on the second quantity and the third quantity; The first quantity and the second quantity are negatively correlated, and the first quantity and the third quantity are negatively correlated.

6. The data synchronization method according to any one of claims 3-5, further comprising: Record the first moment when the current batch of data to be synchronized is sent to the data receiving end; When the second log index corresponding to the current batch is received from the data receiving end within the second preset time period after the first moment, the next batch of data to be synchronized is extracted according to the second log index corresponding to the current batch. If the second log index corresponding to the current batch is not received from the data receiving end within the second preset time period after the first moment, the data to be synchronized for the current batch is resent to the data receiving end.

7. The data synchronization method of any of claims 2-6, wherein, The step of processing the target data according to the business processing logic and sending a snapshot of the processing result to the data receiving end includes: Obtain the system identifier of the data receiving end; The log data is processed according to the business processing logic to obtain the processing result; When the amount of unsent processing results data reaches a preset capacity threshold, or when the time elapsed since the last snapshot data transmission reaches a preset time threshold, Query the third log index corresponding to the last log record currently being processed; Based on a preset snapshot data structure, the unsent processing results are packaged into snapshot data to be sent; The system identifier, the snapshot data to be sent, and the third log index are sent to the data receiving end.

8. The data synchronization method of any of claims 1-7, wherein, The detection of the network transmission status between the electronic device and the data receiving end includes: Detect the network environment parameters between the electronic device and the data receiving end; The step of determining whether the data synchronization mode corresponding to the data receiving end is real-time synchronization mode or snapshot synchronization mode based on the network transmission status includes: Based on the network environment parameters, determine the network quality score corresponding to the data receiving end; When the network quality score is greater than a preset score threshold, the data synchronization mode is determined to be the real-time synchronization mode; When the network quality score is less than or equal to the preset score threshold, the data synchronization mode is determined to be the snapshot synchronization mode.

9. The data synchronization method of any of claims 1-8, wherein, The electronic device is a distributed source data system, which includes a master node and multiple slave nodes. The process of obtaining the target data to be sent to the data receiver includes: The master node is used to obtain raw data, which is then converted to obtain standard data that conforms to a common data exchange format. The master node sends the standard data to each slave node, and the slave nodes reach a consensus on the standard data, determining the consensus-agreed standard data as the target data.

10. The data synchronization method of claim 9, wherein, The step of sending the standard data to each of the slave nodes through the master node, and achieving consensus on the standard data through the slave nodes, includes: The master node performs hash processing on the standard data to obtain a first hash value; the standard data and the first hash value are then sent to each of the slave nodes. The slave node performs hash processing on the standard data to obtain a second hash value, and verifies the consistency between the first hash value and the second hash value to obtain the consistency verification result corresponding to the standard data. When more than half of the slave nodes pass the consistency check, the standard data is determined to have passed consensus.

11. The data synchronization method of any of claims 1-10, wherein, The electronic device is a distributed source data system, which includes a master node and multiple slave nodes; the method further includes: Synchronize the data in the master node to each of the slave nodes; Detect the operating status of the distributed source data system; When it is determined from the operating status that the distributed source data system needs to switch master nodes, a new master node is determined from each of the slave nodes; Detect whether there are any incomplete data synchronization tasks in the old master node; wherein, the incomplete data synchronization task indicates: a log index that has been sent to synchronize data but has not been received from the data receiving end; When there are unfinished data synchronization tasks in the old master node, the sending of new data to be synchronized through the old master node is stopped. The system waits until the log index is received from the data receiving end or the timeout occurs, and then switches from the old master node to the new master node.

12. A data synchronization device, comprising: The acquisition unit is used to acquire the target data to be sent to the data receiving end. The detection unit is used to detect the network transmission status between the electronic device and the data receiving end, and determine whether the data synchronization mode corresponding to the data receiving end is real-time synchronization mode or snapshot synchronization mode based on the network transmission status. The first processing unit is configured to immediately send the target data to the data receiving end when it is determined that the data synchronization mode is the real-time synchronization mode. and, The second processing unit is used to query the pre-stored business processing logic of the data receiving end when the data synchronization mode is determined to be the snapshot synchronization mode, process the target data according to the business processing logic, and send the snapshot data of the processing result to the data receiving end.

13. The data synchronization device according to claim 12, further comprising: The conversion unit is used to convert the target data into multiple log data entries and determine the log index corresponding to each log data entry.

14. The data synchronization apparatus of claim 13, wherein, The first processing unit is configured to: obtain the system identifier of the data receiving end; extract a first number of log data as the data to be synchronized in the current batch according to the log index, and query the first log index corresponding to the first log data at the end of the first number of log data; and send the system identifier, the data to be synchronized in the current batch and the first log index to the data receiving end.

15. The data synchronization apparatus according to claim 13 or 14, wherein The second processing unit is used to: obtain the system identifier of the data receiving end; process the log data according to the business processing logic to obtain the processing result; when the amount of unsent processing result data reaches a preset capacity threshold, or the time elapsed since the last sent snapshot data reaches a preset time elapsed threshold, query the third log index corresponding to the last log data currently being processed; and package the unsent processing result into snapshot data to be sent based on a preset snapshot data structure. The system identifier, the snapshot data to be sent, and the third log index are sent to the data receiving end.

16. The data synchronization apparatus of any of claims 12-15, wherein, The detection unit is used to detect network environment parameters between the electronic device and the data receiving end; determine the network quality score corresponding to the data receiving end based on the network environment parameters; when the network quality score is greater than a preset score threshold, determine the data synchronization mode as the real-time synchronization mode; when the network quality score is less than or equal to the preset score threshold, determine the data synchronization mode as the snapshot synchronization mode.

17. The data synchronization apparatus of any of claims 12-16, wherein, The electronic device is a distributed source data system, which includes a master node and multiple slave nodes. The acquisition unit is used to acquire raw data through the master node, and to convert the raw data to obtain standard data that conforms to a common data exchange format. The master node sends the standard data to each slave node, and the slave nodes reach a consensus on the standard data, determining the consensus-agreed standard data as the target data.

18. An electronic device comprising a memory and a processor, the memory storing a computer program, wherein the processor, when executing the computer program, implements a data synchronization method according to any one of claims 1 to 11.

19. A computer-readable storage medium storing a computer program that, when executed by a processor, implements a data synchronization method according to any one of claims 1 to 11.

20. A computer program product comprising a computer program that, when executed by a processor, implements a data synchronization method according to any one of claims 1 to 11.

Images