Data processing method and system for distributed coordinator, and electronic device

Abstract

The present disclosure relates to the technical field of distributed coordination, and disclosed are a data processing method and system for a distributed coordinator, and an electronic device. The method comprises: in response to a data processing request, determining a service scenario in which a distributed coordinator operates; determining a storage level corresponding to the service scenario, wherein the storage level is used for representing a data granularity at which the distributed coordinator stores data, different data granularities correspond to different storage levels, and between two adjacent storage levels among the different storage levels, the data granularity corresponding to the lower storage level is coarser than the data granularity corresponding to the higher storage level; acquiring a storage strategy set on the storage level corresponding to the service scenario, wherein a data pool allows the data to be referenced in the distributed coordinator, and different service scenarios correspond to different storage strategies; and according to the storage strategy, using the data pool to store the data in a memory of the distributed coordinator.

Description

Data processing methods, systems, and electronic devices for distributed coordinators

[0001] Cross-reference

[0002] This disclosure claims priority to Chinese Patent Application No. 2025101917590, filed on February 20, 2025, entitled “Data Processing Method, System and Electronic Device for Distributed Coordinator”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of distributed coordination technology, and in particular to data processing methods, systems and electronic devices for distributed coordinators. Background Technology

[0004] Currently, in scenarios with a large number of nodes and watchers, distributed coordinators store a significant amount of node state data and watch data structures, resulting in substantial memory consumption. When the memory usage of the distributed coordinator reaches a certain threshold, garbage collection (GC) is frequently triggered to clean up unused objects.

[0005] However, frequent garbage collection (GC) can significantly increase the pause time of distributed systems, affecting the overall performance of distributed systems and resulting in the technical problem of low information processing efficiency for biological objects.

[0006] There is currently no effective solution to the above problems. Summary of the Invention

[0007] This disclosure provides a data processing method, system, and electronic device for a distributed coordinator to at least address the technical problem of low information processing efficiency for biological objects.

[0008] According to one aspect of the embodiments of this disclosure, a data processing method for a distributed coordinator is provided. The method includes: responding to a data processing request, determining a service scenario in which the distributed coordinator operates, wherein the data processing request includes data that the distributed coordinator needs to store during data processing, and the service scenario is the scenario to which a service supported by the distributed coordinator belongs; determining a storage level corresponding to the service scenario, wherein the storage level represents the data granularity at which the distributed coordinator stores data, different data granularities correspond to different storage levels, and in adjacent storage levels, the data granularity corresponding to the lower storage level is coarser than that corresponding to the higher storage level; obtaining a storage policy set at the storage level corresponding to the service scenario, wherein the storage policy represents the rules by which the distributed coordinator uses a data pool to store data, the data pool allows data to be referenced in the distributed coordinator, and different service scenarios correspond to different storage policies; and storing data in the memory of the distributed coordinator using the data pool according to the storage policy.

[0009] According to another aspect of the present disclosure, a data storage method for a distributed coordinator is provided. The method includes: responding to a data processing request from a client; determining the service scenario in which the distributed coordinator operates, wherein the data processing request includes data that the distributed coordinator needs to store during data processing, and the service scenario is the scenario to which a service supported by the distributed coordinator belongs; determining the storage level corresponding to the service scenario, wherein the storage level represents the data granularity at which the distributed coordinator stores data, different data granularities correspond to different storage levels, and in adjacent storage levels, the data granularity corresponding to the lower storage level is coarser than that corresponding to the higher storage level; obtaining a storage policy set at the storage level corresponding to the service scenario, wherein the storage policy represents the rules by which the distributed coordinator stores data using a constant pool, the constant pool allows data to be referenced in the distributed coordinator, and different service scenarios correspond to different storage policies; and caching data in the memory of the distributed coordinator using the constant pool according to the storage policy.

[0010] According to another aspect of the present disclosure, a data processing method for a distributed coordinator is provided. The method includes: obtaining a data processing request by calling a first interface, and responding to the data processing request to determine the service scenario in which the distributed coordinator runs, wherein the first interface includes a first parameter, the value of which includes the data processing request, the data processing request including the data that the distributed coordinator needs to store during data processing, and the service scenario being the scenario to which the services supported by the distributed coordinator belong; determining the storage level corresponding to the service scenario, wherein the storage level is used to represent the data granularity at which the distributed coordinator stores data, different data granularities correspond to different storage levels, and in two adjacent storage levels, the data granularity corresponding to the lower storage level is coarser than that corresponding to the higher storage level; obtaining the storage policy set on the storage level corresponding to the service scenario, wherein the storage policy is used to represent the rules by which the distributed coordinator uses a data pool to store data, the data pool allows data to be referenced in the distributed coordinator, and different service scenarios correspond to different storage policies; storing data in the memory of the distributed coordinator using the data pool according to the storage policy; and outputting the data stored in the data pool by calling a second interface, wherein the second interface includes a second parameter, the value of which includes the data stored in the data pool.

[0011] According to another aspect of the present disclosure, a data processing system for a distributed coordinator is provided. The system includes: a client and a distributed coordinator. The client is configured to send data processing requests, which include data that the distributed coordinator needs to store during data processing. The distributed coordinator, connected to the client, is configured to respond to the data processing requests by: determining the service scenario in which the distributed coordinator operates, where the data processing requests include data that the distributed coordinator needs to store during data processing, and the service scenario is the scenario to which the services supported by the distributed coordinator belong; determining the storage level corresponding to the service scenario, where the storage level represents the granularity of data storage by the distributed coordinator, with different granularities corresponding to different storage levels, and the lower storage level having a coarser granularity than the higher storage level; obtaining the storage policy set at the storage level corresponding to the service scenario, where the storage policy represents the rules by which the distributed coordinator uses a data pool to store data, allowing data to be referenced within the distributed coordinator, and different service scenarios corresponding to different storage policies; and storing the data in the distributed coordinator's memory using the data pool according to the storage policy.

[0012] According to another aspect of the present disclosure, a computing device is also provided, the computing device including a memory and a processor; the memory is used to store an executable program; the processor is used to run the program, wherein the program performs the method of any of the above when it is run.

[0013] According to another aspect of the present disclosure, an electronic device is also provided, the electronic device including a memory and a processor: the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein when the computer-executable instructions are executed by the processor, the method of any one of the above is implemented.

[0014] According to another aspect of the embodiments of this disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the storage medium is located to perform any of the methods described above.

[0015] According to another aspect of the present disclosure, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method described in any of the above.

[0016] In this embodiment of the disclosure, in response to a data processing request, the service scenario in which the distributed coordinator operates is determined. The data processing request includes data that the distributed coordinator needs to store during data processing. The service scenario is the scenario to which the services supported by the distributed coordinator belong. The storage level corresponding to the service scenario is determined. The storage level represents the data granularity at which the distributed coordinator stores data; different data granularities correspond to different storage levels. In adjacent storage levels, the data granularity of the lower storage level is coarser than that of the higher storage level. A storage policy set at the storage level corresponding to the service scenario is obtained. The storage policy represents the rules by which the distributed coordinator uses a data pool to store data. The data pool allows data to be referenced within the distributed coordinator, and different service scenarios correspond to different storage policies. Data is stored in the distributed coordinator's memory using the data pool according to the storage policy. In other words, this embodiment dynamically determines the service scenario in which the distributed coordinator operates and determines the corresponding storage level based on the service scenario. Multiple adjacent storage levels are progressively advanced to obtain the storage strategy set at the corresponding storage level. This allows for efficient use of the data pool in the memory of the distributed coordinator to store data, thereby reducing memory usage and improving the data storage performance of the distributed coordinator. This solves the technical problem of low information processing efficiency for biological objects.

[0017] It is worth noting that the above general description and the following detailed description are merely for illustrative and explanatory purposes and do not constitute a limitation thereof. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0019] Figure 1 is a schematic diagram of an application scenario of a data processing method for a distributed coordinator according to an embodiment of the present disclosure;

[0020] Figure 2 is a flowchart of a data processing method for a distributed coordinator according to an embodiment of the present disclosure;

[0021] Figure 3 is a flowchart of a data storage method for a distributed coordinator according to an embodiment of the present disclosure;

[0022] Figure 4 is a flowchart of another data processing method for a distributed coordinator according to an embodiment of the present disclosure;

[0023] Figure 5 is a schematic diagram of a data processing system of a distributed coordinator according to an embodiment of the present disclosure;

[0024] Figure 6 is a schematic diagram of a string data storage format according to related technologies;

[0025] Figure 7 is a schematic diagram of a storage format for string data based on a data pool according to an embodiment of the present disclosure;

[0026] Figure 8 is a schematic diagram of a storage format for storage path data according to related technologies;

[0027] Figure 9 is a schematic diagram of a storage format of storage path data based on the ZnodePath data structure according to an embodiment of the present disclosure;

[0028] Figure 10 is a schematic diagram of the overall architecture of a hierarchical memory optimization scheme based on a string pool according to an embodiment of the present disclosure;

[0029] Figure 11 is a schematic diagram of Old area memory usage according to related technologies;

[0030] Figure 12 is a schematic diagram of Old area memory usage according to an embodiment of the present disclosure;

[0031] Figure 13 is a schematic diagram of storing data based on a ZnodePath data structure according to an embodiment of the present disclosure;

[0032] Figure 14 is a schematic diagram of structured storage of registration information according to an embodiment of the present disclosure;

[0033] Figure 15 is a schematic diagram of another structured storage of registration information according to an embodiment of the present disclosure;

[0034] Figure 16 is a structural block diagram of a computing environment for a data processing method of a distributed coordinator according to an embodiment of the present disclosure;

[0035] Figure 17 is a schematic diagram of a data processing apparatus for a distributed coordinator according to an embodiment of the present disclosure;

[0036] Figure 18 is a schematic diagram of a data storage device for a distributed coordinator according to an embodiment of the present disclosure;

[0037] Figure 19 is a schematic diagram of a data processing apparatus for another distributed coordinator according to an embodiment of the present disclosure;

[0038] Figure 20 is a structural block diagram of a computing device according to an embodiment of the present disclosure;

[0039] Figure 21 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0041] The terms "first," "second," etc., used in this disclosure, claims, and accompanying drawings are for distinguishing similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0042] First, some nouns or terms that appear in the description of the embodiments of this disclosure shall be interpreted as follows:

[0043] A distributed coordinator is used to provide distributed configuration services, synchronization services, and naming registration for large-scale distributed computing. For example, the distributed coordinator can be Zookeeper. This is just an example and does not impose specific restrictions on the type of distributed coordinator.

[0044] A node is a data storage unit in a distributed coordinator. For example, a data storage unit in ZooKeeper can be represented as a Znode. A Znode contains metadata such as the data content and time stored in ZooKeeper.

[0045] Memory, also known as Random Access Memory (RAM), is a temporary data storage area in a computer used to temporarily store running programs, the operating system, and data being processed. Memory is characterized by its fast read and write speeds, but data is lost when power is off.

[0046] Garbage collection (GC) is an automatic memory management mechanism. When a program no longer accesses a portion of the memory space it occupies, the program will return that portion of the memory space to the operating system using a garbage collection algorithm.

[0047] A weak reference is a type of weak reference provided by the Java Development Kit (JDK). A weak reference does not prevent the garbage collector (GC) from reclaiming an object. When an object is left with only a weak reference, the GC will reclaim the object.

[0048] A WeakHashMap uses a WeakReference as the key. When the key is garbage collected, the corresponding key-value (KV) pair will be deleted from the map.

[0049] The Heap is a memory region within the Java Virtual Machine (JVM) where objects are allocated memory.

[0050] The data processing method for a distributed coordinator provided in this disclosure can be applied to the application scenario shown in Figure 1, but is not limited thereto. Figure 1 is a schematic diagram of an application scenario of a data processing method for a distributed coordinator according to an embodiment of this disclosure. In the application scenario shown in Figure 1, the server 10 can be in the cloud. The server 10 can connect to one or more client devices 20 through a local area network (LAN), a wide area network (WAN), an internet connection, or other types of data networks. The client devices 20 can include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. These client devices collectively constitute the client relative to the server. An operation interface for obtaining the service scenario running by the distributed coordinator can be deployed on the graphical user interface of the client device. The client device 20 can interact with the user through the graphical user interface to implement the data processing method for the distributed coordinator provided in this disclosure.

[0051] In this embodiment of the disclosure, the system comprising client device 20 and server 10 can perform the following steps: Performing corresponding operations on the user interface of client device 20 to obtain the service scenario running by the distributed coordinator. The client device can obtain the service scenario running by the distributed coordinator and send it to the server via the network. After receiving the service scenario run by the distributed coordinator, the server can execute the following steps: Step S102, in response to the data processing request, determine the service scenario run by the distributed coordinator, wherein the data processing request includes the data that the distributed coordinator needs to store during data processing, and the service scenario is the scenario to which the service supported by the distributed coordinator belongs; Step S104, determine the storage level corresponding to the service scenario, wherein the storage level is used to represent the data granularity of the distributed coordinator in storing data, different data granularities correspond to different storage levels, and in two adjacent storage levels, the data granularity corresponding to the lower storage level is coarser than that corresponding to the higher storage level; Step S106, obtain the storage policy set on the storage level corresponding to the service scenario, wherein the storage policy is used to represent the rules by which the distributed coordinator uses a data pool to store data, the data pool allows data to be referenced in the distributed coordinator, and different service scenarios correspond to different storage policies; Step S108, according to the storage policy, store the data in the memory of the distributed coordinator using the data pool.

[0052] In the above operating environment, this disclosure provides a data processing method for a distributed coordinator as shown in Figure 2. Figure 2 is a flowchart of a data processing method for a distributed coordinator according to an embodiment of this disclosure. As shown in Figure 2, the method includes the following steps:

[0053] Step S202: In response to the data processing request, determine the service scenario in which the distributed coordinator is running.

[0054] In the technical solution provided in step S202 of this disclosure, the service scenario in which the distributed coordinator operates can be determined in response to a data processing request. The data processing request includes data that the distributed coordinator needs to store during data processing. For example, this data can be a string, a path, etc. This is merely an example and does not impose specific restrictions on the type of data to be stored. The data processing request can be a request initiated by a client to the distributed coordinator, used to trigger the distributed coordinator to perform data storage, listening, and updating operations. For example, the data processing request can be a write request, a SetWatch request, etc. This is merely an example and does not impose specific restrictions on the content of the data processing request.

[0055] In this embodiment, the distributed coordinator can be a component that provides synchronization, communication, and management services in a distributed system. The distributed coordinator is used to store data, manage data change events, and push change events to clients. The distributed coordinator stores data in memory and manages the data structures that clients listen to. For example, the distributed coordinator can be Zookeeper.

[0056] In this embodiment, the service scenario can be the scenario to which the service supported by the distributed coordinator belongs, or it can be a general service scenario, a service governance scenario, etc. Among them, the general service scenario can be a microservice scenario, a distributed coordination scenario, etc., and the service governance scenario can be a microservice governance scenario. This is only an example for illustration and no specific restrictions are placed on the type of service scenario.

[0057] Optionally, when a client sends a write request or a SetWatch request to ZooKeeper, ZooKeeper can determine whether the service scenario is a microservice scenario, a distributed coordination scenario, or a more complex microservice governance scenario based on the content and type of the received data processing request. Based on the determined service scenario, ZooKeeper can apply specified storage strategies and data structures to respond to data processing requests more efficiently and manage memory, thereby better meeting the needs of the service scenario.

[0058] Step S204: Determine the storage level corresponding to the service scenario.

[0059] In the technical solution provided in step S204 of this disclosure, after determining the service scenario in which the distributed coordinator operates in response to a data processing request, the storage level corresponding to that service scenario can be determined. The storage level can represent the granularity of data storage by the distributed coordinator. For example, the storage level can be one layer, two layers, three layers, etc. This is only an example and does not impose a specific limitation on the number of storage levels. Different data granularities correspond to different storage levels. In adjacent storage levels, the data granularity corresponding to the lower storage level is coarser than that corresponding to the higher storage level. For example, the data granularity corresponding to the first storage level is coarser than that corresponding to the second storage level, and the data granularity corresponding to the second storage level is coarser than that corresponding to the third storage level.

[0060] Optionally, different service scenarios can correspond to different storage tiers. For example, the storage tier for general service scenarios can be tier one or tier two, while the storage tier for more complex service governance scenarios can be tier three. This is just an example and does not impose specific restrictions on the storage tier for service scenarios.

[0061] Step S206: Obtain the storage policy set at the storage level corresponding to the service scenario.

[0062] In the technical solution provided in step S206 of this disclosure, after determining the storage level corresponding to the service scenario, the storage strategy set at the storage level corresponding to the service scenario is obtained. The storage strategy represents the rules by which the distributed coordinator stores data using a data pool; different service scenarios correspond to different storage strategies. The data pool is a memory area or structure for managing and storing data. For example, the data pool may be a string pool (StringPool), also known as a string constant pool or string pool. The data pool allows data to be referenced in the distributed coordinator.

[0063] In this embodiment, after determining the storage layer corresponding to the service scenario, the storage strategy set at the storage layer corresponding to different service scenarios is obtained. For example, the storage strategy set at the first-level storage layer is string pooling, the storage strategy set at the second-level storage layer is hierarchical data pooling, for example, managing the common part and the custom part of the Znode path separately, and the storage strategy set at the third-level storage layer is structured data storage and metadata pooling, for example, distinguishing between general metadata and custom metadata storage strategies.

[0064] For example, when the identified service scenario is a microservice scenario or a distributed coordination scenario, since the corresponding storage layer for this service scenario is either Tier 1 or Tier 2, the storage strategy set at Tier 1 or Tier 2 can be determined. For instance, the storage strategy set at Tier 1 might be to use a string pool to manage duplicate string data, while the storage strategy set at Tier 2 might be to perform hierarchical splitting and pooling of Znode paths.

[0065] For another example, when the identified service scenario is a microservice governance scenario, since the corresponding storage layer for this service scenario can be three layers, the storage strategy set on the three storage layers can be determined. For instance, the storage strategy set on the three storage layers could be to store metadata in a structured manner.

[0066] Optionally, storage strategies suitable for different service scenarios can be determined based on data access patterns, data types, and data lifecycles in different service scenarios.

[0067] For example, in microservice or distributed coordination scenarios, operations such as service discovery, configuration management, and state synchronization occur frequently. Data access patterns are typically read-heavy and write-light, and data structures are relatively fixed, such as paths and metadata. The main data types in this service scenario can include strings and integers. String data may have significant duplication across different service instances, while integer data is usually a fixed value within a small range. In this scenario, Znode paths and metadata are typically created during service registration and remain unchanged throughout the service instance's lifecycle; therefore, this data is persistent and suitable for long-term resident in memory.

[0068] Optionally, based on the data access patterns, data types, and data lifecycles in the aforementioned microservice or distributed coordination scenarios, the storage strategy set at the storage layer corresponding to this service scenario can be determined as follows: use a string pool to store Znode paths and frequently occurring strings to reduce the repeated storage of the same strings in memory; introduce a weak reference mechanism to ensure that when a string is no longer referenced by any client, it is automatically reclaimed by the JVM's garbage collection mechanism to avoid memory leaks; and use a fixed-size data type to store numeric data such as ports to reduce the additional memory overhead caused by string representation.

[0069] For another example, in microservice governance scenarios, data access is more diverse and dynamic, requiring frequent updates and queries of metadata to support real-time policy adjustments. Besides standard string and numeric types, microservice governance involves more complex data types, such as booleans, enumerations, and arrays. In this service scenario, metadata changes throughout the service instance's lifecycle, and some metadata, such as availability zone information and service version numbers, exhibits high repetition in large-scale deployments.

[0070] Optionally, based on the data access patterns, data types, and data lifecycles in the microservice governance scenario described above, the storage strategy set at the storage layer corresponding to this service scenario is determined as follows: metadata is stored in a structured manner, distinguishing between fixed metadata and variable metadata, and reducing unnecessary hashing and reference overhead; for highly repetitive metadata, a mechanism similar to a string constant pool is adopted to ensure that there is only one copy of the same metadata in memory, and instances share the same reference.

[0071] Step S208: According to the storage strategy, the data is stored in the memory of the distributed coordinator using a data pool.

[0072] In the technical solution provided by step S208 of this disclosure, after obtaining the storage policy corresponding to the service scenario, data is stored in the memory of the distributed coordinator using a data pool according to the obtained storage policy.

[0073] In this embodiment, according to the storage strategy, data is stored in a data pool within ZooKeeper's memory management, and references to this data are allowed within ZooKeeper. A shared data storage area, the data pool, is pre-created and managed to store data that is reused or frequently accessed in multiple locations. Storing data in this data pool allows a single instance of data to be stored in memory and accessed by reference rather than copying when needed, thus saving memory resources and improving data access efficiency.

[0074] Optionally, the data in the data pool is not static. For data that can be modified, when the data is updated, the data pool will create a new data instance. When the old instance no longer has references pointing to it, it will be cleaned up by the garbage collection mechanism. Thus, the data pool can not only ensure efficient data storage, but also maintain data consistency through the reference update mechanism when the data changes.

[0075] This embodiment utilizes a data pool to store data in the distributed coordinator and allows data to be referenced within the distributed coordinator. By sharing a single instance of data and using a reference access mechanism, it effectively reduces memory usage, improves data access efficiency, and ensures data consistency.

[0076] By using steps S202 to S208 of this disclosure, the service scenario in which the distributed coordinator operates is dynamically determined, and the corresponding storage level is determined according to the service scenario. Multiple adjacent storage levels are progressively advanced (or progressively layered), thereby obtaining the storage strategy set at the corresponding storage level. This allows for efficient use of the data pool to store data in the memory of the distributed coordinator, thereby reducing memory usage and improving the data storage performance of the distributed coordinator. This solves the technical problem of low information processing efficiency for biological objects.

[0077] The method described in this embodiment will be further described below.

[0078] As an optional implementation, step S206, obtaining the storage policy set at the storage level corresponding to the service scenario, includes: obtaining the storage policy set at the storage level corresponding to the service scenario from the storage policy set, wherein different storage policies corresponding to different storage levels in the storage policy set are interconnected.

[0079] In this embodiment, after determining the storage tier corresponding to the service scenario, the storage policies set at the storage tier corresponding to the service scenario are obtained from the storage policy set. The storage policy set is a collection of predefined storage policies that can be used to guide how data is stored. The storage policy set includes multiple storage policies, and different storage policies corresponding to different storage tiers are interconnected.

[0080] Optionally, the storage strategies corresponding to different storage levels can be progressive. For example, the storage strategy corresponding to the first storage level is called the first storage strategy, the storage strategy corresponding to the second storage level is called the second storage strategy, and the storage strategy corresponding to the third storage level is called the third storage strategy. The first storage strategy is a basic storage strategy, such as using a weak reference string pool to manage static strings to reduce the memory consumption of useless objects. The second storage strategy introduces more complex strategies based on the first storage strategy, such as using hierarchical data structures. The third storage strategy is a deep optimization based on the second storage strategy for complex service scenarios, such as structured storage of registration information in service governance scenarios. This hierarchical and progressive storage strategy allows distributed systems to achieve more efficient and granular resource management when handling different service scenarios.

[0081] After obtaining the service scenario, this embodiment dynamically selects and applies the storage strategy set at the corresponding storage level from the storage strategy set according to the specific needs of the service scenario, so as to adapt to different service scenarios, thereby reducing unnecessary memory and storage resource consumption and reducing operating costs.

[0082] As an optional implementation, step S202, in response to the data processing request, determines the service scenario in which the distributed coordinator runs, including: in response to the data processing request, determining the general service scenario or service governance scenario in which the distributed coordinator runs, wherein the service governance scenario is an extension of the general service scenario.

[0083] In this embodiment, in response to a data processing request, the general service scenario or service governance scenario in which the distributed coordinator operates is determined. The general service scenario refers to the conventional pattern of service interaction over a network in a distributed system, such as a microservice scenario or a distributed coordination scenario. A microservice scenario is one where a complex application is decomposed into a series of small, independent services. The distributed coordination scenario focuses on how to achieve cooperation and consistency between nodes in a distributed system to ensure the correctness of the global state.

[0084] Optionally, service governance scenarios primarily focus on higher-level service management functions within a microservice architecture. They can be seen as an extension and deepening of general service scenarios, emphasizing advanced control and management of services. General service scenarios and service governance scenarios together constitute the service interaction and management framework under a microservice architecture. Besides service governance scenarios, distributed service interaction can be categorized as a general service scenario. In practical applications, many services may simultaneously possess characteristics of both general service scenarios and service governance scenarios. For example, a service might use Zookeeper for service discovery and state coordination (general service scenario) while also utilizing the canary release functionality of a service governance platform (service governance scenario).

[0085] For example, service governance scenarios include microservice governance scenarios, which include, but are not limited to, end-to-end canary releases, tag-based routing, same-availability-zone priority routing, and service preheating. End-to-end canary releases are used in microservice governance to control the scope of new service feature or version releases, allowing new features or versions to be gradually pushed to a subset of users or traffic without affecting overall service stability. Tag-based routing is a scenario that selects routes based on service instance or request tags. Tag-based routing allows traffic to be routed to service instances that match the request's tag, thereby achieving finer-grained service control and optimization. Same-availability-zone priority routing focuses on the physical location of service instances, especially in multi-datacenter or multi-availability-zone deployments, to prioritize routing requests to service instances geographically close to the client, reducing network latency and improving service response speed and user experience. Service preheating involves loading data in advance, preheating caches, and initializing dependencies after the service instance starts, allowing the service to reach a stable state as quickly as possible to reduce startup latency.

[0086] As an optional implementation, the data includes string data. Step S208 involves storing the data in the memory of the distributed coordinator using a data pool according to a storage strategy. This includes: in a general service scenario, performing the following steps in the memory of the distributed coordinator according to a storage strategy: pooling multiple string data with the same value using the data pool, wherein the pooled string data can be referenced by multiple reference objects in the distributed coordinator; and caching the pooled string data in the data pool.

[0087] Optionally, ZooKeeper manages the data stored in ZooKeeper and the Watches on clients through a Map structure. A common service scenario for ZooKeeper is that some clients update data in ZooKeeper, while others listen for these changes through Watches. When a client sends a write request to ZooKeeper, ZooKeeper inserts data into the Map structure where the data is stored. When a client sends a SetWatch request to ZooKeeper, ZooKeeper inserts data into the Map structure managed by the Watches. For the same Znode, the key stored in the data storage Map and the key stored in the Watch-managed Map are two separate objects in memory; that is, the same string constant has multiple redundant objects in memory.

[0088] As mentioned above, in ZooKeeper, multiple string data may have the same value. If these string data are stored independently, it will cause a large amount of memory redundancy. Figure 6 is a schematic diagram of a string data storage format according to related technologies. As shown in Figure 6, the client sends an AddWatchRequest to ZooKeeper to add an event listener (Watcher). When the state or data of a Znode changes, ZooKeeper will trigger the Watcher and notify the client that sent the AddWatchRequest. In addition, the client sends a CreateRequest to ZooKeeper to create a new Znode in ZooKeeper. As can be seen from Figure 6, the reference path of both AddWatchRequest and CreateRequest is " / test". The WatchManager is the component in ZooKeeper that manages Watchers. When the state or data of a Znode changes, the WatchManager will trigger the corresponding event notification according to the list of registered Watchers. The DataTree is an internal data structure in ZooKeeper used to store Znode data. Each Znode is stored in a DataTree, and each node in the DataTree contains information such as the Znode's data, state, and list of child nodes. "Bytes Path: / test" can be the byte representation of the string " / test" in memory. As shown in Figure 6, the "Bytes Path: / test" for both AddWatchRequest and CreateRequest is stored in the JVM Heap. This means that for the same immutable data, two copies are stored, resulting in increased space consumption.

[0089] To address the issue of storing two copies of the same immutable data, which leads to excessive space usage, a data pool can be used to pool multiple strings with the same value. This pooling process determines whether a value identical to the string in the new request exists in the data pool. If it does, a new object is not created; instead, the existing object is referenced. If it does not exist, the string data is created and stored in the data pool for subsequent requests to share.

[0090] Optionally, pooled string data can be referenced by multiple reference objects within the distributed coordinator. For example, the unique instance of pooled string data in memory can be referenced by multiple requests or objects through a weak reference (WeakReference). That is, when a client registers a service or adds a listener to the distributed coordinator, if the string data such as the service path or metadata already exists in the data pool, the client's reference will directly point to that string in the data pool, rather than creating a new object.

[0091] Optionally, Figure 7 is a schematic diagram of a data pool-based string data storage format according to an embodiment of the present disclosure. As can be seen from the left half of Figure 7, for the reference path "Ref Path: / test" of AddWatchRequest and CreateRequest, the data "Bytes Path: / test" in Zookeeper is processed and stored in the string data pool StringPool through a weak reference "WeakRefPath: / test", thereby avoiding the problem of storing two copies of the same immutable data, which leads to space occupation.

[0092] Optionally, after pooling multiple string data with the same value using a data pool, the pooled string data is cached in the data pool. The pooled string data is cached in a WeakHashMap data structure, which supports weak references and automatically releases objects that are no longer referenced when memory pressure is high. The data pool's caching mechanism ensures that string data with the same value are stored in memory only once, reducing memory usage and improving data processing efficiency.

[0093] In a general service scenario, this embodiment uses a string constant pool to pool string objects with the same value, allowing these strings to be reused in multiple references, thereby significantly improving the memory utilization efficiency of the distributed coordinator and reducing memory consumption.

[0094] As an optional implementation, the pooled string data exists uniquely in the data pool and can be accessed by the referenced object in the data pool through a weak reference strategy. The weak reference strategy is used to represent the rules for weakly referencing the pooled string data. The method also includes: in a general service scenario, in response to the absence of a referenced object in the distributed coordinator, performing a recycling operation on the pooled string data.

[0095] In this embodiment, because a data pool is used to pool multiple string data with the same value, when a value identical to the string in a new request exists in the data pool, a new object is not created; instead, an existing object is directly referenced. This ensures that the pooled string data is unique within the data pool. Furthermore, the pooled string data can be accessed by the referenced object within the data pool using a weak reference strategy. The weak reference strategy represents the rules for weakly referencing the pooled string data.

[0096] In this embodiment, after determining that the service scenario running on the distributed coordinator is a general service scenario, in response to the absence of referenced objects in the distributed coordinator, the pooled string data is reclaimed. Since the pooled string data allows multiple requests or referenced objects to be referenced in the data pool via weak references (WeakReference), that is, when the string data already exists in the data pool, the client's reference will directly point to that string data in the data pool, rather than creating a new object. This referencing method effectively reduces memory consumption. Furthermore, because the reference is weak, when no strong references point to data in the data pool, the GC mechanism automatically reclaims unused data, further saving memory.

[0097] In this embodiment, the Path will be automatically reclaimed by the GC when it has no references, thereby achieving automatic memory cleanup. By automatically reclaiming unreferenced string data, the memory usage efficiency and performance of the distributed coordinator can be significantly improved.

[0098] As an optional implementation, the data includes storage path data. The method further includes: in a general service scenario, splitting the storage path data to obtain multiple string data located at different path levels; step S208, storing the data in the memory of the distributed coordinator using a data pool according to a storage strategy, including: performing the following steps in the memory of the distributed coordinator according to the storage strategy: using the data pool to perform pooling operations on multiple string data at different path levels, wherein the pooled string data allows multiple reference objects to reference it in the distributed coordinator; caching the pooled string data in the data pool.

[0099] Optionally, the data stored in ZooKeeper is usually multi-level, and the Znode's path is stored in memory at multiple locations with different granularities. For the Znode itself, it is stored in memory in the form of a complete Path->Znode using a Map (i.e., ZnodeMap). For the parent node to which the Znode belongs, the string of the last level of the Znode is stored as the name of the current Znode in the parent node's child node list. Figure 8 is a schematic diagram of a storage format for path data according to related technologies. As shown in Figure 8, the storage format of the path data " / test / current" in memory is as follows: storing the value corresponding to " / test": hello, the name of the child node (children): ["current"], and storing the value corresponding to " / test / current": hello, current, the name of the children: [].

[0100] In this embodiment, the data includes storage path data. After determining that the service scenario running by the distributed coordinator is a general service scenario, the storage path data is split within this general service scenario to obtain multiple string data located at different path levels. Optionally, as shown in Figure 8, the strings "current" and "test" are redundantly stored. Therefore, this embodiment splits the Znode's Path according to its level to obtain multiple string data located at different path levels.

[0101] In this embodiment, after splitting the storage path data to obtain multiple string data at different path levels, a data pool is used to pool the multiple string data at different path levels, and the pooled string data is further cached in the data pool. The pooled string data can be referenced by multiple reference objects in the distributed coordinator.

[0102] Optionally, after splitting the storage path data into different levels and treating the data at each level as independent strings to reduce memory redundancy and improve data utilization, a data pool is used to pool multiple string data at different path levels. By pooling the split string data, string data with the same value at different path levels are stored in memory only once, while multiple references at different path levels can share a single instance, thus significantly reducing memory usage.

[0103] Optionally, the pooled string data is referenced by multiple reference objects through weak references. Therefore, when a path level is no longer needed (i.e., no strong references point to the string data at that level), the string data corresponding to that level is marked as reclaimable, which allows the data pool to automatically adjust its size and dynamically release unused memory space according to the actual needs of the distributed system.

[0104] This embodiment can significantly improve the memory usage efficiency of the distributed coordinator and speed up data access by splitting and pooling the storage path data and combining weak references and automatic recycling mechanisms.

[0105] As an optional implementation, a data pool is used to perform pooling operations on multiple string data at different path levels, including: associating multiple string data according to different path levels to obtain association results; using a data pool to perform pooling operations on the string data in the association results, wherein the pooled association results can be referenced by multiple reference objects in the distributed coordinator; and caching the pooled association results in the data pool.

[0106] In this embodiment, after splitting the storage path data to obtain multiple string data located at different path levels, the multiple string data at different path levels are associated according to the data structure to obtain the association result.

[0107] Optionally, multiple string data at different path levels are associated based on the data structure (ZnodePath data structure) representing the hierarchical characteristics of Znode paths in ZooKeeper. The ZnodePath data structure is used to manage and represent Znode path information more efficiently. Figure 9 is a schematic diagram of a storage format for path data based on the ZnodePath data structure according to an embodiment of this disclosure. As shown in Figure 9, the level name vector (levelsNameVec) is used to store the names of each level in the path. For the stored path data " / test / current", we can see that the path level name stored in the levelsNameVec of the upper ZnodePath is "test", the corresponding value is "hello", and the corresponding children is "current". The path level names stored in the levelsNameVec of the lower ZnodePath are "test" and "current", the corresponding values ​​are "hello" and "current", and the corresponding children are []. Further, the names corresponding to each path level are stored in the Heap, realizing the connection of the split levels through the ZnodePath data structure.

[0108] In this embodiment, after associating multiple string data to obtain the association result, a data pool is used to perform a pooling operation on the string data in the association result. Optionally, for string data at each level, if it already exists in the data pool, the existing data in the data pool is directly referenced instead of creating a new string instance in memory, thereby reducing the number of string objects in memory, especially for paths with the same prefix or repeated parts.

[0109] Optionally, string data in the data pool is managed using weak references (WeakReference). Weak references allow the JVM's garbage collection mechanism (GC) to automatically reclaim data when there are no strong references pointing to it, thus avoiding memory leaks and ensuring automatic cleanup of the data pool. In other words, when a level in the ZnodePath is no longer referenced by any object, the string data at that level can be garbage collected to release memory resources.

[0110] This implementation uses the ZnodePath data structure to associate strings after splitting the hierarchy and references data in the string pool, avoiding duplicate storage of the same strings in the path hierarchy and significantly reducing memory usage. In high-concurrency, large-scale Znode scenarios, memory consumption can be greatly reduced.

[0111] As an optional implementation, in a general service scenario, the storage path data is split to obtain multiple string data located at different path levels, including: in a general service scenario, determining the tree-like hierarchical structure of the storage path data; and splitting the storage path data according to the tree-like hierarchical structure to obtain string data located at different path levels.

[0112] In this embodiment, the stored path data can be viewed as a tree structure, i.e., a hierarchical tree structure. Each Znode (a node in the tree) in this hierarchical tree structure represents a level in the path. The root node can be used to represent the starting point of the path (usually " / "), and the child nodes are used to represent the next level in the path. In a general service scenario, the stored path data is split according to the hierarchical tree structure to obtain string data located at different path levels.

[0113] For example, by analyzing the hierarchical separators (such as " / ") in the storage path data " / test / current", we can determine that the hierarchical tree structure of the storage path data is: " / test" is the parent node, " / current" is the child node, and so on. Based on this hierarchical tree structure, " / test / current" can be split into the strings "test" and "current", with each substring representing a level.

[0114] Optionally, after splitting the storage path data according to the hierarchical tree structure of the storage path data, string data at different levels can be obtained. String data at different path levels can be managed and cached independently. The ZnodePath data structure can be used to associate string data at different path levels, and multiple reference objects can be allowed to reference them through the string pool. This avoids the duplication of storing strings at the same level, thereby saving memory.

[0115] This embodiment optimizes the storage method of path data in Zookeeper by determining the hierarchical tree structure of the storage path data, splitting and pooling the storage path data, avoiding the repeated storage of string data at the same level, reducing memory consumption, and is suitable for high concurrency and large-scale data processing in general service scenarios.

[0116] As an optional implementation, the data includes service registration information. The method further includes: converting the service registration information into structured data; determining fixed metadata from the structured data, wherein the fixed metadata is used to represent metadata that can be shared by multiple service instances of the distributed coordinator; step S208, storing the data in the memory of the distributed coordinator using a data pool according to a storage strategy, including: in a service governance scenario, performing the following steps in the memory of the distributed coordinator according to a storage strategy: using the data pool to perform a pooling operation on the fixed metadata, wherein the pooled fixed metadata allows to be referenced by multiple reference objects in the distributed coordinator; caching the pooled fixed metadata in the data pool.

[0117] In this embodiment, the data may include service registration information, which can be detailed information about a service instance (i.e., a running service node) registering its existence and providing services with the distributed coordinator in a distributed system. The service registration information can be converted into structured data, which can be data that stores the service registration information in a structured manner. After converting the service registration information into structured data, common metadata can be determined from the structured data. Common metadata can be used to represent metadata that allows multiple service instances of the distributed coordinator to share; for example, common metadata can be the Internet Protocol (IP) of the registered node, the port of the registered node, the interface name, etc.

[0118] For example, in a service governance scenario, a service instance (such as Dubbo) writes relevant information about its registered instance to ZooKeeper. The format of this service registration information can be " / dubbo / {interfaceName} / providers / dubbo: / / {ip}:{port} / {interfaceName}?{meta}". Here, `interfaceName` can represent the Dubbo interface name, `ip` can represent the IP address of the registered node, `port` can represent the port number of the registered node, and `meta` can represent the metadata of the registered node.

[0119] Optionally, in service governance scenarios, metadata representations are added to nodes during service registration. For service governance, the metadata added to the Uniform Resource Locator (URL) of services registered to the registry center has relatively high redundancy. Even in large-scale applications, the metadata on different nodes is generally within an enumerable set. For example, in priority routing within the same availability zone, the metadata redundancy increases with the number of nodes. After pooling constant strings (e.g., dubbo, providers, interfaceName), only one object exists in memory. However, in service governance scenarios, service registration information contains more redundant constant strings, the metadata of registered instance information has significant overlap, with many duplicate keys and values, and the IP and port of registered instances are enumerable and coded values ​​within a range.

[0120] Alternatively, to address the issue of redundant constant strings in the aforementioned service governance scenario, the registered data can be stored in a structured manner and referenced by a string pool to further reduce Zookeeper's memory usage in this service governance scenario.

[0121] For example, service registration information can be converted into structured data as follows using the instance data structure: Public class Instance{private final Integer ip;private final Integer port;private final String interfaceName;private final String[]meta}.

[0122] In this embodiment, since the metadata of registered instances can be divided into fixed metadata and variable metadata, the fixed metadata can be determined from the structured data. Fixed metadata consists of common metadata shared by instances registered with the same service. This common metadata can be extracted and stored uniformly to further reduce the length of the arrays required to store the metadata. A data pool can be used to pool the fixed metadata. Furthermore, the pooled fixed metadata can be cached in the data pool. The pooled fixed metadata can be referenced by multiple reference objects in the distributed coordinator.

[0123] Optionally, a data pool can be used to pool fixed metadata. This means the data pool can be used to store shared metadata for service instances. This storage method reduces the duplicate storage of the same data in memory. By referencing data in the data pool instead of storing the same data separately in each service instance, memory space is significantly saved. Weak references can be used to manage objects in the data pool. When objects in the data pool are no longer referenced by any external source, the JVM's garbage collection mechanism will automatically reclaim these objects, preventing memory leaks.

[0124] This embodiment decomposes service registration information and stores it in a structured manner using the Instance data structure, which can significantly reduce data redundancy in memory. For example, storing immutable IP addresses and port numbers as integers, instead of complete strings, can save a significant amount of memory space. Structured data storage makes data retrieval and manipulation more efficient. For instance, by directly accessing the properties of the Instance object, information such as the service's IP address and port can be quickly obtained without string parsing or complex traversal searches, thereby significantly improving the efficiency of service discovery and load balancing.

[0125] As an optional implementation, the method further includes: determining variable metadata from structured data, wherein the variable metadata is used to represent the metadata that is allowed to change corresponding to the distributed coordinator; and storing the variable metadata using a string array.

[0126] In this embodiment, after converting service registration information into structured data, variable metadata can be determined from the structured data. Variable metadata (customMeta) is further stored using a string array. This variable metadata can represent the metadata that is allowed to change corresponding to the distributed coordinator. Variable metadata can be information that changes over time or with the operating environment for each service instance. For example, variable metadata can be the health status, weight, or custom parameters of a service instance. This is only an example and no specific restrictions are placed on the type of variable metadata.

[0127] Optionally, since variable metadata changes at runtime, a fixed-length string array (String[]) is used to store it to provide some flexibility. In the Instance data structure, the customMeta field is typically defined as a string array to store variable metadata unique to each service instance. By storing variable metadata in a string array instead of using a mapping structure within the Instance, more efficient data storage and access can be achieved. This is because mapping structures internally use hash tables, requiring additional memory to store hash codes and references, as well as to handle hash collisions. String arrays, on the other hand, directly store metadata without additional hash operations, reducing memory overhead and computational costs during access.

[0128] This embodiment uses string arrays to store variable metadata. Since the array's memory usage is contiguous and its size is fixed, space can be pre-allocated based on worst-case scenarios, thus avoiding frequent memory allocation and deallocation. Furthermore, since the variable metadata of a service instance is usually not excessive, a fixed-size array is generally sufficient, reducing the overhead of dynamically adjusting the array size and further optimizing memory efficiency.

[0129] As an optional implementation, the method further includes: obtaining monitoring metrics of the distributed coordinator in the service scenario, wherein the monitoring metrics are used to represent the memory-related performance parameters in the distributed coordinator; and determining the memory metrics of the distributed coordinator based on the monitoring metrics, wherein the memory metrics are used to characterize the memory usage status of the distributed coordinator.

[0130] In this embodiment, monitoring metrics of the distributed coordinator in a service scenario can be obtained. These metrics can represent memory-related performance parameters of the distributed coordinator. For example, monitoring metrics can be the number of Watches on a node, the number of connections between cluster nodes, the number of Znodes in the cluster, the average length of the Watch path, etc. These are merely illustrative examples, and no specific limitations are imposed on the content of the monitoring metrics. Further, based on the monitoring metrics, memory metrics of the distributed coordinator can be determined. These memory metrics can characterize the memory usage status of the distributed coordinator. For example, a memory metric can be the average size of the ZooKeeper Old Generation. The ZooKeeper Old Generation can be the memory area occupied by relatively persistent or long-lived objects in ZooKeeper's internal data structures or memory management. This is also merely an example, and no specific limitations are imposed on the content of the memory metrics.

[0131] Optionally, the memory model of Zookeeper can be quantified by introducing a mathematical model. Based on monitoring metrics, Zookeeper's memory metrics, namely the average size of the Old Generation, can be determined. The average size of the Old Generation can be used to effectively measure Zookeeper's memory usage status.

[0132] This embodiment introduces a mathematical model to quantify Zookeeper's memory usage, especially the occupancy of the Old Generation. This can significantly improve the system's resource management efficiency, enabling operations personnel or system developers to accurately predict memory requirements based on specific monitoring indicators (such as the number of connections, the number of Znodes, the number of Watches, etc.), thereby aiding in resource planning.

[0133] As an optional implementation, the monitoring metrics include at least one of the following: the number of times the nodes in the distributed coordinator are monitored; the number of connections of the nodes in the cluster in the distributed coordinator; the number of nodes in the data storage unit corresponding to the cluster; the space occupied by the data snapshot generated by the cluster when it is stored; and the length of the path for monitoring the nodes in the distributed coordinator.

[0134] In this embodiment, the number of times a node in the distributed coordinator is monitored can be represented by the number of Watches on the node, which can be denoted by Nw. The number of connections between nodes in the cluster of the distributed coordinator can be N. C The number of nodes in a cluster's corresponding data storage unit (Znode) can be represented by Nz. The storage space occupied by a data snapshot generated by the cluster can be the size of the snapshot generated by the cluster, and can be represented by Ssnapshot. The length of the path used to monitor nodes in the distributed coordinator can be the average length of the Watch Path, and can be represented by Sp.

[0135] Optionally, based on the above monitoring metrics, the average usage size of Zookeeper's Old Generation can be determined using the following formula: S O =(Nw*2+Nc+Nz)*32+Ssnapshot+Nw*Sp

[0136] Among them, S O Nw can be used to represent the average size of the Old generation in ZooKeeper, Nc can be used to represent the number of Watches on a node, Nz can be used to represent the number of connections between nodes in the cluster, Ssnapshot can be used to represent the size of the snapshot generated by the cluster, and Sp can be used to represent the average length of the Watch path.

[0137] In this embodiment, by dynamically determining the service scenario in which the distributed coordinator operates and determining the corresponding storage level based on the service scenario, and progressively advancing between multiple adjacent storage levels, the storage strategy set at the corresponding storage level can be obtained. This allows for efficient use of the data pool to store data in the memory of the distributed coordinator, thereby reducing memory usage and improving the data storage performance of the distributed coordinator. This solves the technical problem of low information processing efficiency for biological objects.

[0138] This disclosure also provides a data storage method for a distributed coordinator. Figure 3 is a flowchart of a data storage method for a distributed coordinator according to an embodiment of this disclosure. As shown in Figure 3, the method includes the following steps:

[0139] Step S302: In response to a data processing request from the client, determine the service scenario in which the distributed coordinator is running.

[0140] In the technical solution provided in step S302 of this disclosure, in response to a data processing request from the client, the service scenario in which the distributed coordinator operates can be determined. The data processing request may include data that the distributed coordinator needs to store during data processing. The service scenario may be the scenario to which the services supported by the distributed coordinator belong.

[0141] Optionally, when a client sends a write request or a SetWatch request to the distributed coordinator ZooKeeper, ZooKeeper can determine whether the service scenario is a microservice scenario, a distributed coordination scenario, or a more complex microservice governance scenario based on the content and type of the received data processing request. Based on the determined service scenario, ZooKeeper can apply specified storage strategies and data structures to respond to data processing requests and manage memory more efficiently, thereby better meeting the needs of the service scenario.

[0142] Step S304: Determine the storage level corresponding to the service scenario.

[0143] In the technical solution provided in step S304 of this disclosure, after determining the service scenario in which the distributed coordinator operates in response to a data processing request from the client, the storage level corresponding to the service scenario can be determined. The storage level can represent the granularity of data storage by the distributed coordinator. For example, the storage level can be one layer, two layers, three layers, etc. This is only an example and does not impose a specific limitation on the number of storage levels. Different data granularities correspond to different storage levels. In adjacent storage levels, the data granularity corresponding to the lower storage level is coarser than that corresponding to the higher storage level. For example, the data granularity corresponding to the first storage level is coarser than that corresponding to the second storage level, and the data granularity corresponding to the second storage level is coarser than that corresponding to the third storage level.

[0144] Optionally, different service scenarios can correspond to different storage tiers. For example, the storage tier for general service scenarios can be tier one or tier two, while the storage tier for more complex service governance scenarios can be tier three. This is just an example and does not impose specific restrictions on the storage tier for service scenarios.

[0145] Step S306: Obtain the storage policy set at the storage level corresponding to the service scenario.

[0146] In the technical solution provided in step S306 of this disclosure, after determining the storage level corresponding to the service scenario, the storage strategy set at the storage level corresponding to the service scenario can be obtained. The storage strategy can represent the rules by which the distributed coordinator stores data using a constant pool. The constant pool allows data to be referenced within the distributed coordinator, and different service scenarios correspond to different storage strategies.

[0147] In this embodiment, after determining the storage layer corresponding to the service scenario, the storage strategies set at the storage layers corresponding to different service scenarios can be obtained. For example, the storage strategy set at the first-level storage layer can be string pooling; the storage strategy set at the second-level storage layer can be hierarchical data pooling, such as managing the common and custom parts of the Znode path separately; and the storage strategy set at the third-level storage layer can be structured data storage and metadata pooling, such as distinguishing the storage of general metadata and custom metadata.

[0148] For example, when the identified service scenario is a microservice scenario or a distributed coordination scenario, since the corresponding storage layer for this service scenario is either Tier 1 or Tier 2, the storage strategy set at Tier 1 or Tier 2 can be determined. For instance, the storage strategy set at Tier 1 could be to use a string pool to manage duplicate string data, while the storage strategy set at Tier 2 could be to perform hierarchical splitting and pooling of Znode paths.

[0149] For another example, when the identified service scenario is a microservice governance scenario, since the corresponding storage hierarchy for this service scenario is three-tiered, the storage strategy set at the three-tiered storage hierarchy can be determined. For instance, the storage strategy set at the three-tiered storage hierarchy could be to perform structured storage of metadata.

[0150] Optionally, storage strategies suitable for different service scenarios can be determined based on data access patterns, data types, and data lifecycles in different service scenarios.

[0151] For example, in microservice and distributed coordination scenarios, operations such as service discovery, configuration management, and state synchronization occur frequently. Data access patterns are typically read-heavy and write-light, and data structures are relatively fixed, such as paths and metadata. The main data types in this service scenario can include strings and integers. String data may have significant duplication across different service instances, while integer data is usually a fixed value within a small range. In this service scenario, Znode paths and metadata are typically created during service registration and remain unchanged throughout the service instance's lifecycle; therefore, this data is persistent and suitable for long-term resident in memory.

[0152] Optionally, based on the data access patterns, data types, and data lifecycles in the aforementioned microservice and distributed coordination scenarios, the storage strategy set at the storage layer corresponding to this service scenario can be determined as follows: use a string pool to store Znode paths and other frequently occurring strings to reduce the repeated storage of the same strings in memory; introduce a weak reference mechanism to ensure that when a string is no longer referenced by any client, it can be automatically reclaimed by the JVM's garbage collection mechanism to avoid memory leaks; and use a fixed-size data type to store numeric data such as ports to reduce the additional memory overhead caused by string representation.

[0153] For another example, in microservice governance scenarios, data access is more diverse and dynamic, requiring frequent updates and queries of metadata to support real-time policy adjustments. Besides standard string and numeric types, microservice governance involves more complex data types, such as booleans, enumerations, and arrays. In this service scenario, metadata changes throughout the service instance's lifecycle, and some metadata, such as availability zone information and service version numbers, exhibits high repetition in large-scale deployments.

[0154] Optionally, based on the data access patterns, data types, and data lifecycles in the microservice governance scenario described above, the storage strategy set at the storage layer corresponding to this service scenario can be determined as follows: storing metadata in a structured manner, distinguishing between fixed metadata and variable metadata, and reducing unnecessary hashing and reference overhead; for highly repetitive metadata, adopting a mechanism similar to a string constant pool to ensure that there is only one copy of the same metadata in memory, and instances share the same reference.

[0155] Step S308: According to the storage strategy, data is cached in the memory of the distributed coordinator using a constant pool.

[0156] In the technical solution provided by step S308 of this disclosure, after obtaining the storage strategy corresponding to the service scenario, data can be cached in the memory of the distributed coordinator using a constant pool according to the obtained storage strategy.

[0157] In this embodiment, according to the storage strategy, data is stored in a data pool within ZooKeeper's memory management, and references to this data are allowed within ZooKeeper. A shared data storage area, i.e., a data pool, can be pre-created and managed. This data pool can be used to store data that is reused or frequently accessed in multiple locations. Storing data in this data pool allows a single instance of data to be stored in memory and accessed by reference rather than copying when needed, thereby saving memory resources, improving data access efficiency, and enhancing overall system performance.

[0158] Optionally, the data in the data pool is not static. For data that can be modified, when the data is updated, the data pool will create a new data instance. When the old instance no longer has references pointing to it, it will be cleaned up by the garbage collection mechanism. Thus, the data pool can not only ensure efficient data storage, but also maintain data consistency through the reference update mechanism when the data changes.

[0159] This embodiment utilizes a data pool to store data in the distributed coordinator and allows data to be referenced within the distributed coordinator. By sharing a single instance of data and using a reference access mechanism, it effectively reduces memory usage, improves data access efficiency, and ensures data consistency.

[0160] Through steps S302 to S308 of this disclosure, in response to a data processing request from a client, the service scenario in which the distributed coordinator operates is determined. The data processing request includes data that the distributed coordinator needs to store during data processing, and the service scenario is the scenario to which the services supported by the distributed coordinator belong. The storage level corresponding to the service scenario is determined, where the storage level represents the data granularity at which the distributed coordinator stores data. Different data granularities correspond to different storage levels, and in adjacent storage levels, the data granularity corresponding to the lower storage level is coarser than that corresponding to the higher storage level. The storage strategy set at the storage level corresponding to the service scenario is obtained, where the storage strategy represents the rules by which the distributed coordinator uses a constant pool to store data. The constant pool allows data to be referenced within the distributed coordinator, and different service scenarios correspond to different storage strategies. According to the storage strategy, data is cached in the memory of the distributed coordinator using the constant pool. In other words, this embodiment dynamically determines the service scenario in which the distributed coordinator operates and determines the corresponding storage level based on the service scenario. Multiple adjacent storage levels are progressively advanced, thereby obtaining the storage strategy set at the corresponding storage level. This allows for efficient use of the data pool to store data in the memory of the distributed coordinator, thereby reducing memory usage and improving the data storage performance of the distributed coordinator, thus solving the technical problem of low processing efficiency for biological objects.

[0161] This disclosure also provides a data processing method for a distributed coordinator. Figure 4 is a flowchart of another data processing method for a distributed coordinator according to an embodiment of this disclosure. As shown in Figure 4, the method includes the following steps:

[0162] Step S402: Obtain the data processing request by calling the first interface, and respond to the data processing request to determine the service scenario in which the distributed coordinator is running.

[0163] In the technical solution provided by step S402 of this disclosure, a data processing request can be obtained by calling a first interface, and the service scenario in which the distributed coordinator runs can be determined in response to the data processing request. The first interface may include a first parameter, the value of which may include the data processing request. The data processing request may include data that the distributed coordinator needs to store during data processing, and the service scenario may be the scenario to which the services supported by the distributed coordinator belong.

[0164] Optionally, when a client sends a write request or a SetWatch request to the distributed coordinator ZooKeeper, ZooKeeper can determine whether the service scenario is a microservice scenario, a distributed coordination scenario, or a more complex microservice governance scenario based on the content and type of the received data processing request. Based on the determined service scenario, ZooKeeper can apply specified storage strategies and data structures to respond to data processing requests and manage memory more efficiently, thereby better meeting the needs of the service scenario.

[0165] Step S404: Determine the storage level corresponding to the service scenario.

[0166] In the technical solution provided in step S404 of this disclosure, after obtaining the data processing request by calling the first interface and responding to the data processing request to determine the service scenario in which the distributed coordinator runs, the storage level corresponding to the service scenario can be determined. The storage level can represent the data granularity at which the distributed coordinator stores data. For example, the storage level can be one layer, two layers, three layers, etc. This is only an example and does not impose a specific limitation on the number of storage levels. Different data granularities correspond to different storage levels. In two adjacent storage levels, the data granularity corresponding to the lower storage level is coarser than that corresponding to the higher storage level. For example, the data granularity corresponding to the first storage level is coarser than that corresponding to the second storage level, and the data granularity corresponding to the second storage level is coarser than that corresponding to the third storage level.

[0167] Optionally, different service scenarios can correspond to different storage tiers. For example, the storage tier for general service scenarios can be tier one or tier two, while the storage tier for more complex service governance scenarios can be tier three. This is just an example and does not impose specific restrictions on the storage tier for service scenarios.

[0168] Step S406: Obtain the storage policy corresponding to the service scenario.

[0169] In the technical solution provided in step S406 of this disclosure, after determining the storage level corresponding to the service scenario, the storage strategy set at the storage level corresponding to the service scenario can be obtained. The storage strategy can represent the rules by which the distributed coordinator stores data using a data pool. The data pool allows data to be referenced within the distributed coordinator, and different service scenarios correspond to different storage strategies.

[0170] In this embodiment, after determining the storage layer corresponding to the service scenario, the storage strategies set at the storage layers corresponding to different service scenarios can be obtained. For example, the storage strategy set at the first-level storage layer can be string pooling; the storage strategy set at the second-level storage layer can be hierarchical data pooling, such as managing the common and custom parts of the Znode path separately; and the storage strategy set at the third-level storage layer can be structured data storage and metadata pooling, such as distinguishing the storage of general metadata and custom metadata.

[0171] For example, when the identified service scenario is a microservice scenario or a distributed coordination scenario, since the corresponding storage layer for this service scenario is either Tier 1 or Tier 2, the storage strategy set at Tier 1 or Tier 2 can be determined. For instance, the storage strategy set at Tier 1 could be to use a string pool to manage duplicate string data, while the storage strategy set at Tier 2 could be to perform hierarchical splitting and pooling of Znode paths.

[0172] For another example, when the identified service scenario is a microservice governance scenario, since the corresponding storage hierarchy for this service scenario is three-tiered, the storage strategy set at the three-tiered storage hierarchy can be determined. For instance, the storage strategy set at the three-tiered storage hierarchy could be to perform structured storage of metadata.

[0173] Optionally, storage strategies suitable for different service scenarios can be determined based on data access patterns, data types, and data lifecycles in different service scenarios.

[0174] For example, in microservice and distributed coordination scenarios, operations such as service discovery, configuration management, and state synchronization occur frequently. Data access patterns are typically read-heavy and write-light, and data structures are relatively fixed, such as paths and metadata. The main data types in this service scenario can include strings and integers. String data may have significant duplication across different service instances, while integer data is usually a fixed value within a small range. In this service scenario, Znode paths and metadata are typically created during service registration and remain unchanged throughout the service instance's lifecycle; therefore, this data is persistent and suitable for long-term resident in memory.

[0175] Optionally, based on the data access patterns, data types, and data lifecycles in the aforementioned microservice and distributed coordination scenarios, the storage strategy set at the storage layer corresponding to this service scenario can be determined as follows: use a string pool to store Znode paths and other frequently occurring strings to reduce the repeated storage of the same strings in memory; introduce a weak reference mechanism to ensure that when a string is no longer referenced by any client, it can be automatically reclaimed by the JVM's garbage collection mechanism to avoid memory leaks; and use a fixed-size data type to store numeric data such as ports to reduce the additional memory overhead caused by string representation.

[0176] For another example, in microservice governance scenarios, data access is more diverse and dynamic, requiring frequent updates and queries of metadata to support real-time policy adjustments. Besides standard string and numeric types, microservice governance involves more complex data types, such as booleans, enumerations, and arrays. In this service scenario, metadata changes throughout the service instance's lifecycle, and some metadata, such as availability zone information and service version numbers, exhibits high repetition in large-scale deployments.

[0177] Optionally, based on the data access patterns, data types, and data lifecycles in the microservice governance scenario described above, the storage strategy set at the storage layer corresponding to this service scenario can be determined as follows: storing metadata in a structured manner, distinguishing between fixed metadata and variable metadata, and reducing unnecessary hashing and reference overhead; for highly repetitive metadata, adopting a mechanism similar to a string constant pool to ensure that there is only one copy of the same metadata in memory, and instances share the same reference.

[0178] Step S408: According to the storage strategy, the data is stored in the memory of the distributed coordinator using a data pool.

[0179] In the technical solution provided by step S408 of this disclosure, after obtaining the storage policy corresponding to the service scenario, data can be stored in the memory of the distributed coordinator using a data pool according to the obtained storage policy.

[0180] In this embodiment, according to the storage strategy, data is stored in a data pool within ZooKeeper's memory management, and references to this data are allowed within ZooKeeper. A shared data storage area, i.e., a data pool, can be pre-created and managed. This data pool can be used to store data that is reused or frequently accessed in multiple locations. Storing data in this data pool allows a single instance of data to be stored in memory and accessed by reference rather than copying when needed, thereby saving memory resources, improving data access efficiency, and enhancing overall system performance.

[0181] Optionally, the data in the data pool is not static. For data that can be modified, when the data is updated, the data pool will create a new data instance. When the old instance no longer has references pointing to it, it will be cleaned up by the garbage collection mechanism. Thus, the data pool can not only ensure efficient data storage, but also maintain data consistency through the reference update mechanism when the data changes.

[0182] This embodiment utilizes a data pool to store data in the distributed coordinator and allows data to be referenced within the distributed coordinator. By sharing a single instance of data and using a reference access mechanism, it effectively reduces memory usage, improves data access efficiency, and ensures data consistency.

[0183] Step S410: Output the data stored in the data pool by calling the second interface.

[0184] In the technical solution provided by step S410 of this disclosure, after storing data in the memory of the distributed coordinator using a data pool according to the storage strategy, the data stored in the data pool can be output by calling a second interface. The second interface may include a second parameter, and the value of the second parameter may include the data stored in the data pool.

[0185] Through steps S402 to S410 of this disclosure, a data processing request is obtained by calling a first interface, and in response to the data processing request, the service scenario in which the distributed coordinator runs is determined. The first interface includes a first parameter, the value of which includes the data processing request. The data processing request includes data that the distributed coordinator needs to store during data processing. The service scenario is the scenario to which the services supported by the distributed coordinator belong. The storage level corresponding to the service scenario is determined, where the storage level represents the data granularity at which the distributed coordinator stores data. Different data granularities correspond to different storage levels. In the storage hierarchy, the data granularity of the lower storage level is coarser than that of the higher storage level. The storage strategy set at the storage level corresponding to the service scenario is obtained. This storage strategy represents the rules by which the distributed coordinator uses a data pool to store data. The data pool allows data to be referenced within the distributed coordinator, and different service scenarios correspond to different storage strategies. Data is stored in the distributed coordinator's memory using the data pool according to the storage strategy. The data stored in the data pool is output by calling a second interface, where the second interface includes a second parameter whose value includes the data stored in the data pool. In other words, this embodiment dynamically determines the service scenario in which the distributed coordinator operates and determines the corresponding storage level based on the service scenario. Multiple adjacent storage levels are progressively separated, thereby obtaining the storage strategy set at the corresponding storage level. This allows for efficient use of the data pool in the distributed coordinator's memory to store data, reducing memory usage and improving the data storage performance of the distributed coordinator, thus solving the technical problem of low processing efficiency for biological objects.

[0186] This disclosure also provides a data processing system for a distributed coordinator. Figure 5 is a schematic diagram of another data processing system for a distributed coordinator according to an embodiment of this disclosure. As shown in Figure 5, the system includes a client 502 and a distributed coordinator 504.

[0187] Client 502 is configured to send a data processing request, which includes data that the distributed coordinator needs to store during the data processing process.

[0188] The distributed coordinator 504, connected to the client, is configured to respond to data processing requests by determining the service scenario in which the distributed coordinator operates. The data processing request includes data that the distributed coordinator needs to store during data processing, and the service scenario refers to the scenario to which the services supported by the distributed coordinator belong. It then determines the storage level corresponding to the service scenario, where the storage level represents the granularity of data storage by the distributed coordinator; different granularities correspond to different storage levels, and within adjacent storage levels, the lower storage level corresponds to a coarser granularity than the higher storage level. Next, it retrieves the storage policy set at the storage level corresponding to the service scenario. This storage policy represents the rules by which the distributed coordinator uses a data pool to store data, allowing data to be referenced within the distributed coordinator; different service scenarios correspond to different storage policies. Finally, it stores the data in the distributed coordinator's memory using the data pool according to the storage policy.

[0189] Optionally, when a client sends a write request or a SetWatch request, or other data processing request, to the distributed coordinator ZooKeeper, ZooKeeper can determine whether the service scenario is a microservice scenario, a distributed coordination scenario, or a more complex microservice governance scenario based on the content and type of the received data processing request. Based on the determined service scenario, ZooKeeper can apply specified storage strategies and data structures to respond to data processing requests and manage memory more efficiently, thereby better meeting the needs of the service scenario.

[0190] Optionally, after determining the service scenario in which the distributed coordinator operates, the corresponding storage layer can be determined. Different service scenarios can correspond to different storage layers. For example, a general service scenario can correspond to a storage layer of one or two, while a more complex service governance scenario can correspond to a storage layer of three. This is only an example and does not impose specific restrictions on the storage layer corresponding to the service scenario.

[0191] Optionally, after determining the storage level corresponding to the service scenario, storage strategies corresponding to different service scenarios can be obtained. When the determined service scenario is a microservice scenario or a distributed coordination scenario, the storage strategy can be to reduce redundant string storage and make full use of the JVM's garbage collection mechanism; when the determined service scenario is a microservice governance scenario, the storage strategy can be a strategy that focuses on the efficient storage and management of metadata through structured storage.

[0192] Optionally, according to the storage strategy, data can be stored in a data pool within ZooKeeper's memory management, allowing it to be referenced within ZooKeeper. A shared data storage area, i.e., a data pool, can be pre-created and managed. This data pool can be used to store data that is reused or frequently accessed in multiple locations. Storing data in this data pool allows a single instance of data to be stored in memory and accessed by reference rather than copying when needed, thus saving memory resources, improving data access efficiency, and enhancing overall system performance.

[0193] Optionally, the data in the data pool is not static. For data that is allowed to be modified, a new data instance is created in the data pool when the data is updated. When the old instance no longer has references pointing to it, it will be cleaned up by the garbage collection mechanism. Thus, the data pool can not only ensure efficient storage of data, but also maintain data consistency through the reference update mechanism when data changes.

[0194] This embodiment utilizes a data pool to store data in the distributed coordinator and allows data to be referenced within the distributed coordinator. By sharing a single instance of data and using a reference access mechanism, it effectively reduces memory usage, improves data access efficiency, and ensures data consistency.

[0195] In this system, a data processing request is sent via client 502. This request includes data that the distributed coordinator needs to store during data processing. The distributed coordinator 504 responds to the data processing request, determining the service scenario in which it operates. The service scenario refers to the scenario to which the services supported by the distributed coordinator belong. The system then determines the storage level corresponding to the service scenario. The storage level represents the granularity at which the distributed coordinator stores data; different granularities correspond to different storage levels. Within adjacent storage levels, the lower storage level has a coarser granularity than the higher storage level. The system retrieves the storage policy set at the storage level corresponding to the service scenario. This policy represents the rules by which the distributed coordinator uses a data pool to store data. The data pool allows data to be referenced within the distributed coordinator, and different service scenarios correspond to different storage policies. Finally, the system stores the data in the distributed coordinator's memory using the data pool, according to the storage policy. In other words, this embodiment dynamically determines the service scenario in which the distributed coordinator operates and determines the corresponding storage level based on the service scenario. Multiple adjacent storage levels are progressively advanced, thereby obtaining the storage strategy set at the corresponding storage level. This allows for efficient use of the data pool to store data in the memory of the distributed coordinator, thereby reducing memory usage and improving the data storage performance of the distributed coordinator. This solves the technical problem of low information processing efficiency for biological objects.

[0196] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application, such as the data to be verified, are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0197] Currently, ZooKeeper acts as a distributed coordinator, storing a large amount of state data, managing changes to this data, and pushing these changes to clients. ZooKeeper stores this data and manages the data structures clients are listening to in memory. In scenarios with a large number of nodes and a large number of Watchers, ZooKeeper can consume a significant amount of memory storing state data and Watcher data structures, leading to frequent garbage collection (GC) and impacting overall system performance.

[0198] For example, Zookeeper is used in an online cluster with 9 nodes, 16 CPUs, and 32 gigabytes (GB) of memory. The cluster stores 4.04 million (w) nodes, with an average of 7 million Watches per node and an average memory usage of 3.61GB in the Old Generation. Currently, this Zookeeper does not use service governance capabilities. If governance capabilities such as full-link canary deployments, tag routing, and service preheating are introduced, Zookeeper's memory usage will increase by another 10-30% due to the automatic increase in metadata in service URLs.

[0199] In one optional example, for the string constant pool, the JDK's String type provides a string pooling (intern) method for JDK-level string caching, maintaining a segment of memory in the JVM to store these constants. The location of constant storage differs across versions. In JDK 6 and earlier, these constants were stored in the Meta area, which can easily lead to Out of Memory (OOM) errors when there are many constants. In JDK 7 and later, constants are stored in the Heap area by default to avoid OOM. However, the size of the hash table storing constants is a fixed value, specified as a parameter when the application starts. If the parameter value is not specified appropriately, a large number of hash collisions will occur in scenarios with many constants, significantly increasing the time spent constructing String objects.

[0200] The WatchManager, an optimized version of ZooKeeper, uses a BitHashMap to improve upon the original WatchManager, reducing the memory consumption of the large number of HashMap objects and HashNode objects generated by numerous Watches. However, this introduces the problem of potential accumulation of dead watches and the loss of recursive Watch functionality. This method still suffers from excessive memory consumption in scenarios where many Znodes are watched by multiple clients.

[0201] To optimize string constants for registry center scenarios, the distributed coordinator Eureka also pools instance-related information using a constant pool when storing registration information. However, this distributed coordinator does not separate immutable and user-defined metadata; it uses a single Map for storage.

[0202] To address the aforementioned issues, this disclosure proposes a hierarchical memory optimization scheme based on a string pool. Without significantly impacting the Central Processing Unit (CPU) usage, it optimizes Zookeeper's memory usage, improves Zookeeper's memory efficiency, and reduces system instability caused by a large number of Watch statements. Practical stress testing shows that in microservice and distributed coordination scenarios, the memory allocation usage of this disclosure is reduced by more than 60%; in microservice governance scenarios, the memory allocation usage can be reduced by more than 90%. Furthermore, this disclosure is not limited to Zookeeper; it also demonstrates memory optimization effects to a certain extent for similar distributed coordination services such as Nacos or Eureka. In addition, this disclosure performs more granular structuring of instance registration information and more fine-grained decomposition of metadata, enabling immutable metadata to achieve lower redundancy and thus saving more memory space.

[0203] ZooKeeper manages the data stored in ZooKeeper and the Watches on clients through a Map structure. A common ZooKeeper scenario is that some clients update data in ZooKeeper, while others listen for these changes through Watches. When a client sends a write request to ZooKeeper, ZooKeeper inserts a record into the data storage Map structure, with the key being the Path field from the request. When a client sends a SetWatch request to ZooKeeper, ZooKeeper inserts data into the Watch-managed Map structure. As shown in Figure 6, for the same Znode, the key stored in the data storage Map and the key stored in the Watch-managed Map are two separate objects in memory; that is, the same string constant exists as multiple redundant objects in memory.

[0204] In this embodiment of the disclosure, a mathematical model is introduced to quantify the memory model of Zookeeper, thereby obtaining an approximate average size of the Old Generation of Zookeeper. This can more effectively measure the memory usage status of Zookeeper, and subsequent stress test results show that it is consistent with the representation of the average size of the Old Generation.

[0205] The more Znodes and Watches there are, the more redundancy there will be, resulting in higher memory consumption. When a large number of Znodes have long-running Watches, these strings will be moved to the Old Generation. When memory resources are insufficient, this can easily lead to frequent garbage collection (GC) in the cluster, thus affecting the stability of the ZooKeeper cluster.

[0206] To address the memory resource consumption issue caused by redundant string constants, this embodiment optimizes the memory using a three-layer architecture. Figure 10 is a schematic diagram of the overall architecture of a layered memory optimization scheme based on a string pool according to an embodiment of this disclosure. As shown in Figure 10, the overall optimization scheme is divided into three layers: the first layer is the string pool, the second layer is the split node path pool, and the third layer is the structured instance information pool. Each layer is progressive, providing finer-grained optimization for caching data stored in ZooKeeper. The first and second layers support general scenarios, ultimately achieving a 60% memory saving. The third layer supports service governance scenarios, ultimately achieving a 90% memory saving.

[0207] The three-layer architecture described in this embodiment will be further described below.

[0208] The first layer is an optimization based on WeakReference's StringPool. In the original memory storage structure, for the same Znode, WatchManager and DataTree reference two different String objects as the value of Path, even though this value is the same and immutable. Therefore, a constant pool is considered to pool string objects with the same value, allowing these strings to be reused across multiple references. As shown in Figure 7, the left half of the figure represents long-lived data, where the string pool ensures that these objects with the same value have only one copy in memory. The right half of the figure shows that strings generated by new requests are no longer directly referenced, and are thus directly reclaimed by the GC after the request is processed, reducing memory usage. The network flow is handled via sockets, which provide the channel for network communication. Non-blocking input / output (NIO) and byte buffers are used to improve data transmission efficiency. The client sends AddWatchRequest, CreateRequest, and GetDataRequest to Zookeeper, all three requests referencing the path "Ref Path: / test". The "Bytes Path: / test" corresponding to the request header is stored in the JVM's Heap.

[0209] When a Path has no references, it will be automatically reclaimed by the garbage collector (GC), thus achieving automatic memory cleanup. This can be managed using a combination of WeakHashMap and WeakReference. When a new string constant needs to be added to the string pool, the string object is wrapped using WeakReference, and WeakHashMap is called to insert the string and its weakly referenced value. Because WeakHashMap uses insertion, the key is automatically wrapped with a WeakReference. Therefore, the string referenced by WeakHashMap is a weak reference, so the string is not referenced in WatchManager or DataTree. Only when a weak reference exists in WeakHashMap will the GC automatically reclaim this string.

[0210] In the above optimization, by optimizing the items in the Zookeeper memory model, all keys stored in the Map managed by the Watch use the keys from the data storage. Therefore, the average usage of the Zookeeper Old Generation based on this scheme can be expressed by the following formula: S O = (Nw*2+Nc+Nz)*32+Ssnapshot

[0211] From the above formulas, it can be seen that in scenarios with a large number of Watches (tens of millions), memory usage can be reduced through optimization. Figure 11 is a schematic diagram of Old Zone memory usage according to related technologies. As shown in Figure 11, in a scenario with 10 million Watches, a snapshot size of 100 kilobytes (KB), 1000 connections, and 10,000 Znodes, the approximate Old Zone memory usage is 1.7GB. Figure 12 is a schematic diagram of Old Zone memory usage according to an embodiment of this disclosure. As shown in Figure 12, the optimized scheme optimizes Old Zone memory usage to 467 megabytes (MB), reducing memory usage by approximately one-third.

[0212] The second layer further reduces memory usage by splitting the path into fine-grained strings. Considering that data stored in ZooKeeper is generally multi-level, and the Znode path is stored in memory at multiple locations with different granularities, the Znode itself is stored in memory in the form of a complete Path->Znode using a Map. For the parent node to which the Znode belongs, the string of the last level of the Znode is stored as the name of the current Znode in the list of child nodes of the parent node, as shown in Figure 8, which illustrates the memory storage format of the data " / test / current".

[0213] As shown in Figure 8, there is redundant storage of the strings "current" and "test". To solve this problem, as shown in Figure 9, we consider splitting the Znode's Path according to its hierarchy, and linking the split hierarchy through the ZnodePath data structure to provide relevant functionality when used.

[0214] Figure 13 is a schematic diagram of data storage based on a ZnodePath data structure according to an embodiment of this disclosure. The right half of Figure 13 is the same as the right half of Figure 7, and will not be described again here. As can be seen from the left half of Figure 13, by replacing the Path previously stored as String with ZnodePath for storage, such as "Ref ZnodePath: / test / 1", "Ref ZnodePath: / test / 2", and "Ref ZnodePath: / test / 3", when storing Znodes, the key type is changed from String to ZnodePath, and the redundancy of strings stored in memory will be further reduced.

[0215] Further optimizations to the memory model will reduce the Ssnapshot coefficient to varying degrees depending on the use case. For example, in a three-layer structure with 10 million child nodes, a single-layer Path length of 50 bytes (B) and a data size of 100B, the Znode memory footprint can be calculated using the following formula: Sz = (Sp + Sn + Sd) * Nz

[0216] Here, Sz represents the approximate size of a Znode in memory, Sp represents the length of the complete Znode path, Sn represents the length of the Znode name (the length of the last level), Sd represents the size of the data stored in the Znode, and Nz represents the number of Znodes. Substituting the data into the above formula, we can obtain an approximate memory size of 2.79GB for the Znodes stored in memory.

[0217] In this embodiment, the optimization scheme pools the names of Znodes at each level. Therefore, the approximate size of the Znodes stored in memory in the optimization scheme can be estimated using the following formula: Sz = Nz*Sl1 + Nz*Sl2 + Nz*Sl3 + Nz*Sd

[0218] The final calculated memory footprint of the optimized scheme for Znodes in this scenario is approximately 2.32GB. This calculation is based on the worst-case scenario where Znode names are unique across all levels. However, in most scenarios, a large number of intermediate-level Znodes share the same name. Assuming that the first two levels of 10,000 Znodes have the same Znode name, the memory footprint can be approximated using the following formula: Sz = Sl1 + Sl2 + Nz * (Sl3 + Sd)

[0219] The approximate memory size required to calculate the Znode is 1.39GB, and the memory usage of the stored Znode is further reduced to 50% of the original.

[0220] The third layer focuses on optimizing registration information in service governance scenarios. Microservice governance scenarios include, but are not limited to, end-to-end canary deployments, tag-based routing, priority routing within the same availability zone, and service preheating. In each of these scenarios, metadata representations are added to nodes during service registration. For microservice governance scenarios, the metadata added to the URLs of services registered with the registry center has relatively high redundancy. Even in large-scale applications, the metadata on different nodes is generally within an enumerable set. For example, with priority routing within the same availability zone, the redundancy increases as the number of nodes increases.

[0221] From the perspective of the registry center ZooKeeper, in service governance scenarios, for example, Dubbo writes information about registered instances to ZooKeeper. The format of this service registration information can be " / dubbo / {interfaceName} / providers / dubbo: / / {ip}:{port} / {interfaceName}?{meta}". Here, `interfaceName` represents the Dubbo interface name, `ip` represents the IP address of the registered node, `port` represents the port number of the registered node, and `meta` represents the metadata of the registered node.

[0222] In the second-layer optimization, constants such as `dubbo`, `providers`, and `interfaceName` have already been pooled, with only one object existing in memory. However, in the service governance registration scenario, the registration information contains even more redundant constant strings. In service registration, the metadata of registered instance information has significant overlap, with many duplicate keys and values, and the IP and port of registered instances are enumerable and coded values ​​within a range. Further structuring the registration data and using a string pool for constant references can further reduce Zookeeper's memory usage in this scenario.

[0223] In this embodiment, an Instance data structure is defined to store registration information in a structured manner: Public class Instance{private final Integer ip; private final Integer port; private final String interfaceName; private final String[]meta}, and the toString method is overridden to format the data as a String when returning it to the client.

[0224] Figure 14 is a schematic diagram of structured storage of registration information according to an embodiment of this disclosure. As shown in Figure 14, the IP address is encoded into a 32-bit integer (int) value and boxed and stored in the Heap. Multiple instances registered with the same IP address can all reference the same integer (Integer) value, and the same applies to the Port. Since the registered instance information is immutable data, using a Map for storage would introduce unnecessary memory usage for HashNodes and HashTables. Therefore, for storing metadata, a fixed-length String array is considered, with Keys and Values ​​stored adjacently.

[0225] Since the metadata of a registered instance can be divided into two parts—intrinsic data and custom data—the storage of the key can be omitted. Instead, the key can be associated with the index of the data. Furthermore, instances registered for the same service share common metadata, which can be extracted and stored uniformly, further reducing the length of the array needed to store the metadata. The timestamp field in the registration information can be further extracted and stored as a long integer. Thus, Instance can be defined in a more granular way: Public class Instance{private final Integer ip; private final Integer port; private final String interfaceName; private final Object[] commonMeta; private final String[] customMeta}.

[0226] Shared metadata is stored using objects. Timestamps can be stored using Long, boolean values ​​using Boolean, and version information can be encoded as int, thus saving significant memory consumption associated with string storage. Figure 15 is a schematic diagram of another structured storage of registration information according to an embodiment of this disclosure. As shown in Figure 15, the final referencing effect can be seen.

[0227] Calculations show that registering 1 million Dubbo nodes and 10,000 services, with consistent metadata keys, no user-defined keys or values, and service names 32 bytes long, results in an approximate size of 400 bytes of data stored in ZooKeeper for a single Dubbo node. Registering instance information for 1 million service nodes would require approximately 715 MB of space. By optimizing the storage of registration information using a structured approach, the final memory footprint of the registration information can be calculated using the following formula: Sz = (Sip + Sport + Si + St + Spid) * Ni + 150

[0228] In this calculation, Sip represents the memory size for storing the encoded IP address, with a fixed value of 4; Sport represents the memory size for storing the port, with a fixed value of 4; Si represents the memory size for storing the timestamp, with a fixed value of 8; St represents the size of the string constant for storing the storage service name, assumed to be 32; Spid represents the memory size for storing the process identifier (PID), with a fixed value of 8; and Ni represents the total number of registered instances. The optimized scheme calculated using the above formulas results in a registration information memory footprint of approximately 53MB, significantly reducing the memory usage of the registration information.

[0229] In this embodiment, Zookeeper memory storage is optimized by using a string cache based on a constant pool, significantly reducing redundant strings stored in Zookeeper. This effectively reduces the memory usage of Zookeeper in scenarios with a large number of Znodes and Watchers, resulting in a 60% memory saving in such scenarios. Furthermore, the storage structure of the stored Znode paths is optimized, allowing the tree-structured storage to reuse string constants in the parent node names, further reducing memory usage by 50%. Finally, structured storage of registration information is proposed for service registration scenarios. This effectively compresses registered instance information and further reduces redundant string constants in the metadata of the registration information through a constant pool, resulting in a 90% memory saving in such scenarios.

[0230] This implementation introduces a constant pool based on weak references to cache string constants within ZooKeeper, reducing the memory footprint of numerous redundant strings. It also utilizes the JVM's own garbage collection (GC) mechanism to automatically reclaim constants that are no longer referenced. Furthermore, based on ZooKeeper's hierarchical data storage structure, a fine-grained string storage structure is introduced, and combined with the constant pool, this further reduces string redundancy in memory, improving memory utilization efficiency.

[0231] This embodiment introduces a structured registration information storage structure based on the data storage structure of service registration in the service governance scenario, separating mutable and immutable data, and further reducing redundant constants by combining a constant pool, thereby further reducing Zookeeper's memory usage in this scenario. This embodiment proposes a quantifiable memory model based on Zookeeper observable metrics, which can be used to conveniently measure and analyze memory usage within Zookeeper.

[0232] In this embodiment of the disclosure, in response to a data processing request, the service scenario in which the distributed coordinator operates is determined. The data processing request includes data that the distributed coordinator needs to store during data processing. The service scenario is the scenario to which the services supported by the distributed coordinator belong. The storage level corresponding to the service scenario is determined. The storage level represents the data granularity at which the distributed coordinator stores data; different data granularities correspond to different storage levels. In adjacent storage levels, the data granularity of the lower storage level is coarser than that of the higher storage level. A storage policy set at the storage level corresponding to the service scenario is obtained. The storage policy represents the rules by which the distributed coordinator uses a data pool to store data. The data pool allows data to be referenced within the distributed coordinator, and different service scenarios correspond to different storage policies. Data is stored in the distributed coordinator's memory using the data pool according to the storage policy. In other words, this embodiment dynamically determines the service scenario in which the distributed coordinator operates and determines the corresponding storage level based on the service scenario. Multiple adjacent storage levels are progressively advanced, thereby obtaining the storage strategy set at the corresponding storage level. This allows for efficient use of the data pool to store data in the memory of the distributed coordinator, thereby reducing memory usage and improving the data storage performance of the distributed coordinator. This solves the technical problem of low information processing efficiency for biological objects.

[0233] Figure 16 is a structural block diagram of a computing environment for a data processing method of a distributed coordinator according to an embodiment of the present disclosure. As shown in Figure 16, the computing environment 1601 includes multiple computing nodes (such as servers) running on a distributed network (shown as 1610-1, 1610-2, ..., in the figure). Each computing node contains local processing and memory resources, and the end user 1602 can remotely run applications or store data within the computing environment 1601. Applications can be provided as multiple services 1620-1, 1620-2, 1620-3, and 1620-4 in the computing environment 1601, representing services "A", "D", "E", and "H", respectively.

[0234] End user 1602 can provide and access services through a web browser or other software applications on a client. In some embodiments, the provisioning and / or requests of end user 1602 can be provided to ingress gateway 1630. Ingress gateway 1630 may include a corresponding agent to handle the provisioning and / or requests for services (one or more services provided in computing environment 1601).

[0235] The service is provided or deployed based on various virtualization technologies supported by the computing environment 1601. In some embodiments, the service may be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. VM-based virtualization can simulate a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the machine is virtualized by a virtual machine, container-based virtualization can launch containers to virtualize an entire operating system (OS), allowing multiple workloads to run on a single OS instance.

[0236] In one embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, as shown in Figure 16, service 1620-2 can be equipped with one or more Pods 1640-1, 1640-2, ..., 1640-N (collectively referred to as Pods). A Pod can include a proxy 1645 and one or more containers 1642-1, 1642-2, ..., 1642-M (collectively referred to as containers). One or more containers in a Pod handle requests related to one or more corresponding functions of the service. The proxy 1645 typically controls service-related network functions such as routing and load balancing. Other services can also be equipped with Pods similar to Pods.

[0237] During operation, executing a user request from end user 1602 may require calling one or more services in computing environment 1601, and executing one or more functions of one service may require calling one or more functions of another service. As shown in Figure 16, service "A" 1620-1 receives a user request from end user 1602 from ingress gateway 1630. Service "A" 1620-1 can call service "D" 1620-2, and service "D" 1620-2 can request service "E" 1620-3 to execute one or more functions.

[0238] The aforementioned computing environment can be a cloud computing environment, where resource allocation is managed by cloud services, allowing functionality development without needing to consider implementation, adjustment, or server scaling. This computing environment allows developers to execute event-responsive code without building or maintaining complex infrastructure. Services can be partitioned into a set of functions that can automatically and independently scale, rather than scaling a single hardware device to handle potential loads.

[0239] According to embodiments of this disclosure, a data processing apparatus for a distributed coordinator for implementing the data processing method of the distributed coordinator shown in FIG2 is also provided.

[0240] Figure 17 is a schematic diagram of a data processing device for a distributed coordinator according to an embodiment of the present disclosure. As shown in Figure 17, the data processing device 1700 for the distributed coordinator includes: a first determining component 1702, a second determining component 1704, a first acquiring component 1706, and a first storage component 1708.

[0241] The first determining component 1702 is configured to determine the service scenario in response to a data processing request, wherein the data processing request includes data that the distributed coordinator needs to store during data processing, and the service scenario is the scenario to which the services supported by the distributed coordinator belong.

[0242] The second determining component 1704 is configured to determine the storage level corresponding to the service scenario. The storage level is used to represent the data granularity at which the distributed coordinator stores data. Different data granularities correspond to different storage levels. In two adjacent storage levels, the data granularity corresponding to the lower storage level is coarser than that corresponding to the higher storage level.

[0243] The first acquisition component 1706 is configured to acquire the storage policy set at the storage level corresponding to the service scenario. The storage policy is used to represent the rules by which the distributed coordinator uses the data pool to store data. The data pool allows data to be referenced in the distributed coordinator. Different service scenarios correspond to different storage policies.

[0244] The first storage component 1708 is configured to store data in the memory of the distributed coordinator using a data pool, according to a storage strategy.

[0245] Here, the first determining component 1702, the second determining component 1704, the first acquiring component 1706, and the first storage component 1708 correspond to steps S202 to S208. The four components and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed above. The above modules or components may be hardware or software components stored in memory and processed by one or more processors. The above modules may also be part of the device and run in the server 10 provided in the above embodiments.

[0246] In the data processing device of this distributed coordinator, the service scenario in which the distributed coordinator runs is dynamically determined, and the corresponding storage level is determined according to the service scenario. Multiple adjacent storage levels are progressively advanced, thereby obtaining the storage strategy set at the corresponding storage level. Data can be efficiently stored in the memory of the distributed coordinator by utilizing the data pool, thereby reducing memory usage and improving the data storage performance of the distributed coordinator. This solves the technical problem of low information processing efficiency for biological objects.

[0247] According to embodiments of this disclosure, a data storage device for a distributed coordinator for implementing the data storage method of the distributed coordinator shown in FIG3 is also provided.

[0248] Figure 18 is a schematic diagram of a data storage device for a distributed coordinator according to an embodiment of the present disclosure. As shown in Figure 18, the data storage device 1800 of the distributed coordinator includes: a third determining component 1802, a fourth determining component 1804, a second obtaining component 1806, and a second storage component 1808.

[0249] The third determining component 1802 is configured to determine the service scenario in which the distributed coordinator runs in response to a data processing request from a client. The data processing request includes data that the distributed coordinator needs to store during data processing, and the service scenario is the scenario to which the services supported by the distributed coordinator belong.

[0250] The fourth determining component 1804 is configured to determine the storage level corresponding to the service scenario. The storage level is used to represent the data granularity of the distributed coordinator in storing data. Different data granularities correspond to different storage levels. In two adjacent storage levels, the data granularity corresponding to the lower storage level is coarser than that corresponding to the higher storage level.

[0251] The second acquisition component 1806 is configured to acquire the storage policy set at the storage level corresponding to the service scenario. The storage policy is used to represent the rules by which the distributed coordinator stores data using a constant pool. The constant pool allows data to be referenced in the distributed coordinator. Different service scenarios correspond to different storage policies.

[0252] The second storage component 1808 is configured to cache data in the memory of the distributed coordinator using a constant pool, according to a storage policy.

[0253] Here, the third determining component 1802, the fourth determining component 1804, the second acquiring component 1806, and the second storage component 1808 correspond to steps S302 to S308. The four components and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed above. The above modules or components may be hardware or software components stored in memory and processed by one or more processors. The above modules may also be part of the device and run in the server 10 provided in the above embodiments.

[0254] In the data storage device of this distributed coordinator, the service scenario in which the distributed coordinator runs is dynamically determined, and the corresponding storage level is determined according to the service scenario. Multiple adjacent storage levels are progressively advanced, thereby obtaining the storage strategy set at the corresponding storage level. Data can be efficiently stored in the memory of the distributed coordinator by utilizing the data pool, thereby reducing memory usage and improving the data storage performance of the distributed coordinator. This solves the technical problem of low information processing efficiency for biological objects.

[0255] According to embodiments of this disclosure, a data processing apparatus for a distributed coordinator for implementing the data processing method of the distributed coordinator shown in FIG4 is also provided.

[0256] Figure 19 is a schematic diagram of a data processing apparatus for a distributed coordinator according to an embodiment of the present disclosure. As shown in Figure 19, the data processing apparatus 1900 of the distributed coordinator includes: a fifth determining component 1902, a sixth determining component 1904, a third acquiring component 1906, a third storage component 1908, and an output component 1910.

[0257] The fifth determining component 1902 is configured to obtain a data processing request by calling a first interface, and in response to the data processing request, determine the service scenario in which the distributed coordinator runs. The first interface includes a first parameter, the value of which includes the data processing request. The data processing request includes the data that the distributed coordinator needs to store during the data processing process, and the service scenario is the scenario to which the service supported by the distributed coordinator belongs.

[0258] The sixth determining component 1904 is configured to determine the storage level corresponding to the service scenario. The storage level is used to represent the data granularity of the distributed coordinator in storing data. Different data granularities correspond to different storage levels. In two adjacent storage levels, the data granularity corresponding to the lower storage level is coarser than that corresponding to the higher storage level.

[0259] The third acquisition component 1906 is configured to acquire the storage policy set at the storage level corresponding to the service scenario. The storage policy is used to represent the rules by which the distributed coordinator uses the data pool to store data. The data pool allows data to be referenced in the distributed coordinator. Different service scenarios correspond to different storage policies.

[0260] The third storage component 1908 is configured to store data in the memory of the distributed coordinator using a data pool, according to a storage strategy.

[0261] Output component 1910 is configured to output data stored in the data pool by calling a second interface, wherein the second interface includes a second parameter, the value of which includes the data stored in the data pool.

[0262] Here, the fifth determining component 1902, the sixth determining component 1904, the third acquiring component 1906, the third storage component 1908, and the output component 1910 correspond to steps S402 to S410. The five components and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed above. The aforementioned modules or components can be hardware or software components stored in memory and processed by one or more processors. These modules can also run as part of the device in the server 10 provided in the above embodiments.

[0263] In the data processing device of this distributed coordinator, the service scenario in which the distributed coordinator runs is dynamically determined, and the corresponding storage level is determined according to the service scenario. Multiple adjacent storage levels are progressively advanced, thereby obtaining the storage strategy set at the corresponding storage level. Data can be efficiently stored in the memory of the distributed coordinator by utilizing the data pool, thereby reducing memory usage and improving the data storage performance of the distributed coordinator. This solves the technical problem of low information processing efficiency for biological objects.

[0264] The preferred embodiments involved in the above-described embodiments of this disclosure are the same as the solutions, application scenarios, and implementation processes provided in the above-described embodiments, but are not limited to the solutions provided in the above-described embodiments.

[0265] Embodiments of this disclosure may provide a computing device. FIG20 is a structural block diagram of a computing device according to an embodiment of this disclosure. As shown in FIG20, the computing device 2000 may include: one or more (only one is shown in the figure) processors 2002, memory 2004, memory controller, and peripheral interfaces.

[0266] The aforementioned computing device can be understood as an integrated smart terminal, including but not limited to servers, desktop computers, personal computers (PCs), all-in-one model machines, etc., and the computing device may have the model described in the above embodiments of this disclosure pre-installed.

[0267] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API calling capabilities, allowing models to be called into created applications through API interfaces, and providing application management tools to achieve application control.

[0268] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0269] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0270] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0271] Embodiments of this disclosure can provide an electronic device. FIG21 is a structural block diagram of an electronic device according to an embodiment of this disclosure. As shown in FIG21, the electronic device includes: an input / output device 2102; a memory 2104; and a processor 2106, wherein the processor 2106 is connected to the input / output device 2102 and the memory 2104 via a bus 2108.

[0272] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0273] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0274] It will be understood by those skilled in the art that the structure shown in the figure is merely illustrative, and the computing device may also be a smartphone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal device. This figure does not limit the structure of the aforementioned computing device. For example, the computing device 1000 may also include more or fewer components (e.g., network interface, display device, etc.) than those shown in the figure, or may have a different configuration than that shown in the figure.

[0275] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0276] Embodiments of this disclosure also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0277] Optionally, in this embodiment, the storage medium may be located in a computing device.

[0278] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, which, when the executable program is running, controls the device where the computer-readable storage medium is located to execute the method described in any of the above embodiments.

[0279] Embodiments of this disclosure also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0280] Embodiments of this disclosure also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which can be used to store a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0281] Embodiments of this disclosure also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0282] In the above embodiments of this disclosure, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0283] In the several embodiments provided in this disclosure, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0284] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0285] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0286] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0287] The above description is only a preferred embodiment of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this disclosure, and these improvements and modifications should also be considered within the scope of protection of this disclosure. Industrial applicability

[0288] This disclosure provides a data processing method for a distributed coordinator. By dynamically determining the service scenario in which the distributed coordinator operates and determining the corresponding storage level based on the service scenario, and progressively advancing between multiple adjacent storage levels, a storage strategy set at the corresponding storage level can be obtained. This allows for efficient use of the data pool to store data in the memory of the distributed coordinator, thereby reducing memory usage and improving the data storage performance of the distributed coordinator. This solves the technical problem of low information processing efficiency for biological objects.

Claims

1. A data processing method for a distributed coordinator, comprising: In response to a data processing request, the service scenario in which the distributed coordinator operates is determined, wherein the data processing request includes data that the distributed coordinator needs to store during the data processing process, and the service scenario is the scenario to which the services supported by the distributed coordinator belong; The storage level corresponding to the service scenario is determined, wherein the storage level is used to represent the data granularity at which the distributed coordinator stores the data, and different data granularities correspond to different storage levels. In different storage levels, among two adjacent storage levels, the data granularity corresponding to the lower storage level is coarser than the data granularity corresponding to the higher storage level. Obtain the storage policy set at the storage level corresponding to the service scenario, wherein the storage policy is used to represent the rules by which the distributed coordinator stores the data using a data pool, the data pool allows the data to be referenced in the distributed coordinator, and different service scenarios correspond to different storage policies; According to the storage strategy, the data is stored in the memory of the distributed coordinator using the data pool.

2. The method of claim 1, wherein, Obtaining the storage policy set at the storage tier corresponding to the service scenario includes: In the storage policy set, the storage policy set at the storage level corresponding to the service scenario is obtained, wherein the different storage policies corresponding to different storage levels in the storage policy set are interconnected.

3. The method of claim 1, wherein, In response to data processing requests, determine the service scenario in which the distributed coordinator operates, including: In response to the data processing request, the general service scenario or service governance scenario in which the distributed coordinator operates is determined, wherein the service governance scenario is an extension of the general service scenario.

4. The method of claim 3, wherein, The data includes string data, and according to the storage strategy, the data is stored in the memory of the distributed coordinator using the data pool, including: In the general service scenario, the following steps are performed in the memory of the distributed coordinator according to the storage strategy: The data pool is used to pool multiple string data with the same value, wherein the pooled string data can be referenced by multiple reference objects in the distributed coordinator; The pooled string data is cached in the data pool.

5. The method of claim 4, wherein, The pooled string data is unique within the data pool and can be accessed by the referenced object within the data pool using a weak reference strategy. The weak reference strategy represents the rules for weakly referencing the pooled string data. The method further includes: In the general service scenario, in response to the absence of the referenced object in the distributed coordinator, the pooled string data is recycled.

6. The method of claim 3, wherein, The data includes storage path data, and the method further includes: In the general service scenario, the storage path data is split to obtain multiple string data located at different path levels.

7. The method of claim 6, wherein, According to the storage strategy, the data is stored in the memory of the distributed coordinator using the data pool, including: The following steps are performed in the memory of the distributed coordinator according to the storage strategy described above: The data pool is used to perform pooling operations on multiple string data at different path levels, wherein the pooled string data can be referenced by multiple reference objects in the distributed coordinator; The pooled string data is cached in the data pool.

8. The method of claim 6, wherein, The data pool is used to perform pooling operations on multiple string data at different path levels, including: According to different path levels, multiple string data are associated to obtain association results; The string data in the association result is pooled using the data pool, wherein the pooled association result can be referenced by multiple reference objects in the distributed coordinator; The pooled association results are cached in the data pool.

9. The method of claim 6, wherein, In the aforementioned general service scenario, the storage path data is split to obtain multiple string data located at different path levels, including: In the general service scenario, determine the tree-like hierarchical structure of the storage path data; According to the tree-like hierarchical structure, the storage path data is split to obtain the string data located at different path levels.

10. The method of claim 3, wherein, The data includes service registration information, and the method further includes: Convert the service registration information into structured data; Fixed metadata is determined from the structured data, wherein the fixed metadata is used to represent metadata that allows multiple service instances of the distributed coordinator to share.

11. The method of claim 10, wherein, According to the storage strategy, the data is stored in the memory of the distributed coordinator using the data pool, including: In the service governance scenario, the following steps are executed in the memory of the distributed coordinator according to the storage policy: The fixed metadata is pooled using the data pool, wherein the pooled fixed metadata can be referenced by multiple reference objects in the distributed coordinator; The fixed metadata, after being pooled, is cached in the data pool.

12. The method of claim 10, wherein, The method further includes: Variable metadata is determined from the structured data, wherein the variable metadata is used to represent the metadata that is allowed to change corresponding to the distributed coordinator; The variable metadata is stored using a string array.

13. The method of any one of claims 1 to 12, wherein, The method further includes: Obtain the monitoring metrics of the distributed coordinator in the service scenario, wherein the monitoring metrics are used to represent the performance parameters of the distributed coordinator associated with the memory; Based on the monitoring metrics, the memory metrics of the distributed coordinator are determined, wherein the memory metrics are used to characterize the memory usage status of the distributed coordinator.

14. The method of claim 13, wherein, The monitoring indicators include at least one of the following: The number of times the nodes in the distributed coordinator are monitored; The number of connections between nodes in the cluster of the distributed coordinator; The number of nodes in the data storage unit corresponding to the cluster; The space occupied by the data snapshots generated by the cluster during storage, and the length of the path for monitoring nodes in the distributed coordinator.

15. A data storage method for a distributed coordinator, comprising: In response to a data processing request from a client, the service scenario in which the distributed coordinator operates is determined, wherein the data processing request includes data that the distributed coordinator needs to store during data processing, and the service scenario is the scenario to which the services supported by the distributed coordinator belong; The storage level corresponding to the service scenario is determined, wherein the storage level is used to represent the data granularity at which the distributed coordinator stores the data, and different data granularities correspond to different storage levels. In different storage levels, among two adjacent storage levels, the data granularity corresponding to the lower storage level is coarser than the data granularity corresponding to the higher storage level. Obtain the storage policy set at the storage level corresponding to the service scenario, wherein the storage policy is used to represent the rules by which the distributed coordinator stores the data using a constant pool, the constant pool allows the data to be referenced in the distributed coordinator, and different service scenarios correspond to different storage policies; According to the storage strategy, the data is cached in the memory of the distributed coordinator using the constant pool.

16. A data processing method for a distributed coordinator, comprising: By calling the first interface to obtain a data processing request and responding to the data processing request, the service scenario in which the distributed coordinator runs is determined. The first interface includes a first parameter, the value of which includes the data processing request. The data processing request includes the data that the distributed coordinator needs to store during the data processing process. The service scenario is the scenario to which the service supported by the distributed coordinator belongs. The storage level corresponding to the service scenario is determined, wherein the storage level is used to represent the data granularity at which the distributed coordinator stores the data, and different data granularities correspond to different storage levels. In different storage levels, among two adjacent storage levels, the data granularity corresponding to the lower storage level is coarser than the data granularity corresponding to the higher storage level. Obtain the storage policy set at the storage level corresponding to the service scenario, wherein the storage policy is used to represent the rules by which the distributed coordinator stores the data using a data pool, the data pool allows the data to be referenced in the distributed coordinator, and different service scenarios correspond to different storage policies; According to the storage strategy, the data is stored in the memory of the distributed coordinator using the data pool; The data stored in the data pool is output by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the data stored in the data pool.

17. A data processing system of a distributed coordinator, comprising: Clients and distributed coordinators, where, The client is configured to send a data processing request, wherein the data processing request includes data that the distributed coordinator needs to store during the data processing process; The distributed coordinator, connected to the client, is configured to respond to data processing requests by: determining the service scenario in which the distributed coordinator operates, wherein the data processing request includes data that the distributed coordinator needs to store during data processing; and the service scenario being the scenario to which the services supported by the distributed coordinator belong. The coordinator then determines the storage level corresponding to the service scenario, wherein the storage level represents the data granularity at which the distributed coordinator stores the data, with different data granularities corresponding to different storage levels. Within adjacent storage levels, the data granularity of the lower storage level is coarser than that of the higher storage level. The coordinator then obtains a storage policy set at the storage level corresponding to the service scenario, wherein the storage policy represents the rules by which the distributed coordinator uses a data pool to store the data, and the data pool allows the data to be referenced within the distributed coordinator. Different service scenarios correspond to different storage policies. Finally, the coordinator stores the data in its memory using the data pool according to the storage policy.

18. A computing device, comprising: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 16.

19. An electronic device comprising: Memory, which stores executable programs; A processor, connected to the memory via a bus, is used to run the program, wherein the program, when running, executes the method according to any one of claims 1 to 16.

20. A computer readable storage medium comprising a stored executable program, wherein, When the executable program is executed, it controls the device containing the computer-readable storage medium to perform the method according to any one of claims 1 to 16.

Images