Data processing method and apparatus for es cluster, electronic device and storage medium

Abstract

A data processing method for an ES cluster, the ES cluster comprising a network layer. The method comprises: when the network layer receives a request sent by an ES client, determining the type of the request (S101); if it is determined that the type of the request is a write type, on the basis of at least one of a preset first relationship and the current available memory of the network layer, determining whether to allow the network layer to cache the request of the write type (S102a); and if it is determined that the type of the request is a query type, allowing the network layer to cache the request of the query type (S102b).

Description

Data processing methods, devices, electronic equipment, and storage media for ES clusters

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on January 23, 2025, with application number 202510113875.0, entitled "Data Processing Method, Apparatus, Electronic Device and Storage Medium for ES Cluster", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of cloud technology, and more specifically, to a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product for an Elasticsearch cluster. Background Technology

[0004] An Elasticsearch cluster (ES cluster) is an open-source, distributed search engine server cluster built on Lucene. An ES cluster receives business requests from ES clients and processes these requests using the combined indexes and search capabilities of multiple nodes within the cluster.

[0005] Elasticsearch clusters include various circuit breakers to prevent business operations from filling up memory. Each circuit breaker specifies a limit on the amount of memory that can be used.

[0006] When a business request enters the Elasticsearch cluster, it is first cached at the network layer. However, since the circuit breaker cannot limit the memory consumption of the network layer cache, the memory of the network layer may be filled up due to a sudden increase in concurrent requests, leading to Elasticsearch cluster failure. Summary of the Invention

[0007] This application provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product for an ES cluster.

[0008] According to one aspect of the embodiments of this application, a data processing method for an Elasticsearch (ES) cluster is provided, the ES cluster including a network layer, the method comprising:

[0009] When the network layer receives a request from an ES client, it determines the type of the request;

[0010] If the type of the request is determined to be write type, then based on at least one of the preset first relationship and the current available memory of the network layer, it is determined whether the network layer is allowed to cache the write type request;

[0011] If the type of the request is determined to be a query type, then the network layer is allowed to cache the query type request;

[0012] The first relationship is a linear mapping between available memory and cache ratio; the cache ratio refers to the proportion of the number of write-type requests that are allowed to be cached to the number of write-type requests received.

[0013] According to another aspect of the embodiments of this application, a data processing apparatus for an Elasticsearch (ES) cluster is provided, the ES cluster including a network layer, the apparatus comprising:

[0014] The request type determination module is used to determine the type of the request when the network layer receives a request sent by the ES client;

[0015] A write-type request processing module is used to determine, if the type of the request is determined to be write-type, whether to allow the network layer to cache the write-type request based on at least one of a preset first relationship and the currently available memory of the network layer.

[0016] A query-type request processing module is configured to allow the network layer to cache the query-type request if it is determined that the type of the request is a query type.

[0017] The first relationship is a linear mapping between available memory and cache ratio; the cache ratio refers to the proportion of the number of write-type requests that are allowed to be cached to the number of write-type requests received.

[0018] According to another aspect of the present application, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the data processing method for the ES cluster described above.

[0019] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the data processing method of the ES cluster described above.

[0020] According to one aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the data processing method for the ES cluster described above.

[0021] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.

[0023] Figure 1A is a schematic diagram of a typical architecture of an ES cluster provided in an embodiment of this application;

[0024] Figure 1B is a schematic diagram of a data writing process for an ES cluster provided in an embodiment of this application;

[0025] Figure 1C is a schematic diagram of a data query process for an ES cluster provided in an embodiment of this application;

[0026] Figure 1D is a schematic diagram of the working structure of the network layer provided in the embodiment of this application;

[0027] Figure 1E is a schematic diagram of a scenario provided by an embodiment of this application, in which an input data query type request is processed and a query response result is returned;

[0028] Figure 2 is a flowchart illustrating the data processing method for an ES cluster provided in an embodiment of this application;

[0029] Figure 3A is a schematic diagram of a current limiting effect provided by an embodiment of this application;

[0030] Figure 3B is a comparison of write throughput between a dynamic rate limiting method provided in the embodiment of this application and the native rate limiting method of the ES engine when facing different concurrency levels.

[0031] Figure 3C is a comparison chart of write throughput for various rate limiting methods provided in the embodiments of this application;

[0032] Figure 4 is a flowchart of a write-type request processing method provided in an embodiment of this application;

[0033] Figure 5 is a schematic diagram of a rate limiting optimization for the total memory of the coordinating node provided in an embodiment of this application;

[0034] Figure 6 is a schematic diagram of a data processing method for an ES cluster provided in an embodiment of this application;

[0035] Figure 7 is a flowchart illustrating the process of determining whether a request of the network layer cache write type is allowed, according to an embodiment of this application.

[0036] Figure 8 is a schematic diagram of the structure of a data processing device for an ES cluster provided in an embodiment of this application;

[0037] Figure 9 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0039] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.”

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0041] First, let's introduce and explain several terms used in this application:

[0042] 1) Elasticsearch (ES) cluster

[0043] Elasticsearch is a popular distributed, highly scalable, and real-time database and analytics engine that provides the ability to store, search, and analyze data and vectors. ES clusters are widely used for log and time-series data analysis, as well as real-time search and analysis tasks in cloud computing and big data environments. For example, in e-commerce websites, ES clusters can be used to index and search product information to improve the user's shopping experience; in the financial sector, ES clusters can be used to perform real-time analysis and monitoring of transaction logs to identify potential risks and anomalies.

[0044] In an Elasticsearch cluster, each node has one of the following two roles:

[0045] The master node is responsible for metadata management and leader election.

[0046] Data nodes are responsible for data sharding, storage, reading, and writing.

[0047] In an Elasticsearch (ES) cluster, the roles of each node can be freely combined. Figure 1A shows a schematic diagram of a typical ES cluster architecture. As shown, multiple data nodes can interact with one of the multiple master nodes, forwarding metadata requests to each other. Multiple master nodes can also forward metadata requests to each other, and multiple data nodes can also interact, forwarding data read and write requests to each other. Furthermore, each data node includes a primary shard and multiple (two in the figure) secondary shards. It can be understood that a shard is part of an index, used for distributed storage of index data. The primary shard is the basic unit of the index, responsible for storing the index data and handling all index-related read and write operations. The replica shard is a copy of the primary shard, used to improve system availability and fault tolerance.

[0048] 2) Data write process of ES cluster

[0049] Please refer to Figure 1B, which illustrates an exemplary data write process for an Elasticsearch cluster. As shown in the figure, the process includes the following four steps:

[0050] 1. The ES client selects a node as the coordinating node (which can be any master node or data node in the ES cluster, node 1 in the diagram) and sends a write request to the coordinating node;

[0051] 2. The coordinating node calculates which primary shard the data should fall on using a hash algorithm. Based on the data routing table information in the cluster status, the coordinating node forwards write requests to the corresponding actual processing node. In the diagram, node 1 forwards write requests to node 3 for processing. It should be noted that each node in the ES cluster has a complete data routing table, which can provide request routing capabilities.

[0052] 3. Once the primary shard on node 3 has finished writing, write requests are concurrently forwarded to the secondary shard on the corresponding node (in the diagram, the secondary shards corresponding to the primary shard are located on nodes 1 and 2 respectively) for writing.

[0053] 4. After all writes to the replica shards are successful, the primary shard returns a confirmation of successful write to the coordinating node. After receiving the confirmations from the primary and replica shards, the coordinating node returns a response confirming successful write to the client.

[0054] 3) Data query process of ES cluster

[0055] Please refer to Figure 1C, which illustrates an exemplary data query process for an Elasticsearch cluster. As shown in the figure, the process includes the following three steps:

[0056] 1. When a user sends a query request to a node in the ES cluster through the ES client, the node that receives the request is called the coordinating node (which can be any node in the ES cluster, node 3 in the diagram).

[0057] 2. The coordinating node parses the query type request, determines the index, field, query conditions, etc. to be accessed, and calculates the list of shards to be accessed based on the index sharding strategy and query conditions. The coordinating node distributes the query type request to the data nodes where the relevant shards are located (in the diagram, the corresponding slave shards of the primary shard are located on node 1 and node 2 respectively) to execute the subquery task. Both the primary and replica shards can provide query capabilities.

[0058] 3. Data nodes execute query-type requests locally, using the Lucene search engine for efficient search operations. The data nodes return the query results to the coordinating node. In some embodiments, the query results typically only contain the document IDs and sorting field values ​​that meet the conditions to reduce data transmission. The coordinating node aggregates the sharded query subtasks and returns them to the client.

[0059] 4) Network transmission framework for ES clusters – Netty network layer

[0060] Network transmission and communication between nodes in an Elasticsearch cluster are implemented based on the Netty network layer, and its working architecture is shown in Figure 1D:

[0061] The Netty network layer consists of one Boss thread (Boss NioEventLoop) and one Worker event loop group (Worker NioEventGroup). NioEventLoopGroup is equivalent to an event loop group, which contains multiple event loops NioEventLoop. Each NioEventLoop contains one multiplexer (Selector) and one event loop thread.

[0062] The tasks executed in each Boss NioEventLoop loop include:

[0063] Step 1: Poll the Accept event;

[0064] Step 2: Process the Accept I / O event, establish a connection with the ES client, generate a NioSocketChannel, and register the NioSocketChannel to the Selector of a certain Worker NioEventLoop;

[0065] Step 3: Process the tasks in the task queue by running AllTasks.

[0066] Each Worker NioEventLoop performs the following tasks in a loop:

[0067] Step 1: Poll for write, query, and other events;

[0068] Step 2: Handle I / O events, i.e., write, query, etc., and process them when NioSocketChannel is readable or writable.

[0069] Step 3: Process the tasks in the task queue by running AllTasks.

[0070] The Netty network layer provides AutoRead as a flow control tool. This tool is a property set on the Channel, which mainly controls the automatic read operation of the Channel (one of the core components of the Netty network, providing an interaction channel with the underlying I / O service)—whether to perform a read operation after the first read operation is completed. It's important to note that if a NioSocketChannel is registered to a Worker thread for the first time, even if the channel is set to non-autoread, a read operation will still be performed. In other words, each channel will perform at least one read operation.

[0071] However, the Netty network layer also has the following drawbacks:

[0072] Disadvantage 1: It cannot distinguish between control flow and data flow (prone to misjudgment) and cannot perform control based on memory.

[0073] Disadvantage 2: Each adjustment to the automatic read operation involves system calls, which impacts performance. It's important to note that the design shouldn't be such that automatic read operations are immediately disabled when no data can be processed, and immediately enabled when processing resumes. Automatic read operations should have a buffer zone; that is, if the queued data reaches a preset high-water mark, automatic read operations should be disabled, and only enabled when it falls below a low-water mark, thus avoiding frequent enabling and disabling of automatic read operations.

[0074] Disadvantage 3: It may cause ES client timeouts, potentially requiring connection rebuilds and ultimately leading to the creation of a large number of connections.

[0075] Netty's network layer manages the cache memory pool as follows:

[0076] The Netty network layer memory pool has a hierarchical structure consisting of five levels: Arena, ChunkList, Chunk, Page, and Subpage. The order of these levels from largest to smallest is as follows:

[0077] Arena: Represents a memory region. To optimize concurrent access to memory regions, Netty's memory pool consists of an array of Arenas. When allocating memory, each thread selects an Arena for allocation using a round-robin strategy. Internally, an Arena contains two PoolSubpage arrays (tinySubpagePools and smallSubpagePools) and multiple ChunkLists.

[0078] ChunkList: A doubly linked list structure that manages chunks. Each Arena in Netty contains multiple ChunkLists, which are divided according to their memory utilization, such as qInit (0-25%), q025 (25-50%), q050 (50-75%), q075 (75-100%), q100 (100%), etc. The number of chunks in each ChunkList changes dynamically; for example, when the memory utilization of a chunk changes, it may be moved to other ChunkLists.

[0079] Chunk: The unit by which the Netty network layer requests memory from the operating system, and it is also the basis for memory allocation operations. Each Chunk has a default size of 16MB and contains multiple Pages (2048 by default). The Chunk is responsible for managing and organizing the memory allocation and release of its internal Pages.

[0080] Page: A Chunk is a unit used for memory management, with each page being 8KB in size. A page can be further divided into multiple subpages to meet different memory allocation needs.

[0081] Subpage: A memory allocation unit within a page, used to satisfy memory allocation requests smaller than a page. Based on size, subpages can be further divided into tiny and small types. If a subpage is less than 512 bytes, it is classified as tiny; if it is greater than 512 bytes but less than 8KB, it is classified as small. Subpage allocation and deallocation are managed by Pool Subpage arrays, which are stored in Arena's tiny Subpage Pools and small Subpage Pools, respectively.

[0082] 5) Fuse (breaker)

[0083] The Elasticsearch engine includes multiple circuit breakers to prevent business operations from exhausting memory. Each circuit breaker specifies a memory limit on how much memory can be used. However, these limits are not precise enough and currently have the following drawbacks:

[0084] 1. Circuit breakers cannot accurately predict the memory consumption of large queries. If users suddenly query too many large queries, it will cause the memory to be filled.

[0085] 2. When a request enters the Elasticsearch cluster, it is first cached at the network layer. The circuit breaker cannot limit the memory consumption of the network layer cache.

[0086] 3. When the memory is completely full, the GC cannot reclaim the memory, the ES cluster cannot recover on its own, and the operation and maintenance personnel need to restart the node, resulting in a long period of continuous unavailability.

[0087] Currently, the availability of the Elasticsearch cluster is approximately 99%, with two main types of anomalies: one is Old Generation (Old GC) failure, accounting for about 25%, and the other is red (Red) failure, accounting for about 75%.

[0088] Old GC failure: Write operations, queries, and other reasons can cause the JVM to trigger Old GC. When Old GC is short and there are intervals, the cluster will exhibit continuous Old GC.

[0089] RED failure: When multiple nodes are continuously undergoing Old GC, the nodes may leave the cluster, which may lead to RED failure.

[0090] The data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product for ES clusters provided in this application aim to solve the above-mentioned technical problems of the prior art, so as to improve the capability range of fuses, specifically in the following aspects:

[0091] 1) Intercept write requests before the cache completely fills the network layer's memory;

[0092] 2) When the total memory usage of the coordinating node is full, release the memory occupied by the query type request that triggered the memory usage saturation.

[0093] 3) When large queries (requests that consume a lot of memory) reach the memory threshold, or before they fill up the memory usage, large queries will be intercepted.

[0094] 4) After the memory is completely full, the system will automatically restart and recover when the difference between the duration of old GC failure and the duration of normal operation exceeds a specified threshold by monitoring the duration of the failure and normal operation.

[0095] The embodiments of this application can reduce query anomalies and REDs caused by Old GC on one or more nodes in the ES cluster, and improve the availability of the ES engine to 99.9%.

[0096] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0097] Figure 1E is a schematic diagram illustrating a scenario where an input data query type request is processed and a query response result is returned, according to an embodiment of this application. A user terminal can send a data query type request to a server via a network, allowing the server to search for a query response result corresponding to the information in the data query type request. Subsequently, the server can return the determined query response result to the user terminal via the network for display to the user.

[0098] Optionally, the user terminal may specifically include smartphones, tablets, laptops, desktop computers, in-vehicle terminals, wearable devices, etc., but is not limited to these. The user terminal may also be a client with a browser or various applications (including system applications and third-party applications) installed. Optionally, the network may be an Internet of Things (IoT) based on the Internet and / or telecommunications networks. It can be a wired or wireless network, for example, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a cellular data communication network, or other electronic networks capable of information exchange. The user terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions. The server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0099] In the server shown in Figure 1E, documents related to a data query type can be retrieved from the system based on the data query type request sent by the user terminal. For example, in a distributed system, the received data query type request can be forwarded to various nodes in the distributed system for independent querying, and the query results from each node can be integrated to obtain the final query response returned to the user terminal.

[0100] This application provides a method for data processing in an ES cluster, as shown in Figure 2. The method includes:

[0101] S101. When the network layer receives a request sent by the ES client, it determines the type of the request.

[0102] When a request sent by an ES client enters the ES cluster, considering that directly processing the request by ES nodes may lead to network congestion or request loss due to long processing times and resource limitations, the request body is generally cached at the network layer first to improve performance.

[0103] After the Netty network layer caches the request body, it does not directly pass the entire request body to the subsequent processing stage of the Elasticsearch engine. Instead, it passes a reference (or pointer) to the cached data to the coordinating node. The coordinating node will access the cached request body as needed. After the coordinating node determines the actual processing node, these actual processing nodes use the reference to locate and read the data in the cache.

[0104] In this embodiment, after caching the request body, the network layer can parse the request header to determine whether the request type is a write request. Specifically, the network layer needs to use a decoder to convert the byte stream into a higher-level data structure, such as a Java object. For HTTP requests, the network layer may use a decoder such as an Http Request Decoder to decode the byte stream into a Full Http Request object, which contains the request line, request headers, and request body.

[0105] Once the byte stream is decoded into a FullHttpRequest object, the network layer can access the request headers. The request headers are a collection of key-value pairs containing various metadata sent by the client, such as content type, content length, and authentication information. The network layer iterates through these header fields, parsing and processing them as needed.

[0106] During the parsing of request headers, the network layer pays particular attention to the method field in the request line. The HTTP protocol defines various request methods, such as GET, POST, PUT, and DELETE, each corresponding to a different type of operation. For write-type requests, the network layer typically looks for methods indicating write operations, such as POST or PUT. If the method field in the request line matches the expected write method, the network layer can determine that this is a write-type request.

[0107] It should be understood that, for write-type requests, once the write-type request is successfully processed and all relevant shards have been successfully written with data, the execution entity can notify the network layer to release the previously cached request body.

[0108] S102a. If it is determined that the type of the request is a write type, then based on at least one of the preset first relationship and the currently available memory of the network layer, it is determined whether the network layer is allowed to cache the write type request.

[0109] Specifically, if the request type is determined to be write type, when judging solely based on a preset first relationship, the cache ratio is determined from the first relationship according to the current available memory of the network layer. If the cache ratio is greater than a second preset threshold, the cache threshold is obtained by multiplying the cache ratio by a preset number, allowing the network layer to cache the write type request and updating the first count of the current rate limiting stage. If the cache ratio is not greater than the second preset threshold, the network layer is not allowed to cache the write type request. When combining both methods, the decision is made by comprehensively considering both the judgment based solely on the current available memory of the network layer and the judgment based solely on the preset first relationship.

[0110] The first count is a statistical count of the number of write-type requests cached by the network layer during the current rate-limiting phase. During the rate-limiting phase, the first count is compared with a cache threshold to determine whether the network layer can continue to cache write-type requests. The cache threshold is the maximum number of write-type requests allowed to be cached when the network layer is in the rate-limiting phase. It is calculated by multiplying the cache ratio by a preset number and is used to determine whether the network layer can continue to cache write-type requests during the rate-limiting phase. The preset number is the expected number of write-type requests received by the network layer in a single rate-limiting phase, typically determined based on the number of write-type requests the network layer has previously processed under different loads, combined with a comprehensive evaluation of system processing capacity and performance requirements.

[0111] In this embodiment of the application, for write-type requests, considering that the purpose of a write-type request is to write new data to the Elasticsearch cluster, it is necessary to first determine whether the network layer is allowed to cache write-type requests. Only if caching of write-type requests is allowed will the network layer cache the write-type request. If the network layer is not allowed to cache write-type requests, the write-type request will be discarded, and correspondingly, the data in the write-type request will not be written to the Elasticsearch cluster.

[0112] This application embodiment determines whether to allow network layer cache write type requests by using a first relationship and the current available memory of the network layer. The first relationship refers to the linear mapping relationship between available memory and cache ratio. By establishing a linear mapping relationship, a corresponding cache ratio can be found for any possible value of available memory. Compared with establishing a discrete mapping relationship between available memory and cache ratio (that is, establishing a limited number of mapping relationships between available memory and cache ratio), when applied to this application scenario, different cache ratios can be smoothly determined for different available memory, avoiding abrupt interception of requests.

[0113] The caching ratio in this embodiment refers to the ratio of the number of write type requests that are allowed to be cached to the number of write type requests received. For example, if the caching ratio is 10%, that is, if 100 write type requests are received, only 10 of them will be cached.

[0114] In some embodiments, this application embodiment determines the cache ratio corresponding to the current available memory based on the network layer's current available memory and in conjunction with a first relationship, and determines whether to allow the network layer to cache write-type requests based on the cache ratio.

[0115] This application dynamically determines the caching ratio (hereinafter referred to as dynamic rate limiting) based on the current available memory of the network layer and the first relationship. Compared with using a fixed single threshold for rate limiting (i.e. whether to allow network layer cache write type requests), it has a smoother and gentler effect.

[0116] In some embodiments, the first relation can be represented by the following function: y = M × cos(x × π / 5) + N

[0117] Where M and N represent different coefficients, both of which are positive integers, x represents memory utilization, and y represents cache ratio.

[0118] The above function is a variation of the cosine function, specifically obtained by translating and scaling the cosine function. The current limiting effect of the function obtained with M and N both set to 50 is shown in Figure 3A. It can be seen that when the memory utilization rate is 80%, the cache ratio is 100%. As the memory utilization rate increases, the cache ratio decreases smoothly. When it reaches 85%, the cache ratio drops to 0%.

[0119] Please refer to Figure 3B, which exemplarily illustrates a comparison of write throughput between the dynamic rate limiting method of this application and the native rate limiting method of the ES engine when facing different concurrency levels. As shown in the figure, when facing no more than 15 concurrent requests, the throughput of the dynamic rate limiting method and the native rate limiting method in this application are almost identical. However, when facing 16-30 concurrent requests, the throughput of the native rate limiting method drops rapidly to 0, while the throughput of the dynamic rate limiting method provided by this application only drops slightly, and the overall curve remains flat.

[0120] Please refer to Figure 3C, which exemplarily illustrates the dynamic rate limiting method of this application, the native rate limiting method of the ES engine, and the single threshold rate limiting method. In the comparison chart of write throughput for the same concurrency, as shown in the figure, the write throughput of the native rate limiting method quickly drops to 0 over time, while the single threshold rate limiting method shows obvious fluctuations, and these fluctuations still result in the write throughput dropping to 0. The dynamic rate limiting method of this application is relatively balanced overall. Although there are fluctuations, the fluctuation amplitude is very small and completely acceptable.

[0121] In some embodiments, this application can provide a random number generator with a range of 1 to 100. If the quotient of the number generated by the random number generator and 100 is not greater than the cache ratio, then it is determined that the network layer is allowed to cache write-type requests, thereby achieving the effect of randomly caching write-type requests under the current cache ratio. It is understood that the range of the random number generator in the above example is merely illustrative; as long as the ratio of the number generated by the random number generator to the upper limit of the range is not greater than the cache ratio, it can be determined that the network layer is allowed to cache write-type requests.

[0122] S102b. If it is determined that the type of the request is a query type, then the network layer is allowed to cache the query type request.

[0123] For query-type requests, since query-type requests do not add new data to the ES cluster, but only query the data already stored in the ES cluster, in order not to affect the timeliness of the query, this application embodiment allows the network layer to cache without going through a judgment process, so as to avoid affecting the data flow.

[0124] Please refer to Figure 4, which exemplarily illustrates the processing flowchart for write-type requests provided in this embodiment of the application. As shown in the figure, the ES client of this embodiment of the application sends a request to the network layer. When the type of the request is write-type, it obtains the current available memory of the network layer, inputs the current available memory into the first relation, determines the current cache ratio, generates a random number through a random number generator, and calculates the ratio between the random number and the upper limit of the generation range of the random number generator. If the ratio is not greater than the current cache ratio, it instructs the network layer to cache the write-type request. If the ratio is greater than the current cache ratio, it instructs the network layer not to cache the write-type request. When the type of the request is query-type, it directly allows the network layer to cache the query-type request.

[0125] The data processing method for an ES cluster provided in this application determines the type of request when the network layer receives a request from an ES client. On one hand, when the request type is write, it determines whether to allow the network layer to cache the write request based on at least one of a preset first relationship and the current available memory of the network layer. Since the first relationship describes a linear relationship, the caching ratio will change as the available memory changes. Using this dynamic caching ratio to determine whether to allow the network layer to cache write requests not only prioritizes limiting the network layer's memory consumption, but more importantly, ensures that the network layer's write throughput can fluctuate gently and not suddenly drop to zero. This overcomes the problem that circuit breakers cannot limit the network layer's memory consumption. Furthermore, rate limiting for write requests at the network layer can reduce the subsequent performance impact on ES nodes. On the other hand, when the request type is query, to avoid affecting the data flow, a method of directly allowing caching without judgment is adopted to ensure the smooth execution of query requests.

[0126] As can be seen from the above embodiments, a query request involves two layers of processing: the coordinating node and the data node. If a query request needs to scan multiple shards, then the coordinating node needs to collect the shard query results of 10 shards, aggregate them, and then return them to the ES client.

[0127] In practical applications, occasionally a situation may occur where the coordinating node receives the shard query result of the third shard from a query type request sent by a data node, triggering the parent breaker, which has reached 90% total memory usage.

[0128] The total memory circuit breaker is the highest-level circuit breaker in an Elasticsearch cluster. It monitors and manages the JVM heap memory usage across the entire cluster. When the cluster's memory usage reaches a preset threshold, the total memory circuit breaker triggers, causing the cluster to stop accepting new client requests and return a circuit breaker exception. This mechanism helps prevent cluster resource exhaustion, request backlog, and performance degradation, thereby improving cluster stability.

[0129] Generally, the circuit breaker is triggered when the cluster's JVM heap memory usage reaches the threshold set by the total memory circuit breaker. After triggering, the cluster stops accepting new requests and returns a circuit breaker exception. This mechanism prevents the system from crashing due to excessive resource consumption by limiting memory usage. However, although the cluster stops accepting new requests, already accepted requests still need to be processed. That is, the memory of the cached query results for the shards of that query type in the coordinating node is not immediately released but continues to wait for the query results of the remaining 7 shards to return. During this waiting period, the total memory usage of the coordinating node may continue to rise to 100%. Understandably, once the total memory usage of the coordinating node reaches 100%, it will have a series of negative effects, such as:

[0130] Performance degradation: Memory is one of the critical resources for the normal operation of an Elasticsearch system, directly affecting data processing speed and query performance. When memory utilization reaches 100%, the system cannot allocate more memory to new requests or tasks, resulting in a significant decline in overall performance.

[0131] Service interruption: High memory usage may prevent the coordinating node from handling new requests, potentially leading to service interruption. This will prevent users from accessing the Elasticsearch system, impacting the normal operation of services.

[0132] Data loss risk: While reaching 100% memory utilization does not directly lead to data loss, the resulting service interruptions or performance issues may indirectly affect data integrity and availability. For example, in the event of insufficient memory, the system may be unable to process write requests in a timely manner, resulting in data loss or delayed writes.

[0133] Increased garbage collection pressure: When memory is insufficient, the Java Virtual Machine (JVM) triggers garbage collection to attempt to free up unused memory. However, frequent garbage collection consumes significant CPU resources, further degrading system performance. In extreme cases, garbage collection may fail to release enough memory in time, leading to system crashes.

[0134] Uneven node load: In a cluster environment, the coordinating node is typically responsible for distributing requests to data nodes. When a coordinating node reaches 100% memory utilization, it may be unable to handle any more requests, causing these requests to be forwarded to other coordinating nodes or data nodes. This can lead to uneven node load across the cluster, impacting overall performance and stability.

[0135] To address the aforementioned issues, as an optional embodiment based on the above embodiments, the ES cluster further includes a coordinating node, and the method further includes:

[0136] When the coordinating node caches the sharded query results of requests for each query type, the total memory circuit breaker is triggered, and the target request of the query type that triggers the total memory circuit breaker is determined.

[0137] The coordinating node is instructed to terminate the target request and release the memory occupied by the sharded query results of the target request.

[0138] In other words, in this embodiment of the application, when it is determined that the total memory usage of the coordinating node reaches the alarm condition (i.e., the total memory circuit breaker is triggered), the coordinating node is no longer allowed to continue waiting for the subsequent shard query results of the target request of the query type that triggered the total memory circuit breaker. Instead, the coordinating node is immediately instructed to terminate the target request and release the memory occupied by the shard query results of the target request, so as to ensure the normal operation of the total memory of the coordination interface and avoid continuous Old GC failure.

[0139] In some embodiments, instructing the coordinating node to terminate a target request may involve instructing the coordinating node to send a termination request to the data nodes containing each shard involved in the target request, so that the data nodes receiving the termination request stop obtaining the shard query results for the corresponding shard. In some embodiments, considering that some data nodes may have already sent shard query results to the coordinating node before triggering the total memory circuit breaker, the coordinating node may also only be instructed to send the termination request to those data nodes that have not yet returned shard query results.

[0140] Please refer to Figure 5, which exemplarily illustrates a schematic diagram of rate limiting optimization for the total memory of the coordinating node in an embodiment of this application. As shown in the figure, the coordinating node interacts with multiple data nodes and caches the sharded query results of two query types. As the number of cached sharded query results of query types increases, the memory occupancy of the coordinating node also increases. When the coordinating node buffers the sharded query result of shard m+k of query type 1, the memory occupancy reaches 90%, triggering the total content circuit breaker. This instructs the coordinating node to terminate query type 1 and release the memory occupied by all sharded query results of query type 1. Afterward, as the memory occupancy drops to a normal level (50%), the coordinating node has enough memory to store the sharded query results of query type 2.

[0141] It should be noted that the above embodiments provide a rate limiting optimization method for the total memory of the coordinating node, which releases the query type requests that just trigger the total memory circuit breaker. In reality, the query type requests may not be the query type requests that consume the most memory. Therefore, it is necessary to find a rate limiting optimization scheme for individual query type requests.

[0142] In an Elasticsearch cluster, when executing sharded queries, the query results often need to be transferred between different nodes, especially when transferring data from shard nodes (also known as data nodes or worker nodes) to the coordinating node (also known as the master node or control node). To ensure an efficient and reliable transmission process, the data objects in memory need to be converted into a format suitable for network transmission. This process is called "serialization".

[0143] Serialization is the process of converting data objects in memory into a continuous bitstream (byte stream). This conversion not only facilitates data transmission over networks but also functions as a compression mechanism, reducing data size and improving transmission efficiency to some extent. The specific implementation of serialization typically depends on the structure of the data object and the programming language used, but the core goal remains the same: to generate a compact, standardized byte representation that can be unambiguously exchanged between different systems or processes.

[0144] Once the shard nodes serialize the query result data object into a byte stream, this data can be sent to the coordinating node via the network connection. During network transmission, the byte stream undergoes encapsulation and routing to ensure that the data arrives at the target node accurately.

[0145] After receiving these byte streams, the coordinating node needs to perform a process that is the reverse of serialization: deserialization. Deserialization is the process of converting the received byte streams back into data objects in memory. This step is crucial for the coordinating node because it needs to restore the original data structure for subsequent data processing or merging of query results.

[0146] Serialization and deserialization mechanisms play a crucial role in the transmission of sharded query results in distributed database systems. They not only ensure efficient and reliable data transmission between different nodes but also guarantee data consistency and integrity, making them an indispensable part of building high-performance distributed systems.

[0147] The rate limiting optimization scheme for a single query type provided in this application embodiment is from the stage where the data node sends the bit stream of the fragmented query result to the coordinating node and then stores the fragmented query result (after deserialization) in memory.

[0148] Based on the above embodiments, as an optional embodiment, this application also provides a scheme for rate limiting optimization for requests of a single query type, specifically:

[0149] In this embodiment of the application, for each query type of request, the estimated memory occupied by the coordinating node for caching the query results of each shard of the request is obtained based on the bit stream of the query results of each shard of the query type of the request.

[0150] Based on the estimated memory usage of the sharded query results of the request of the query type cached by the coordinating node and the memory usage of the coordinating node, determine whether to allow the coordinating node to cache the sharded query results of the request of the query type.

[0151] Specifically, for each shard query result of a query type request, the product of the bitstream size of the shard query result and the memory expansion coefficient λ can be used as the expected memory occupied by the shard query result. When determining whether the coordinating node is allowed to cache the shard query results of the query type request, each time the bitstream of the shard query result of the query type request is detected, the expected memory occupied by the detected shard query results of the query type request is accumulated to obtain a cumulative prediction value. If the sum of the memory currently occupied by the coordinating node and the cumulative prediction value is not less than a preset proportion (e.g., 80%) of the coordinating node's memory, the coordinating node is instructed to terminate the query type request corresponding to the shard query result; if the sum of the memory currently occupied by the coordinating node and the cumulative prediction value is less than the preset proportion of the coordinating node's memory, the coordinating node is not instructed to terminate the query type request corresponding to the shard query result.

[0152] The cumulative predicted value is the accumulated value of the memory expected to be occupied by the detected fragmented query results each time a fragmented query result bitstream of a query type request is detected during the processing of query type requests. By comparing the sum of the current memory occupied by the coordinating node and the cumulative predicted value with a preset proportion of the coordinating node's memory, it is determined whether the coordinating node is allowed to continue caching the fragmented query results of query type requests.

[0153] This application embodiment can determine the memory inflation coefficient λ based on experience. The memory inflation coefficient is equal to the memory consumption of a single query type request divided by the byte stream size of the single request. Therefore, in this application embodiment, for each fragment query result of a query type request, the product of the bit stream size of the fragment query result and the memory inflation coefficient can be used as the expected memory occupied by the fragment query result. This application embodiment can determine the memory inflation coefficient λ through multiple experiments. Specifically, multiple different query type requests are selected, and the memory consumption and byte stream size of each request are measured respectively. Then, the memory consumption of each request divided by its byte stream size is calculated, and finally, the average value of these values ​​is taken as the memory inflation coefficient λ. The memory inflation coefficient is equal to the memory consumption of a single query type request divided by the byte stream size of the single request. Therefore, in this application embodiment, for each fragment query result of a query type request, the product of the bit stream size of the fragment query result and the memory inflation coefficient can be used as the expected memory occupied by the fragment query result.

[0154] By obtaining the estimated memory usage of each shard query result and the memory usage of the coordinating node, it can be determined whether the coordinating node has enough memory to store all shard query results for a request of that query type. For ease of description, memory usage percentage is used. If the estimated memory usage percentage for all three shard query results of a query type is 5%, then the estimated memory usage percentage for the coordinating node to store all shard query results for that query type is 15%. In one scenario, if the coordinating node's memory usage percentage for storing all shard query results for that query type has already reached 90%, then... Clearly, storing all the sharded query results for this query type would cause memory usage to exceed 100%, so it is determined that the coordinating node is not suitable for buffering all the sharded query results for this query type, leading to an instruction to the coordinating node to terminate the request for this query type. In another case, if the coordinating node's memory usage is only 10% at this time, i.e., without storing all the sharded query results for this query type, and even if it stores all the sharded query results for this query type, the memory usage is only 25%, which is a very healthy memory usage. Therefore, the coordinating node is allowed to cache each sharded query result of the query type request.

[0155] It should be understood that if the coordinating node is allowed to cache the shard query results of a query type request, then the coordinating node needs to deserialize the bitstream of each shard query result and restore the original data structure buffer to memory.

[0156] This application embodiment can flexibly limit the rate based on the available memory of the coordinating node and the size of the shard query results. Users of the ES cluster no longer need to worry about the maximum memory limit. As long as there is enough memory, business needs can be met to the maximum extent.

[0157] The embodiments of this application reveal the following main reasons for the persistent Full GC problem:

[0158] Reason 1: For nodes with smaller memory, the memory occupancy may increase from 50% to 100% before the breaker is triggered and the node's memory occupancy is full.

[0159] Reason 2: The fuse cannot accurately estimate memory consumption, and the actual memory consumption may be too high. When the fuse is triggered, the memory occupancy rate is already full.

[0160] Reason 3: When the circuit breaker is triggered, write or read / write requests have not ended and continue to occupy memory, which may further consume memory.

[0161] Therefore, this application also relates to a fallback solution in case the above three optimization measures fail to achieve the desired effect—the optimization for Old GC failures includes the following two points:

[0162] 1. Monitor the duration of old GC failures;

[0163] 2. If the duration of an old GC failure exceeds a preset threshold, the node will be restarted.

[0164] This application embodiment separately calculates the first duration of an Old Generation Garbage Collection (Old GC) failure and the second duration of normal operation for the JVM virtual machine on the ES node. When the difference between the first and second durations exceeds a preset value, all requests processed by the ES node are terminated, and the first and second durations are reset. The preset value can be determined by analyzing the historical operating data of the ES node. Specifically, the operating data of the ES node under different loads is collected, and the difference between the failure duration and the normal operation duration when an Old GC failure occurs, as well as the impact of these differences on system performance, are calculated. Based on the maximum performance degradation that the system can tolerate, a suitable difference is determined as the preset value.

[0165] Whenever an old GC failure occurs in the JVM virtual machine of a single ES node (which can be a coordinator node or a data node), the first duration of the old GC failure is recorded. For ease of understanding, this first duration can be regarded as "debt." During normal operation, the ES node consumes this "debt." By monitoring the "debt" in real time, when the "debt" exceeds a preset duration, the requests being processed by the ES node experiencing the old GC failure are terminated. For example, if an ES node's JVM virtual machine experiences an old GC failure for 200 milliseconds (the first duration), the ES node will increment its debt counter by 200 milliseconds. The second duration spent by the ES node running its program code normally "pays off" all accumulated debt. When the debt counter of the JVM virtual machine exceeds a <threshold> seconds during the old GC failure, all requests processed by the ES node will be terminated, and the first and second durations will be reset.

[0166] Please refer to Figure 6, which exemplarily illustrates a data processing method for an ES cluster according to an embodiment of this application. As can be seen from the figure, the data processing method of this embodiment is divided into four levels:

[0167] Level 1: Rate limiting optimization for the network layer

[0168] When a write request enters the Elasticsearch cluster, the request body is first cached at the network layer, and then the reference is passed to the Elasticsearch node. Therefore, this embodiment first optimizes the network layer by rate limiting. The idea is to determine whether to allow the network layer to cache the request based on the type of the request, the preset mapping relationship, and the current remaining memory of the network layer.

[0169] This application embodiment optimizes rate limiting for write-type requests at the network layer, including:

[0170] A. Parse the request body while it is cached at the network layer to reduce performance impact;

[0171] B. Only intercept write-type requests, and do not intercept non-write-type requests (such as query-type requests), thereby avoiding impact on the data flow;

[0172] C. Smooth rate limiting is achieved based on the number of real-time write-type requests, using at least one of the preset first relationship and the currently available memory of the network layer.

[0173] Level 2: Rate limiting optimization for the total memory of the coordinating node

[0174] The coordinating node will simultaneously aggregate the responses (sharded query results) of multiple query types. During the aggregation process, the total memory usage of the coordinating node may be exhausted, which may lead to the overall unavailability of the coordinating node. Therefore, the second level of this application considers rate limiting optimization of the total memory of the coordinating node. The idea of ​​this application for rate limiting optimization of the total memory of the coordinating node is to trigger the total memory circuit breaker when the coordinating node caches the sharded query results of various query types, terminate the target request that triggered the total memory circuit breaker, and release the memory occupied by the cached sharded query results of the target request.

[0175] Therefore, by capturing the parent breaker during sharding request processing, even in the event of partial failure, requests of the query type that triggered the breaker should be rejected to ensure node stability. Specifically:

[0176] 1. Catching the Parent CircuitBreakerException exception, which is a total memory circuit breaker exception encountered on the coordinating node. This exception usually occurs when the memory usage of the Elasticsearch cluster exceeds the preset limit, causing the circuit breaker to trigger and thus preventing further memory usage to protect system stability;

[0177] 2. Intercept query-type requests that trigger the Parent breaker, causing these requests to be terminated quickly, memory to be released rapidly, and a response to be returned to the user.

[0178] Level 3: Rate limiting optimization for requests of a single query type

[0179] Level 2 optimizes rate limiting from the perspective of the overall memory usage of the coordinating node. However, this optimization method can release the query type requests that trigger the total memory circuit breaker. In layman's terms, it is like removing the last straw that breaks the camel's back. But usually, the query type requests that trigger the total memory circuit breaker are not the query type requests that consume the most memory. Therefore, this application goes further to optimize rate limiting for single query type requests in Level 3. The idea is to predict the memory occupied by the sharded query results of the query type requests cached by the coordinating node for each query type request, and combine this with the memory already occupied by the coordinating node to determine whether to terminate the query type request.

[0180] The key design points are as follows:

[0181] Estimate the memory consumption of query type requests: Estimate the size of the deserialized data by inflating the value and save it to the response.

[0182] Accumulation: Accumulate the predicted memory for each received shard query result, and determine whether the accumulated amount triggers a request to terminate the query type.

[0183] Level 4, also the fallback level: for old GC failures.

[0184] If an old GC failure still occurs in a query node after the first three levels of processing, then a fallback method is used in level 4. The idea is to count the first duration of the old GC failure in the JVM virtual machine of the ES node and the second duration of normal operation. If the difference between the first duration and the second duration is greater than a preset value, then all requests processed by the ES node are terminated.

[0185] Based on the above embodiments, this application embodiment determines whether to allow the network layer to cache the write-type request based on at least one of a preset first relationship and the currently available memory of the network layer, including the following four cases:

[0186] Case 1: If it is determined that the available memory of the network layer is not less than the first preset threshold, then the network layer is determined to be in the non-rate-limiting stage, and the network layer is allowed to cache the write-type request.

[0187] The first preset threshold is a memory standard used to determine whether the network layer has entered the rate-limiting phase. It can be determined by statistically analyzing the minimum available memory value when the network layer is running normally and there is no risk of memory exhaustion through multiple experiments; alternatively, a memory utilization threshold can be set, and the product of the actual memory capacity of the network layer and this threshold can be used as the first preset threshold.

[0188] The non-rate-limited phase refers to the phase in Elasticsearch cluster data processing where the network layer's available memory is not less than a first preset threshold. In this phase, the network layer is allowed to freely cache write requests without rate-limiting them.

[0189] The first preset threshold can be determined through multiple experiments, statistically analyzing the minimum available memory value when the network layer operates normally without the risk of memory exhaustion. Specifically, under different system loads and operating environments, the memory usage of the network layer is tested multiple times. The available memory value when the network layer operates normally without the risk of memory exhaustion in each test is recorded, and the minimum of these values ​​is taken as the first preset threshold.

[0190] In other words, this application embodiment will pre-set a first preset threshold for available memory. If the network layer receives a request and detects that the current available memory of the network layer is not less than the first preset threshold, it indicates that the memory usage of the network layer is within the normal range and the network layer can be allowed to cache the write type request.

[0191] Since the memory size of the network layer may change with system updates, in order to ensure that the rate limiting optimization of the network layer is not affected, this application embodiment can set a threshold for memory occupancy, such as 90%, so that the product of the actual memory capacity of the network layer and the threshold for memory occupancy is used as the first preset threshold.

[0192] Case 2: If it is determined that the available memory of the network layer is less than the first preset threshold, then it is determined whether the network layer is in the rate limiting stage.

[0193] The rate limiting phase refers to the stage in Elasticsearch cluster data processing where, when the available memory at the network layer is less than a first preset threshold, conditional caching of write requests is implemented to control network layer memory consumption. During this phase, the caching ratio is determined based on the current available memory at the network layer and a first preset threshold to limit the number of cached write requests.

[0194] If it is determined that the network layer is not in the rate limiting stage, then the stage in which the network layer is located is updated to the rate limiting stage. Based on the current available memory of the network layer and the first relationship, the current cache ratio is determined. If the cache ratio is greater than the second preset threshold, then the cache threshold is obtained by multiplying the cache ratio and the preset number, allowing the network layer to cache the write type of the request, and the first count of the current rate limiting stage is updated.

[0195] The second preset threshold is a lower limit for the cache ratio. It is determined through multiple experiments, considering the impact of different cache ratios on request caching and system performance under different network layer memory usage conditions; alternatively, it can be determined based on past experience, combined with system performance requirements and network layer processing capabilities. When the cache ratio exceeds this value, a certain amount of request caching is considered acceptable. Specifically, the impact of different cache ratios on request caching and system performance can be tested through multiple experiments under different network layer memory usage conditions, and a suitable lower limit for the cache ratio can be determined as the second preset threshold based on the experimental results; or, a suitable lower limit for the cache ratio can be determined based on past experience, combined with system performance requirements and network layer processing capabilities.

[0196] If it is determined that the available memory of the network layer is less than the first preset threshold, it indicates that the available memory of the network layer is low, and conditional rate limiting is required for the received write-type requests. This embodiment further determines whether the network layer is already in the rate limiting stage. It should be noted that when the network layer is in the rate limiting stage, this embodiment starts to count the first count, which is the count of write-type requests cached by the network layer in the current rate limiting stage. If the network layer is not yet in the rate limiting stage, the stage of the network layer will be updated to the rate limiting stage, and the first count will be set to 0. Then, based on the current available memory of the network layer and the first relationship, the caching ratio will be determined. At the same time, this embodiment can set a second preset threshold. In some cases, the value of the second preset threshold can be 0, that is, the obtained rate limiting ratio is greater than 0, which is considered as a certain number of write-type requests can be cached to the network layer from the current moment. Since the write-type request being processed at this time is the first write-type request in the current rate limiting stage, the network layer is allowed to cache the write-type request, and the first count of the current rate limiting stage is updated from 0 to 1. It should be understood that if the cache ratio is not greater than the second preset threshold, it means that it is not suitable to cache write-type requests at present, so the network layer will be instructed not to cache write-type requests.

[0197] Furthermore, when the cache ratio exceeds the second preset threshold, this application also needs to multiply the cache ratio by a preset number to obtain the cache threshold. The preset number is the estimated number of write-type requests received by the network layer during a rate-limiting phase, configured in this application embodiment. This value can be an empirical value. The empirical value can be determined by referring to the number of write-type requests processed by the network layer under different load conditions in the past, combined with the system's processing capacity and performance requirements, and a suitable estimated number is determined as the preset number after comprehensive evaluation. This application embodiment provides a scheme to obtain the maximum number of write-type requests allowed to be cached (i.e., the cache threshold) when the network layer is in the rate-limiting phase by setting a preset number. For example, if the cache ratio is 20 and the preset number is 100, it means that after the network layer updates to the rate-limiting phase, the first 20 write-type requests are allowed to be received, and caching is abandoned from the 21st write-type request onwards.

[0198] Case 3: If it is determined that the network layer is in the rate limiting phase and the first count has not reached the cache threshold, then the network layer is allowed to cache the write-type request and update the first count.

[0199] In scenario 3, if it is determined that the network layer is in a rate-limiting phase, and the first count has not yet reached the cache threshold, it means that the network layer can continue to cache write-type requests. Taking the cache ratio of 20 and the preset quantity of 100 as an example, if the current first count has not yet reached 20, the network layer can continue to cache write-type requests and increment the first count by 1. Scenario 4: If it is determined that the network layer is in a rate-limiting phase and the first count has reached the cache threshold, then the network layer is not allowed to cache write-type requests.

[0200] In scenario 4, if it is determined that the network layer is in the rate limiting phase and it is found that the first count has reached the cache threshold, it means that the network layer has already cached enough write-type requests in the current rate limiting phase, and it is not suitable to continue caching new write-type requests. Therefore, the network layer does not need to cache the write-type request.

[0201] The embodiments of this application can combine the available memory of the network layer to smoothly determine whether the network layer should cache write-type requests, avoiding frequent allowing or denying of network layer cached write-type requests.

[0202] Please refer to Figure 7, which exemplarily illustrates a flowchart of an embodiment of this application for determining whether a request of the network layer cache write type is allowed. As shown in the figure, it includes:

[0203] S201. Determine whether the current available memory of the network layer is less than the first preset threshold. If the current available memory is less than the first preset threshold, then execute S202. If the current available memory is not less than the first preset threshold, then execute S210.

[0204] S202. Determine whether the network layer is in the rate limiting stage. If the network layer is not in the rate limiting stage, execute S203. If the network layer is in the rate limiting stage, execute S208.

[0205] S203. Update the stage of the network layer to the rate limiting stage;

[0206] S204. Determine the current cache ratio based on the current available memory of the network layer and the first relationship;

[0207] S205. Determine whether the cache ratio is greater than the second preset threshold. If the cache ratio is greater than the second preset threshold, then execute S206. If the cache ratio is not greater than the second preset threshold, then execute S209.

[0208] S206. Obtain the cache threshold based on the product of the cache ratio and the preset number;

[0209] S207. Allow the network layer to cache the write type of the request, and update the first count of the current rate limiting phase;

[0210] S208. Determine whether the first count has reached the cache threshold. If the first count has not reached the cache threshold, execute S207. If the first count has reached the cache threshold, execute S209.

[0211] S209. The network layer is not allowed to cache the write type of the request;

[0212] S210, Allow the network layer to cache the write type of the request.

[0213] Based on the above embodiments, this application embodiment has two methods to update the network layer phase from a rate-limiting phase to a non-rate-limiting phase:

[0214] Method 1: When the network layer receives a write-type request each time, determine whether the available memory of the network layer is not less than the first preset threshold. If it is not less than the first preset threshold, then determine that the network layer is in the non-rate-limiting stage.

[0215] Method 2: After the network layer is updated to the rate-limiting phase, when the number of received write-type requests reaches a preset number, the network layer is updated from the rate-limiting phase to the non-rate-limiting phase. Further, for this method, this embodiment of the application also starts counting a second count after the network layer enters the rate-limiting phase. Each time the network layer receives a write-type request, the second count is incremented by 1. The second count is a statistical count of the number of received write-type requests after the network layer enters the rate-limiting phase. When the second count reaches a preset number, the network layer can be updated from the rate-limiting phase to the non-rate-limiting phase.

[0216] This application embodiment can use any of the above methods to determine whether to update the network layer from the rate-limiting stage to the non-rate-limiting stage. For example, taking a preset number of 100 as an example, when the network layer receives the 80th write type request after entering the current rate-limiting stage, if the available memory of the network layer is not less than the first threshold, the network layer can be updated from the rate-limiting stage to the non-rate-limiting stage first, and then the write type request can be cached. Alternatively, it can continue to determine whether to reject caching the write type request based on whether the network layer has already cached the first count of write type requests.

[0217] In some embodiments, the present application embodiments may also set priorities for the above two methods. For example, method 1 may be prioritized in determining whether to update the network layer phase to a non-rate-limiting phase. Further, the present application embodiments may update a preset quantity by statistically analyzing a second count when the network layer updates from a rate-limiting phase to a non-rate-limiting phase multiple times, so that the number of write-type requests rejected by the network layer is basically matched with the amount of write-type requests processed by the network layer.

[0218] Based on the above embodiments, as an optional embodiment, when the coordinating node caches the sharded query results of requests for each query type, a total memory circuit breaker is triggered. The target request for the query type that triggers the total memory circuit breaker includes:

[0219] For each query type of request, after the coordinating node enters the processing response stage of the query type of request, it is determined whether the coordinating node sends a circuit breaker exception message to the target callback function. If it is determined that the coordinating node sends a circuit breaker exception message to the target callback function, then the abnormal circuit breaker indicated by the message is determined.

[0220] For each query type request, if it is determined that the abnormal circuit breaker indicated by the message is the total memory circuit breaker, then the request of that query type is taken as the target request of the query type.

[0221] The target callback function is a function used by the coordinating node to receive circuit breaker exception messages after entering the response processing phase during the Elasticsearch cluster's handling of query-type requests. By determining whether the coordinating node sends a circuit breaker exception message to this function, the type of circuit breaker exception that was triggered can be identified.

[0222] It's important to note that the coordinating node goes through several phases during request and response processing, one of which is the `handleResponse` phase. In the `handleResponse` phase, the coordinating node performs a series of operations to process the received response data. During this process, the coordinating node also periodically or as needed checks its own memory usage. Memory usage is a key metric reflecting the system's current memory load. When the coordinating node detects that memory usage has reached or exceeded a preset threshold (e.g., 90%), it indicates that the system may be at risk of running out of memory. To prevent system crashes or severe performance degradation due to memory overflow, the coordinating node triggers a protection mechanism called the parent breaker.

[0223] Once the parent breaker is triggered, the coordinating node performs a series of fault recovery and protection measures. One key measure is sending an exception message to registered callback functions (such as onShardFailure). This exception message is encapsulated in a CircuitBreakerException, which contains details of the circuit breaker being triggered.

[0224] The CircuitBreakerException message contains several key pieces of information to help developers or system administrators diagnose problems and take appropriate action. One of the most important pieces of information is the type of circuit breaker that triggered the exception, namely the parent breaker. This indicates that the currently triggered circuit breaker is the parent breaker protecting the entire system or a set of critical resources.

[0225] In an Elasticsearch cluster, the `onShardFailure` callback is a mechanism for handling shard failures, while `CircuitBreakerException` is a type of exception that the Elasticsearch engine may encounter when processing requests. When the `onShardFailure` callback receives a `CircuitBreakerException`, it usually means that the circuit breaker has been triggered due to resource limitations (such as memory or index size) to protect the system from resource exhaustion.

[0226] Therefore, in this embodiment of the application, the onPhaseFailure function can be used to intercept only the query type requests that trigger the breaker, thereby protecting the total memory during the write phase, avoiding continuous Old GC, and releasing the memory consumed by the existing query results. The query type requests that trigger the breaker are quickly terminated, and a Response is returned to the ES client to notify the ES client that the query type request processing has failed.

[0227] Based on the above embodiments, as an optional embodiment, determining whether to allow the coordinating node to cache the query type of the request, based on the expected memory occupied by the sharded query results of the request cached by the coordinating node and the memory usage of the coordinating node, includes:

[0228] Each time a bitstream of the fragmented query result of a request of the query type is detected, the memory expected to be occupied by the fragmented query result of the detected request of the query type is accumulated to obtain a cumulative prediction value.

[0229] If the sum of the memory currently occupied by the coordinating node and the cumulative predicted value is not less than a preset proportion of the memory of the coordinating node, then the coordinating node is instructed to terminate the request for the query type corresponding to the sharded query result.

[0230] If the sum of the memory currently occupied by the coordinating node and the cumulative predicted value is less than a preset proportion of the coordinating node's memory, then the coordinating node is not instructed to terminate the request for the query type corresponding to the sharded query result.

[0231] It's important to note that when the coordinating node forwards a query request to the data nodes containing the required shards, these data nodes perform a search operation and return the shard query results. If the search is successful, they return the shard query results to the coordinating node via the `onShardResult` callback. If a data node fails during the search process or is unable to return shard query results, that node will notify the coordinating node via the `onShardFailure` callback.

[0232] This application can obtain the estimated memory occupied by the sharded query results based on the bit stream of the sharded query results each time the coordinating node receives an onShardResult callback, and accumulate this amount as a cumulative prediction value. Each time the cumulative prediction value is updated, the coordinating node's currently occupied memory and the cumulative prediction value are summed. If the sum is less than a preset proportion (e.g., 80%) of the coordinating node's memory, the coordinating node is allowed to continue receiving sharded query results for the query type. If, after receiving a certain onShardResult callback, it is found that the sum is not less than the preset proportion of the coordinating node's memory, the coordinating node is promptly instructed to terminate the request for the query type corresponding to the sharded query result.

[0233] In some embodiments, the executing entity instructs the coordinating node to terminate the request for the query type corresponding to the sharded query result. This may involve instructing the coordinating node to stop waiting for incomplete sharded query results and returning a termination or error response to the ES client.

[0234] Through testing, this application's embodiments demonstrate that when performing a query operation on a single shard, the size of the returned shard query result is typically within a relatively fixed range, generally around 28,984,750 bytes, which can be approximated to approximately 27MB. It is worth noting that this size is close to the result obtained after performing a memory dump operation on the shard (approximately 30MB), thus verifying the accuracy of the memory size prediction in this application's embodiments.

[0235] In this embodiment of the application, the `onShardResult` function records and accumulates the total size of all sharded query results during the query processing flow. For example, in some cases, the accumulated result size may reach 60MB or 90MB, depending on the number of shards involved in the query and the amount of data returned by each shard. Based on the above accumulated result, this embodiment of the application can calculate the actual memory consumption required to process a single query type request relatively accurately. Although this calculation process has some approximation (such as rounding when converting bytes to MB), its accuracy is usually acceptable in practical applications. Through this method, this embodiment of the application can better understand the system's memory usage, thereby enabling more reasonable resource allocation and optimization.

[0236] It is important to note that current request breaker mechanisms are implemented based on a global breaker service. While this design performs well in most scenarios, it does not provide fine-grained control over requests of individual query types. To further enhance system stability and responsiveness, this embodiment introduces a local, request-specific breaker mechanism during the query process, as described above. This mechanism creates an independent breaker service instance for each search request. This local breaker service is dedicated to recording and limiting the memory resources consumed by a single query type request during processing. In this way, this embodiment implements a true single-request breaker that makes decisions based on the actual load of a single request, thereby preventing the request from affecting the stability of the entire system due to excessive resource consumption.

[0237] In implementing this partial circuit breaker service, this application embodiment predicts the memory size occupied by each onShardResult callback, thereby accurately accumulating the various resource consumption generated by a single request during processing.

[0238] This application provides a data processing device for an Elasticsearch (ES) cluster, as shown in FIG8. The ES cluster data processing device may include: a request type determination module 801, a write-type request processing module 802, and a query-type request processing module 803, wherein...

[0239] The request type determination module 801 is used to determine the type of the request when the network layer receives a request sent by the ES client;

[0240] The write-type request processing module 802 is used to determine whether to allow the network layer to cache the write-type request if it is determined that the type of the request is write-type, based on at least one of a preset first relationship and the currently available memory of the network layer.

[0241] The query type request processing module 803 is configured to allow the network layer to cache the query type request if it is determined that the type of the request is a query type.

[0242] The first relationship is a linear mapping between available memory and cache ratio; the cache ratio refers to the proportion of the number of write-type requests that are allowed to be cached to the number of received write-type requests.

[0243] The apparatus in this application embodiment can execute the method provided in this application embodiment, and the implementation principle is similar. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0244] As an optional implementation, the ES cluster further includes ES nodes, the type of which includes coordinating nodes, and the apparatus further includes:

[0245] The request determination module is used to determine the target request of the query type that triggers the total memory circuit breaker when the coordination node caches the sharded query results of requests for each query type.

[0246] The request release module is used to instruct the coordinating node to terminate the target request and release the memory occupied by the sharded query results of the target request.

[0247] As an optional implementation, the ES cluster further includes ES nodes, the type of which includes coordinating nodes, and the apparatus further includes:

[0248] The memory estimation module is used to obtain, for each query type request, the estimated memory occupied by the coordinating node for caching the query results of each shard of the request based on the bit stream of the query results of each shard of the request of the query type.

[0249] The cache determination module is used to determine whether to allow the coordinating node to cache the shard query results of the request of the query type based on the estimated memory occupied by the shard query results of the request cached by the coordinating node and the memory usage of the coordinating node.

[0250] As an optional implementation, the ES cluster further includes ES nodes, and the apparatus further includes:

[0251] The duration statistics module is used to separately count the first duration of the JVM virtual machine on the ES node experiencing an Old GC failure and the second duration of normal operation.

[0252] The reset module is used to terminate all requests processed by the ES node and reset the first and second durations when the difference between the first duration and the second duration is greater than a preset value.

[0253] As an optional implementation, the write-type request processing module is specifically used for:

[0254] If it is determined that the available memory of the network layer is not less than a first preset threshold, then the network layer is determined to be in a non-rate-limiting phase, and the network layer is allowed to cache the write-type request;

[0255] If it is determined that the available memory of the network layer is less than a first preset threshold, then it is determined whether the network layer is in a rate-limiting phase.

[0256] If it is determined that the network layer is not in the rate limiting stage, the stage in which the network layer is located is updated to the rate limiting stage. Based on the current available memory of the network layer and the first relationship, the current cache ratio is determined. If the cache ratio is greater than the second preset threshold, the cache threshold is obtained by multiplying the cache ratio and the preset number, allowing the network layer to cache the write type of the request, and the first count of the current rate limiting stage is updated.

[0257] If it is determined that the network layer is in a rate-limiting phase and the first count has not reached the cache threshold, then the network layer is allowed to cache the write-type request and update the first count;

[0258] If it is determined that the network layer is in a rate-limiting phase and the first count reaches the cache threshold, then the network layer is not allowed to cache the write-type request.

[0259] As an optional implementation, the request determination module is specifically used for:

[0260] For each query type of request, after the coordinating node enters the processing response stage of the query type of request, it is determined whether the coordinating node sends a circuit breaker exception message to the target callback function. If it is determined that the coordinating node sends a circuit breaker exception message to the target callback function, then the abnormal circuit breaker indicated by the message is determined.

[0261] For each query type request, if it is determined that the abnormal circuit breaker indicated by the message is the total memory circuit breaker, then the request of that query type is taken as the target request of the query type.

[0262] As an optional implementation, the cache determination module is specifically used for:

[0263] Each time a bitstream of the fragmented query result of a request of the query type is detected, the memory expected to be occupied by the fragmented query result of the detected request of the query type is accumulated to obtain a cumulative prediction value.

[0264] If the sum of the memory currently occupied by the coordinating node and the cumulative predicted value is not less than a preset proportion of the memory of the coordinating node, then the coordinating node is instructed to terminate the request for the query type corresponding to the sharded query result.

[0265] If the sum of the memory currently occupied by the coordinating node and the cumulative predicted value is less than a preset proportion of the coordinating node's memory, then the coordinating node is not instructed to terminate the request for the query type corresponding to the sharded query result.

[0266] This application provides an electronic device including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of a data processing method for an Elasticsearch (ES) cluster. Compared with related technologies, this method can achieve the following: When the network layer receives a request sent by an ES client, it determines the type of the request. On the one hand, when the request type is write, it determines whether to allow the network layer to cache the write request based on at least one of a preset first relationship and the network layer's current available memory. Since the first relationship describes a linear relationship, the caching ratio will change with the change in available memory. Using this dynamic caching ratio to determine whether to allow the network layer to cache the write request not only prioritizes limiting the network layer's memory consumption but, more importantly, ensures that the network layer's write throughput fluctuates gently, preventing sudden drops to zero. This overcomes the problem that circuit breakers cannot limit the network layer's memory consumption. Furthermore, by performing rate limiting on write requests at the network layer, the subsequent performance impact on ES nodes can be reduced. On the other hand, when the request type is query, to avoid affecting the data flow, a method of directly allowing caching without judgment is adopted to ensure the smooth execution of query requests.

[0267] In one optional embodiment, an electronic device is provided, as shown in FIG9. The electronic device 4000 shown in FIG9 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0268] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0269] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, bus 4002 is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.

[0270] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0271] The memory 4003 stores computer programs that execute embodiments of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0272] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the steps and corresponding content of the aforementioned method embodiments.

[0273] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0274] In summary, this application provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product for an Elasticsearch (ES) cluster. When the network layer receives a request from an ES client, it determines the request type by parsing the request header. For write requests, it determines whether caching is allowed based on at least one of a preset first relationship (a linear mapping between available memory and cache ratio) and the current available memory of the network layer. This linear mapping allows the cache ratio to change dynamically with available memory. Compared to a discrete mapping, it can smoothly determine the cache ratio for different available memory, avoiding sudden interception of requests. The dynamic cache ratio determination method can precisely control the network layer's caching of write requests, prioritizing the limitation of network layer memory consumption, preventing the network layer memory from being filled due to a sudden increase in concurrent requests, reducing the risk of ES cluster failure due to network layer memory issues, and improving system stability. At the same time, rate limiting of write requests at the network layer can reduce the processing burden of subsequent ES nodes and improve resource utilization. For query requests, caching is directly allowed at the network layer, avoiding unnecessary judgments in the query process and ensuring query timeliness and efficiency.

[0275] Furthermore, when the coordinating node caches the sharded query results of various query types, triggering the total memory circuit breaker, the target request of the query type that triggered the circuit breaker is identified. The coordinating node is then instructed to terminate the target request and release the memory occupied by its sharded query results. The triggering of the total memory circuit breaker indicates that the coordinating node's memory usage has reached a dangerous threshold. If not handled promptly, this can lead to performance degradation of the coordinating node, service interruptions, and even data loss. By quickly terminating the target request and releasing memory, the coordinating node's total memory usage can be prevented from continuously increasing, preventing persistent OldGC failures, ensuring normal total memory usage of the coordinating node, maintaining the stability and availability of the Elasticsearch cluster, and reducing system anomalies caused by memory issues.

[0276] Building upon the above, for each query type request, the estimated memory usage of the coordinating node for caching these shard query results is obtained based on the bitstream of each shard's query result. This, combined with the coordinating node's memory usage, determines whether caching is permitted. This approach allows for real-time monitoring of the coordinating node's memory usage during query processing, predicting the memory consumption of each query type request. By comparing the coordinating node's current memory usage with the sum of cumulative predicted values ​​and a preset ratio of the coordinating node's memory, query type requests that may lead to memory shortages can be identified promptly, and appropriate measures can be taken. When the sum of these two values ​​is not less than the preset ratio, the query type request is terminated in a timely manner to prevent system crashes due to memory exhaustion, thus improving system security and stability. Simultaneously, this flexible rate limiting method eliminates the need for ES cluster users to worry about maximum memory limits; as long as sufficient memory is available, business needs can be maximized, improving resource utilization and system flexibility.

[0277] Simultaneously, the duration of the first OldGC (Old Generation Garbage Collection) failure and the duration of normal operation on the ES (Elasticsearch) node's JVM are recorded separately. When the difference between the first and second durations exceeds a preset value, all requests processed by the ES node are terminated, and both durations are reset. Since prolonged FullGC issues can lead to node memory exhaustion and system performance degradation, monitoring the difference between the OldGC failure duration and the normal operation duration allows for timely detection of abnormal node memory usage. When the difference exceeds the preset value, it indicates that the node's memory problem has severely impacted the normal operation of the system. At this point, terminating all requests and resetting the duration prevents further deterioration of the memory problem from causing system crashes, thus protecting and recovering the ES node and improving the fault tolerance and reliability of the ES cluster.

[0278] If it is determined that the available memory of the network layer is not less than the first preset threshold, then the network layer is determined to be in the non-rate-limiting phase, allowing the network layer to cache write-type requests. The first preset threshold is the memory standard for determining whether the network layer has entered the rate-limiting phase. When the available memory of the network layer is not less than this threshold, it indicates that the network layer's memory usage is within the normal range. At this time, allowing free caching of write-type requests can ensure the system's processing efficiency when memory is sufficient, avoid unnecessary rate-limiting operations affecting the normal processing of write-type requests, and improve system performance.

[0279] If the available memory of the network layer is determined to be less than the first preset threshold, it is then determined whether the network layer is in the rate-limiting phase. If it is not in the rate-limiting phase, the phase of the network layer is updated to the rate-limiting phase. The current cache ratio is determined based on the current available memory of the network layer and the first relationship. If the cache ratio is greater than the second preset threshold, the cache threshold is obtained by multiplying the cache ratio by a preset number, allowing the network layer to cache write-type requests and updating the first count of the current rate-limiting phase. This approach can promptly enter the rate-limiting phase when the network layer is short of memory. By dynamically adjusting the cache ratio and setting the cache threshold, write-type requests are conditionally cached, avoiding excessive consumption of network layer memory. The second preset threshold serves as the lower limit of the cache ratio, ensuring that requests are cached only when the cache ratio is reasonable, improving the rationality and effectiveness of memory use.

[0280] If it is determined that the network layer is in the rate-limiting phase and the first count has not reached the cache threshold, the network layer is allowed to cache write-type requests and update the first count. This ensures that during the rate-limiting phase, as long as the cache threshold has not been reached, the network layer can continue to cache a certain number of write-type requests, maintain the system's write throughput, avoid system performance degradation due to excessive rate limiting, and ensure system processing capacity while controlling memory consumption.

[0281] If the network layer is determined to be in the rate-limiting phase and the first count reaches the cache threshold, then caching of write-type requests is not allowed in the network layer. This effectively controls the memory consumption of the network layer during the rate-limiting phase, prevents the network layer memory from becoming full due to excessive caching of write-type requests, ensures stable use of network layer memory, and improves the stability and reliability of the system.

[0282] When a total memory circuit breaker is triggered in response to sharded query results cached by the coordinating node for various query types, the target request for the query type that triggered the total memory circuit breaker is determined by checking whether the coordinating node sends a circuit breaker exception message to the target callback function after entering the processing response phase for that query type request. If so, the exception circuit breaker indicated by the message is identified; if it is the total memory circuit breaker, then that query type request is designated as the target request. This method accurately identifies the query type request that triggered the total memory circuit breaker. The target callback function, as the key to receiving circuit breaker exception messages, can promptly report abnormal memory usage by the coordinating node, enabling the system to take timely actions to terminate and release memory, improving system response speed and stability, and preventing system failures caused by memory issues.

[0283] Based on the estimated memory usage of sharded query results for a query type request cached by the coordinating node and the coordinating node's memory usage, a decision is made as to whether to allow the coordinating node to cache the query type request. Each time a bitstream of a sharded query result for a query type request is detected, the estimated memory usage of the detected sharded query results is accumulated to obtain a cumulative prediction value. If the sum of the coordinating node's current memory usage and the cumulative prediction value is not less than a preset proportion of the coordinating node's memory, the coordinating node is instructed to terminate the request for that query type corresponding to the sharded query result; otherwise, no termination is instructed. This real-time accumulation and judgment method dynamically monitors the coordinating node's memory usage during query processing, proactively identifying query type requests that may lead to memory shortages and taking timely termination actions to avoid system crashes due to memory exhaustion, thus improving system security and stability. Simultaneously, accurate estimation of the estimated memory usage of sharded query results allows for more rational allocation of memory resources to the coordinating node, improving memory resource utilization.

[0284] In terms of data transmission, Elasticsearch clusters employ serialization and deserialization mechanisms when transmitting sharded query results between different nodes. Serialization converts data objects in memory into a continuous bitstream, facilitating network transmission and acting as a compression mechanism to reduce data size and improve transmission efficiency. Deserialization, on the other hand, converts the received byte stream back into data objects in memory, ensuring efficient and reliable data transmission between different nodes and guaranteeing data consistency and integrity. This is an indispensable part of building high-performance distributed systems.

[0285] After caching the request body at the network layer, a reference to the cached data is passed to the coordinating node instead of the entire request body. The coordinating node accesses the cached request body as needed, and the actual processing node uses the reference to locate and read the cached data. This approach reduces data transmission volume, lowers network bandwidth usage, and improves data processing efficiency.

[0286] There are two methods for updating the network layer from a rate-limited phase to a non-rate-limited phase. Method 1 checks if the available memory of the network layer is not less than a first preset threshold each time a write request is received. If it is, the network layer is in a non-rate-limited phase. This method allows for flexible adjustment of the rate-limiting strategy based on the real-time memory situation of the network layer, ensuring efficient system operation when memory is sufficient. Method 2 updates the network layer to a rate-limited phase after it has been updated, and then updates it to a non-rate-limited phase when the number of received write requests reaches a preset number, and a second count is recorded. This method dynamically adjusts the rate-limiting phase based on the number of requests, improving system adaptability. Furthermore, by counting the second count when the network layer updates from a rate-limited phase to a non-rate-limited phase multiple times, the preset number can be updated to match the number of write requests rejected and cached by the network layer with the processing volume of write requests, further optimizing system performance.

[0287] In optimizing for OldGC failures, the system tracks the first duration of an OldGC failure and the second duration of normal operation on the ES node's JVM. When the difference between the first and second durations exceeds a preset value, all requests processed by the ES node are terminated, and both durations are reset. The preset value is determined through analysis of historical ES node runtime data, allowing for precise judgment based on the node's actual operating conditions. By monitoring the duration of OldGC failures in real time and restarting the node, a fallback mechanism is implemented when other optimization measures fail, preventing severe performance degradation due to OldGC failures and improving the fault tolerance and reliability of the ES cluster.

[0288] The terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0289] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0290] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0291] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A data processing method for an Elasticsearch (ES) cluster, the ES cluster including a network layer, the method comprising: When the network layer receives a request from an ES client, it determines the type of the request; If the type of the request is determined to be write type, then based on at least one of the preset first relationship and the current available memory of the network layer, it is determined whether the network layer is allowed to cache the write type request; If the type of the request is determined to be a query type, then the network layer is allowed to cache the query type request; The first relationship is a linear mapping between available memory and cache ratio; the cache ratio refers to the proportion of the number of write-type requests that are allowed to be cached to the number of write-type requests received.

2. The method according to claim 1, wherein the ES cluster further includes ES nodes, the type of the ES nodes including coordinating nodes, and the method further includes: When the coordinating node caches the sharded query results of requests for each query type, the total memory circuit breaker is triggered, and the target request of the query type that triggers the total memory circuit breaker is determined. The coordinating node is instructed to terminate the target request and release the memory occupied by the sharded query results of the target request.

3. The method according to claim 1 or 2, wherein the ES cluster further includes ES nodes, the type of which includes coordinating nodes, and the method further includes: For each query type request, based on the bitstream of each shard query result of the request for that query type, the estimated memory occupied by the coordinating node for caching each shard query result of the request for that query type is obtained; Based on the estimated memory usage of the sharded query results of the request of the query type cached by the coordinating node and the memory usage of the coordinating node, determine whether to allow the coordinating node to cache the sharded query results of the request of the query type.

4. The method according to claim 3, wherein the ES cluster further includes ES nodes, and the method further includes: The duration of the first Old GC failure and the duration of normal operation of the JVM virtual machine on the ES node were recorded separately. When the difference between the first duration and the second duration is greater than a preset value, all requests processed by the ES node are terminated, and the first duration and the second duration are reset.

5. The method according to any one of claims 1 to 4, wherein determining whether to allow the network layer to cache the write-type request based on at least one of a preset first relationship and the currently available memory of the network layer includes: If it is determined that the available memory of the network layer is not less than a first preset threshold, then the network layer is determined to be in a non-rate-limiting phase, and the network layer is allowed to cache the write-type request.

6. The method according to any one of claims 1 to 5, wherein determining whether to allow the network layer to cache the write-type request based on at least one of a preset first relationship and the currently available memory of the network layer further comprises: If it is determined that the available memory of the network layer is less than a first preset threshold, then it is determined whether the network layer is in a rate-limiting phase. If it is determined that the network layer is not in the rate limiting stage, then the stage in which the network layer is located is updated to the rate limiting stage. Based on the current available memory of the network layer and the first relationship, the current cache ratio is determined. If the cache ratio is greater than the second preset threshold, then the cache threshold is obtained by multiplying the cache ratio and the preset number, allowing the network layer to cache the write type of the request, and the first count of the current rate limiting stage is updated.

7. The method according to claim 6, wherein determining whether to allow the network layer to cache the write-type request based on at least one of a preset first relationship and the currently available memory of the network layer further comprises: If it is determined that the network layer is in a rate-limiting phase and the first count has not reached the cache threshold, then the network layer is allowed to cache the write-type request and update the first count.

8. The method according to claim 6 or 7, wherein determining whether to allow the network layer to cache the write-type request based on at least one of a preset first relationship and the currently available memory of the network layer further comprises: If it is determined that the network layer is in a rate-limiting phase and the first count reaches the cache threshold, then the network layer is not allowed to cache the write-type request.

9. The method according to claim 2, wherein triggering the total memory circuit breaker in response to the coordinating node caching the sharded query results of requests for each query type, and determining the target request of the query type that triggers the total memory circuit breaker, includes: For each query type of request, after the coordinating node enters the processing response stage of the query type of request, it is determined whether the coordinating node sends a circuit breaker exception message to the target callback function. If it is determined that the coordinating node sends a circuit breaker exception message to the target callback function, then the abnormal circuit breaker indicated by the message is determined. For each query type request, if it is determined that the abnormal circuit breaker indicated by the message is the total memory circuit breaker, then the request of that query type is taken as the target request of the query type.

10. The method according to claim 3 or 4, wherein determining whether to allow the coordinating node to cache the request of the query type based on the estimated memory occupied by the sharded query results of the request cached by the coordinating node and the memory usage of the coordinating node includes: Each time a bitstream of the fragmented query result of a request of the query type is detected, the memory expected to be occupied by the fragmented query result of the detected request of the query type is accumulated to obtain a cumulative prediction value. If the sum of the memory currently occupied by the coordinating node and the cumulative predicted value is not less than a preset proportion of the memory of the coordinating node, then the coordinating node is instructed to terminate the request for the query type corresponding to the sharded query result. If the sum of the memory currently occupied by the coordinating node and the cumulative predicted value is less than a preset proportion of the coordinating node's memory, then the coordinating node is not instructed to terminate the request for the query type corresponding to the sharded query result.

11. A data processing apparatus for an Elasticsearch (ES) cluster, the ES cluster including a network layer, the apparatus comprising: The request type determination module is used to determine the type of the request when the network layer receives a request sent by the ES client; A write-type request processing module is used to determine, if the type of the request is determined to be write-type, whether to allow the network layer to cache the write-type request based on at least one of a preset first relationship and the currently available memory of the network layer. A query-type request processing module is configured to allow the network layer to cache the query-type request if it is determined that the type of the request is a query type. The first relationship is a linear mapping between available memory and cache ratio; the cache ratio refers to the proportion of the number of write-type requests that are allowed to be cached to the number of write-type requests received.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the data processing method for an ES cluster according to any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the data processing method for an ES cluster according to any one of claims 1-10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the data processing method for an ES cluster according to any one of claims 1-10.

Images