Request processing system and method, and electronic device
By using a fully asynchronous link mode and parallel processing of container instances on the Kubernetes platform, the performance bottleneck of the traditional synchronous blocking model under high concurrency requests is solved, achieving fast response and efficient resource utilization.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- CHINA TELECOM ARTIFICIAL INTELLIGENCE TECHNOLOGY (BEIJING) CO LTD
- Filing Date
- 2025-10-31
- Publication Date
- 2026-06-18
Smart Images
Figure CN2025131603_18062026_PF_FP_ABST
Abstract
Description
Request processing systems, methods and electronic devices
[0001] Related applications
[0002] This application claims priority to Chinese patent application No. 202411815009.8, filed on December 10, 2024, entitled "A Request Processing System, Method and Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field
[0003] This disclosure relates to the field of computer software development, and more specifically to a request processing system, method, and electronic device. Background Technology
[0004] In modern internet applications, users have increasingly higher demands for service performance and response speed, especially in scenarios involving massive numbers of users and high concurrency requests. Effectively handling these requests has become a significant challenge for technological development. Many related technologies still rely on traditional synchronous blocking input / output models (BIO) and multithreaded processing methods to cope with high concurrency requests.
[0005] Traditional BIO models suffer from several significant problems when handling high-concurrency requests. First, as requests increase, the system needs to launch more threads to process them, leading to a rapid increase in the number of threads. When the number of threads exceeds the number of CPU (Central Processing Unit) cores, multiple threads compete for limited CPU resources, causing frequent context switching. This not only increases the CPU load but also leads to increased response latency, as each context switch requires saving and restoring the thread's state, consuming valuable computing resources. Second, thread blocking is particularly pronounced in I / O (Input / Output) intensive applications. When one thread is waiting for an I / O operation to complete, other requests cannot be processed promptly, resulting in decreased system throughput. Especially under high concurrency, I / O bottlenecks significantly impact overall system performance, failing to meet users' demands for rapid response. Summary of the Invention
[0006] This disclosure provides a request processing system, method, and electronic device.
[0007] In a first aspect, embodiments of this disclosure provide a request processing system, the system including a service calling terminal, an asynchronous I / O module, and a service providing terminal;
[0008] The service invocation terminal is used to initiate an asynchronous remote procedure call request to the service provider for the target request to be processed. The asynchronous remote procedure call request is used to instruct the service provider to process the target request. The processing result of the target request is tracked according to the asynchronous handle, and the link of the next operation is determined.
[0009] The asynchronous I / O module is used to generate the asynchronous handle according to the target request and return the asynchronous handle to the service caller.
[0010] The service provider is used to respond to the asynchronous remote procedure call request, process the target request, and return the processing result of the target request to the service caller through the asynchronous I / O module.
[0011] In some embodiments, the service caller has multiple container instances deployed on the Kubernetes platform;
[0012] The Kubernetes platform evenly distributes the multiple requests obtained by the service caller to the multiple container instances, and the multiple container instances are used to process the asynchronous remote procedure call requests for the multiple requests in parallel.
[0013] In some embodiments, the number of worker threads of the plurality of container instances is equal to the number of container CPU cores.
[0014] In some embodiments, the system further includes a middleware module and a database;
[0015] The service provider is used to asynchronously call the middleware module and distribute multiple requests from the service caller to multiple business processing instances for parallel processing through the middleware module.
[0016] The business processing instance is used to asynchronously call the database to perform data access operations and return the processing results to the service calling end through the asynchronous IO module.
[0017] In some embodiments, the system further includes a network terminal;
[0018] The network terminal is configured to receive the asynchronous remote procedure call request initiated by the service call terminal, and forward the asynchronous remote procedure call request to the service provider terminal to request the service provider terminal to process the target request.
[0019] In some embodiments, the middleware module distributes the multiple requests to the multiple business processing instances using a load balancing strategy of round-robin, random, or least connections.
[0020] In a second aspect, embodiments of this disclosure provide a request processing method applied to the system described in the first aspect, the method comprising:
[0021] Obtain the target request to be processed through the service caller;
[0022] The asynchronous handle generated by the asynchronous I / O module based on the target request is obtained through the service call terminal;
[0023] The service caller initiates an asynchronous remote procedure call request to the service provider for the target request. The asynchronous remote procedure call request is used to instruct the service provider to process the target request.
[0024] The service caller tracks the processing result of the target request based on the asynchronous handle and determines the next step in the process.
[0025] In some embodiments, obtaining the asynchronous handle generated by the asynchronous I / O module based on the target request through the service caller includes:
[0026] The asynchronous I / O module parses the content of the target request and extracts the request parameters and context information.
[0027] The asynchronous I / O module creates a corresponding asynchronous handle based on the request parameters and the context information.
[0028] The asynchronous I / O module caches the target request and the asynchronous handle corresponding to the target request.
[0029] In some embodiments, the method further includes:
[0030] Through the Kubernetes platform, multiple requests obtained by the service caller are evenly distributed to multiple container instances deployed on the Kubernetes platform, wherein the number of worker threads of the multiple container instances is equal to the number of container CPU cores;
[0031] The asynchronous remote procedure call requests for the multiple requests are processed in parallel by the multiple container instances.
[0032] In some embodiments, initiating an asynchronous remote procedure call request to the service provider for the target request includes:
[0033] The asynchronous remote procedure call request for the target request is sent to the network through the container instance corresponding to the target request;
[0034] The network terminal receives the asynchronous remote procedure call request initiated by the service caller and forwards the asynchronous remote procedure call request to the service provider to request the service provider to process the target request.
[0035] In some embodiments, the step of creating a corresponding asynchronous handle by the asynchronous I / O module based on the request parameters and the context information includes:
[0036] The asynchronous I / O module ensures the uniqueness of the asynchronous handle by selecting one of the following methods: generating a random ID, using a timestamp, and other unique identification methods.
[0037] In some embodiments, after caching the target request and the asynchronous handle corresponding to the target request through the asynchronous I / O module, the method further includes:
[0038] Once the target request has been processed, the service caller clears the target request and the corresponding asynchronous handle from the cache.
[0039] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of a request processing method.
[0040] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a request processing method.
[0041] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the steps of a request processing method. Attached Figure Description
[0042] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 is a schematic diagram of the BIO network programming model;
[0044] Figure 2 is a schematic diagram of the NIO network programming model;
[0045] Figure 3 is a schematic diagram of the Netty network programming model;
[0046] Figure 4 is an architectural block diagram of a request processing system provided in an embodiment of this disclosure;
[0047] Figure 5 is a schematic diagram of the steps of a request processing method provided in an embodiment of this disclosure;
[0048] Figure 6 is a schematic diagram of the overall flow of a request processing method provided in an embodiment of this disclosure;
[0049] Figure 7 is a schematic diagram of concurrent time consumption verification in an embodiment of this disclosure;
[0050] Figure 8 is a schematic diagram of single-machine QPS verification in an embodiment of this disclosure;
[0051] Figure 9 is a schematic diagram of CPU performance verification in an embodiment of this disclosure;
[0052] Figure 10 is a schematic diagram of coding complexity verification in an embodiment of this disclosure. Detailed Implementation
[0053] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0054] Figure 1 illustrates the BIO (Blocking Input / Output) network programming model. As shown, in this model, the server starts a separate thread for each client connection request. When a client initiates a connection request, the server allocates a thread for that request. This thread is blocked until the request is processed or the I / O (Input / Output) operation is completed. Since each connection requires a thread, a large number of connection requests can consume significant thread resources, leading to increased thread overhead. A major drawback of the BIO model is that when the number of threads reaches the server's maximum limit, new connection requests will be rejected, causing service unavailability. Furthermore, if a thread is blocked while processing a request (e.g., waiting for an I / O operation), other requests cannot be processed in a timely manner, impacting the overall system's response speed and throughput. Therefore, the BIO model performs poorly in high-concurrency scenarios and is prone to causing performance bottlenecks.
[0055] Figure 2 illustrates the NIO network programming model. As shown, unlike BIO, NIO allows a single thread to handle multiple connection requests. Client connection requests are registered with a multiplexer (Selector), and the server checks the connection status through polling. In the NIO model, when an I / O request arrives, the Selector notifies the corresponding thread to handle it. This avoids requiring a separate thread for each connection, thus reducing thread overhead and the frequency of context switching. Although the NIO model improves the user experience and avoids blocking at the access layer, threads may still be blocked during I / O operations in actual processing. Therefore, while NIO outperforms BIO in performance, it may still face performance bottlenecks under high concurrency.
[0056] Figure 3 is a schematic diagram of the Netty network programming model. As shown in Figure 3, Netty is a high-performance network application framework based on NIO, which further encapsulates and optimizes NIO to simplify the complexity of network programming. Referring to Figure 3, the elliptical portion represents some common capabilities handled by the Netty framework or SDK, including accepting, reading, sending, decoding, and encoding functions. The Netty network programming model makes it easier for developers to build network applications. The cloud-shaped portion represents the core processing part of the service's business logic. Netty further optimizes resource utilization efficiency by distinguishing between receive and processing thread pools. The receive thread focuses on processing connection requests, while the processing thread focuses on executing business logic.
[0057] The analysis of BIO, NIO, and Netty network programming models above reveals the different challenges and limitations each model faces when handling connection requests and I / O operations. The BIO model, requiring an independent thread for each connection, leads to excessive resource consumption under high concurrency, easily causing performance bottlenecks. While NIO allows a single thread to handle multiple connection requests through multiplexing, reducing thread overhead, thread blocking can still occur during I / O operations, impacting overall system performance. With the increasing complexity of network applications and the rise in user concurrency, traditional synchronous blocking and non-blocking models can no longer meet the performance demands of modern high-concurrency scenarios. Therefore, Netty, as a high-performance framework based on NIO, improves system concurrency through modular design and thread pool optimization; however, it still has limitations, especially in handling I / O-intensive tasks, where thread blocking can still affect response speed. This disclosure proposes a fully asynchronous link mode, aiming to completely eliminate the limitations of thread blocking in traditional models. The fully asynchronous link mode ensures that threads are not blocked due to I / O operations or the execution of business logic during the entire request processing process by processing the server-side, business process, and client-side operations asynchronously.
[0058] Figure 4 is an architectural block diagram of a request processing system provided in an embodiment of this disclosure. As shown in Figure 4, the system includes a service calling end, an asynchronous I / O module, and a service providing end.
[0059] The service invoking end is used to initiate an asynchronous remote procedure call (RPC) request to the service provider for the target request to be processed. The asynchronous remote procedure call request is used to instruct the service provider to process the target request; the processing result of the target request is tracked according to the asynchronous handle, and the link of the next operation is determined.
[0060] The service caller is the client or system component that initiates the request, responsible for sending a request to the service provider to obtain the required service or data. The target request is the request currently being processed among multiple I / O requests received by the service caller. When the service caller needs to process the target request, it generates a corresponding asynchronous remote procedure call request, which contains the necessary request information for the target request, such as the type of the target request, parameters, and the address of the target service, so that the service provider can understand the target request.
[0061] An asynchronous remote procedure call (RPC) request is a non-blocking request. This means that after initiating an RPC request, the service caller does not need to wait for the service provider to complete its processing and can continue executing other tasks, avoiding resource waste and performance bottlenecks caused by waiting for a response. The RPC request contains a reference to an asynchronous handle, used to uniquely identify the target request and track its processing status and result in subsequent processing.
[0062] Upon receiving an asynchronous remote procedure call (RPC) request, the service provider immediately begins processing the target request. It's important to note that the target request represents the specific operation that the user or other services wish to perform. Upon receiving the target request, the service caller initiates an RPC request to the network, requesting the service provider to handle the target request. At this point, the target request and the RPC request are two distinct requests: the former represents the user's specific operational requirement, while the latter is a request from the service caller to the network, intended to forward the target request to the service provider for processing. Due to the fully asynchronous mode, the service provider can respond to other requests while processing the target request, thereby improving the system's concurrency capabilities. During processing, the service provider does not block any threads but converts I / O operations (such as database queries and external service calls) into asynchronous operations. When the service provider initiates an I / O request, it immediately returns control, allowing other requests to be processed.
[0063] When processing a request, the service provider associates the processing result with an asynchronous handle. Once the target request is processed, the service provider returns the result to the service caller via the asynchronous handle. The service caller can use the asynchronous handle to track the processing status of the request and perform subsequent operations when the result is available. This allows the service caller to obtain the processing result through queries or callbacks without blocking. Upon receiving the asynchronous handle, the service caller determines the next step in the operation chain. This next step includes, but is not limited to, continuing to process other requests, executing specific business logic based on the processing result, providing feedback to the user, or updating the interface.
[0064] The asynchronous I / O module is used to generate the asynchronous handle according to the target request and return the asynchronous handle to the service caller.
[0065] The asynchronous I / O module handles asynchronous remote procedure call requests from service callers and generates corresponding asynchronous handles. This ensures non-blocking I / O operations, improving system concurrency and response speed. When a service caller initiates a target request, the asynchronous I / O module receives the request and generates a unique asynchronous handle based on the request parameters and context information. An asynchronous handle is an identifier, an object, or a data structure containing request parameters such as the request ID, status, and timestamp, used to track the request's processing progress and result.
[0066] Once the asynchronous handle is generated, the asynchronous I / O module returns it to the service caller. The service caller can then use this handle to track and manage subsequent operations on the request. The returned asynchronous handle can contain methods or properties, allowing the service caller to query the request's status (such as "processing," "completed," "failed," etc.) or obtain the final processing result. In its implementation, the asynchronous I / O module can use event-driven programming, callback functions, or the Promise / Future pattern to ensure the asynchronous nature of I / O operations. The asynchronous I / O module also needs to interact with the underlying I / O system (such as databases, file systems, network connections, etc.) to ensure that I / O operations do not block the main thread.
[0067] The service provider is used to respond to the asynchronous remote procedure call request, process the target request, and return the processing result of the target request to the service caller through the asynchronous I / O module.
[0068] The service provider is a system component that responds to asynchronous remote procedure call requests initiated by the service caller. It is responsible for executing the specific business logic for the target request and returning the processing result of the target request to the service caller. When the service provider receives an asynchronous remote procedure call request, it parses the request content to understand the specific requirements of the target request, including the request type, parameters, address of the target service, and other information.
[0069] During the processing of asynchronous remote procedure call requests, the service provider executes corresponding business logic, including but not limited to the following: accessing the database to obtain or store data, calling other microservices or external APIs to obtain the required information, and performing computation or data processing tasks.
[0070] To achieve high concurrency and non-blocking processing, the service provider designs all potentially blocking operations (such as I / O operations) to be executed asynchronously. That is, when the service provider initiates an I / O request, it does not wait for the request to complete but immediately returns control, allowing other requests to proceed. Once the target request is processed, the service provider returns the processing result to the service caller via the asynchronous I / O module. Specifically, the service provider encapsulates the processing result (such as a success status, returned data, or error message) into a response object. This response object is associated with a previously generated asynchronous handle so that the service caller can recognize and process the result. The service provider then invokes the asynchronous I / O module to send the response object back to the service caller. At this point, the asynchronous I / O module is responsible for passing the result to the corresponding service caller.
[0071] Understandably, the asynchronous I / O module acts as a bridge in this process, ensuring that communication between the service provider and the service caller is non-blocking.
[0072] This disclosure introduces a fully asynchronous architecture, effectively solving the I / O bottleneck problem inherent in traditional synchronous blocking models. First, the service caller initiates an asynchronous remote procedure call request, enabling it to quickly send the target request to the service provider without blocking the current thread. This makes request initiation and processing independent, avoiding thread blocking caused by I / O operations and significantly improving system response speed and resource utilization. Second, by generating an asynchronous handle corresponding to the request and tracking the processing result of the target request, the service caller can continue processing other requests while waiting for the processing result, maximizing the CPU's concurrent processing capabilities. This disclosure, through its fully asynchronous link mode, significantly improves system throughput and response speed, exhibiting stronger stability and efficiency when facing high-concurrency requests, meeting the demands of modern internet applications for rapid response and high-concurrency processing.
[0073] In one alternative implementation, the service caller has multiple container instances deployed on a Kubernetes platform;
[0074] The Kubernetes platform evenly distributes the multiple requests obtained by the service caller to the multiple container instances, and the multiple container instances are used to process the asynchronous remote procedure call requests for the multiple requests in parallel.
[0075] It's important to note that Kubernetes is an open-source container orchestration platform used for automating the deployment, scaling, and management of containerized applications. To improve system availability and concurrency, multiple container instances are deployed on the Kubernetes platform for the service caller. Each container instance runs the same service caller application and can independently handle requests. This approach distributes the load across multiple containers, ensuring good system performance even under high concurrency. Kubernetes provides powerful scheduling and management capabilities, enabling multiple container instances to collaborate efficiently. In this embodiment, the Kubernetes platform receives requests from the service caller and distributes these requests evenly across multiple container instances. When the service caller receives multiple asynchronous remote procedure call requests, the Kubernetes platform distributes these requests to different container instances based on a load balancing strategy. This distribution can be based on strategies such as round-robin, random, or least connections to ensure a relatively balanced load on each container instance. This even distribution strategy effectively avoids performance bottlenecks caused by excessive requests to a single container instance, thereby improving the overall system's response speed and processing capacity.
[0076] When a request is assigned to a specific container instance, the container instance processes these asynchronous remote procedure call requests in parallel. Because each container instance is independent, they can handle multiple requests simultaneously without interfering with each other. During processing, the container instance utilizes an asynchronous I / O module to perform I / O operations, ensuring that waiting for I / O completion does not block the processing of other requests, enabling the system to maintain high throughput under high concurrency.
[0077] The Kubernetes platform supports dynamically scaling up and down the number of container instances. When request volume increases, more container instances can be automatically added based on load to handle high concurrency demands; when request volume decreases, the number of container instances can be reduced, thereby saving resources. This allows service clients to flexibly respond to different traffic conditions, ensuring the system always runs efficiently. Kubernetes also provides monitoring and management capabilities, enabling real-time monitoring of container instance performance, health status, and resource usage. Through monitoring data, operations personnel can promptly identify and resolve potential problems, ensuring system stability.
[0078] In this embodiment of the disclosure, the deployment and management of the service caller on the Kubernetes platform achieves efficient request processing capabilities, and can maintain good performance and resource utilization efficiency in high-concurrency scenarios.
[0079] In one alternative implementation, the number of worker threads of the plurality of container instances is equal to the number of container CPU cores.
[0080] The number of worker threads refers to the number of threads that can execute concurrently in each container instance. A thread is the basic unit for processing requests; multiple threads can execute tasks in parallel, thereby improving system throughput.
[0081] The number of CPU cores refers to the number of CPU cores allocated to a container instance. Each CPU core can execute one thread simultaneously; therefore, the number of CPU cores directly affects the parallel processing capability of a container instance.
[0082] In the fully asynchronous cascading mode disclosed herein, the number of worker threads across multiple container instances equals the number of CPU cores in the container, avoiding thread context switching. If the number of worker threads exceeds the number of CPU cores, the system will frequently perform thread context switching, leading to performance degradation. Each switch requires saving and restoring the thread's state, which consumes CPU resources. By setting the number of worker threads to the number of CPU cores, it ensures that each CPU core is fully utilized, preventing resource waste due to excessive threads.
[0083] Assume a Kubernetes cluster contains a container instance for a service call, with each container instance allocated 4 CPU cores (i.e., 4 CPU cores). Following the design principles of this disclosure, the number of worker threads is set to 4, meaning each container instance will create 4 worker threads. Assume 10 asynchronous remote procedure call requests arrive at the container instance simultaneously. The container instance's 4 worker threads will be activated and begin processing the requests. Since the number of worker threads equals the number of CPU cores, the 4 threads can process 4 requests in parallel. The first request is processed by thread 1, the second by thread 2, the third by thread 3, and the fourth by thread 4. While processing the remaining requests, once the first request is completed, thread 1 will immediately begin processing the fifth request. Threads 2, 3, and 4 will also process subsequent requests sequentially until all requests are processed.
[0084] During processing, if a request involves I / O operations (such as database queries), the thread will initiate an asynchronous I / O request and immediately return control, allowing other threads to continue processing the remaining requests without waiting for the I / O to complete.
[0085] In one alternative implementation, the system further includes a middleware module and a database;
[0086] The service provider is used to asynchronously call the middleware module and distribute multiple requests from the service caller to multiple business processing instances for parallel processing through the middleware module.
[0087] The business processing instance is used to asynchronously call the database to perform data access operations and return the processing results to the service calling end through the asynchronous IO module.
[0088] As mentioned earlier, the service provider responds to asynchronous remote procedure call requests initiated by the service caller and distributes these requests through an asynchronous call middleware module. This middleware module acts as the request distribution center, responsible for allocating multiple requests from the service provider to multiple business processing instances to achieve parallel processing. Each business processing instance is responsible for specific business logic processing and asynchronously calls the database for data storage and retrieval operations. The processing results are returned to the service provider. The database stores business data and supports data reading and writing operations by the business processing instances.
[0089] When a service caller initiates multiple asynchronous requests, the service provider receives these requests and asynchronously invokes the middleware module, passing the multiple requests to the middleware module. Because it is an asynchronous call, the service provider does not block while waiting for the middleware module's response, but can continue to process other tasks.
[0090] After receiving a request from the service provider, the middleware module distributes these requests to multiple business processing instances according to a load balancing strategy (such as round-robin, random, or least connections). This distribution can be dynamic, allowing the middleware to intelligently schedule requests based on the current load of the business processing instances. Each business processing instance, upon receiving its assigned request, begins executing the corresponding business logic. During processing, if database access (such as reading or writing data) is required, the business processing instance initiates an asynchronous call to the database via the asynchronous I / O module. Similarly, after initiating a database request, the business processing instance does not need to wait for a database response but can continue processing other tasks or requests.
[0091] After receiving a request, the database performs the corresponding data access operations (such as querying, inserting, and updating). Once the database operation is complete, it returns the results to the corresponding business processing instance. Upon receiving the database response, the business processing instance completes the corresponding business logic processing and returns the processing results to the service provider via the asynchronous I / O module.
[0092] In one alternative implementation, the system further includes a network terminal;
[0093] The network terminal is configured to receive the asynchronous remote procedure call request initiated by the service call terminal, and forward the asynchronous remote procedure call request to the service provider terminal to request the service provider terminal to process the target request.
[0094] The network endpoint is responsible for listening to and receiving asynchronous remote procedure call requests from the service caller. The network endpoint forwards the received asynchronous remote procedure call requests to the service provider, ensuring that the requests are processed correctly. In this embodiment, the network endpoint can also process responses from the service provider and return them to the service caller.
[0095] After the service caller initiates an asynchronous remote procedure call request, the network end listens on a specific port using network protocols such as HTTP (Hypertext Transfer Protocol) or TCP (Transmission Control Protocol), waiting to receive requests from the service caller. When the request arrives, the network end immediately receives the asynchronous remote procedure call request and forwards it to the service provider.
[0096] When forwarding asynchronous remote procedure call (RPC) requests, the network end can use load balancing strategies (such as round-robin or random) to distribute the requests to multiple business processing instances on the service provider side, thereby improving system processing capacity. The network end will also receive processing results from the service provider side, especially when it needs to return results to the service caller. If the network end receives a response from the service provider side, it will return the processing result to the service caller that initially initiated the request, completing the entire request processing flow.
[0097] Figure 5 is a schematic diagram of the steps of a request processing method according to an embodiment of this disclosure. As shown in Figure 5, the method is applied to the request processing system described above, and the method includes:
[0098] Step S101: Obtain the target request to be processed through the service caller.
[0099] The service caller, acting as the request initiator, is responsible for receiving and processing requests from users or other system components. Specifically, the service caller receives multiple I / O requests, including the target request currently pending processing. These multiple I / O requests contain specific operations or data requests that the user wishes to perform. Based on business logic and the current processing status, the service caller selects the target request from the received requests for processing.
[0100] Step S102: Obtain the asynchronous handle generated by the asynchronous IO module based on the target request through the service calling terminal.
[0101] The service caller invokes the asynchronous I / O module, passing the content of the target request. The asynchronous I / O module generates a unique asynchronous handle based on the target request content. Once generated, the asynchronous I / O module returns the handle to the service caller. The service caller saves this handle for later use. The service caller uses the asynchronous handle to track the processing progress and result of the target request, ensuring timely response to subsequent operation needs.
[0102] Step S103: The service calling end initiates an asynchronous remote procedure call request for the target request to the service provider. The asynchronous remote procedure call request is used to instruct the service provider to process the target request.
[0103] The service caller constructs an asynchronous remote procedure call (RPC) request based on the content of the target request and the asynchronous handle. The RPC request includes information such as the type of the target request, parameters, and the address of the target service, ensuring that the service provider can understand and process the request. The service caller sends the constructed RPC request to the service provider via the network. Upon receiving the request, the network immediately forwards it to the service provider for processing, ensuring that the request is handled promptly.
[0104] Step S104: The service caller tracks the processing result of the target request based on the asynchronous handle and determines the link for the next operation.
[0105] The service caller uses asynchronous handles to track the processing results of target requests and determines subsequent operations based on those results. Specifically, the service caller uses asynchronous handles to query the processing status of target requests. The request status, such as "processing," "completed," or "failed," can be checked through methods or properties provided by the asynchronous I / O module. When the target request is completed, the service provider associates the processing result with the asynchronous handle and returns it to the service caller through the asynchronous I / O module. The service caller can then use the asynchronous handle to obtain the final processing result. Based on the result, the service caller determines the next step in the process, which may include continuing to process other requests, executing specific business logic based on the result, providing feedback to the user, or updating the user interface. The service caller flexibly adjusts subsequent operations according to business needs to ensure efficient system operation.
[0106] In one optional implementation, obtaining the asynchronous handle generated by the asynchronous I / O module based on the target request through the service caller includes: parsing the content of the target request through the asynchronous I / O module to extract request parameters and context information; creating a corresponding asynchronous handle through the asynchronous I / O module based on the request parameters and the context information; and caching the target request and the asynchronous handle corresponding to the target request through the asynchronous I / O module.
[0107] After receiving the target request, the service caller parses it to extract necessary information, ensuring that the subsequent asynchronous handle accurately reflects the request's requirements. Specifically, the service caller parses the content of the target request, including decoding its format (e.g., JSON, JavaScript Object Notation, XML, Extensible Markup Language) and reading its various fields. From the parsed request content, the service caller extracts the request parameters, including the request type, input data, and target service address. In addition to the request parameters, the service caller also extracts context information to better understand the request's background and execution environment. Context information includes user information, session information, and timestamps. The service caller creates an asynchronous handle based on the extracted request parameters and context information. The service caller then invokes the asynchronous I / O module, passing the extracted request parameters and context information. The asynchronous I / O module generates a unique asynchronous handle based on this information. When creating asynchronous handles, it's crucial to ensure the uniqueness of each handle to avoid confusion between different requests. This can be achieved by generating random IDs, using timestamps, or other unique identification methods.
[0108] The service caller caches the target request and its corresponding asynchronous handle, which can be stored in memory or a database for subsequent processing and state tracking. Storing it in memory or a database allows for quick access when needed. During caching, the generated asynchronous handle needs to be associated with the target request and stored in a data structure (such as a mapping table or dictionary). When caching asynchronous handles, their lifecycle management must also be considered. For example, the cache should be cleared promptly after the request is processed to avoid memory leaks or cache expiration issues.
[0109] Through the above steps, the service caller can effectively generate and manage asynchronous handles, thereby achieving non-blocking processing and state tracking of target requests, ensuring efficient response to user requests, and improving overall concurrency processing capabilities.
[0110] In an optional implementation, the method further includes: evenly distributing multiple requests obtained by the service caller to multiple container instances deployed on the Kubernetes platform, wherein the number of worker threads of the multiple container instances is equal to the number of container CPU cores; and processing asynchronous remote procedure call requests for the multiple requests in parallel through the multiple container instances.
[0111] The Kubernetes platform evenly distributes the multiple requests obtained by the service caller to the multiple container instances, and the multiple container instances are used to process the asynchronous remote procedure call requests for the multiple requests in parallel.
[0112] To improve system availability and concurrency capabilities, the service caller deploys multiple container instances on the Kubernetes platform. Each container instance runs the same service caller application and can independently handle requests. This distributes the load across multiple containers, ensuring good system performance even under high concurrency. Kubernetes is an open-source container orchestration platform responsible for the automated deployment, scaling, and management of containerized applications. Kubernetes provides powerful scheduling and management capabilities, enabling multiple container instances to collaborate efficiently. In this embodiment, the Kubernetes platform receives requests from the service caller and distributes these requests evenly across multiple container instances. When the service caller receives multiple asynchronous remote procedure call requests, the Kubernetes platform distributes these requests to different container instances according to a load balancing strategy. This distribution can be based on strategies such as round-robin, random, or least connections to ensure a relatively balanced load on each container instance. This even distribution strategy effectively avoids performance bottlenecks caused by excessive requests to a single container instance, thereby improving the overall system's response speed and processing capacity.
[0113] When a request is assigned to a specific container instance, the container instance processes these asynchronous remote procedure call requests in parallel. Because each container instance is independent, they can handle multiple requests simultaneously without interfering with each other. During processing, the container instance utilizes an asynchronous I / O module to perform I / O operations, ensuring that waiting for I / O completion does not block the processing of other requests, enabling the system to maintain high throughput under high concurrency.
[0114] The Kubernetes platform supports dynamically scaling up and down the number of container instances. When request volume increases, more container instances can be automatically added based on load to handle high concurrency demands; when request volume decreases, the number of container instances can be reduced, thereby saving resources. This allows service clients to flexibly respond to different traffic conditions, ensuring the system always runs efficiently. Kubernetes also provides monitoring and management capabilities, enabling real-time monitoring of container instance performance, health status, and resource usage. Through monitoring data, operations personnel can promptly identify and resolve potential problems, ensuring system stability.
[0115] In this embodiment of the disclosure, the deployment and management of the service caller on the Kubernetes platform achieves efficient request processing capabilities, and can maintain good performance and resource utilization efficiency in high-concurrency scenarios.
[0116] In the fully asynchronous cascading mode disclosed herein, the number of worker threads across multiple container instances equals the number of CPU cores in the container, avoiding thread context switching. If the number of worker threads exceeds the number of CPU cores, the system will frequently perform thread context switching, leading to performance degradation. Each switch requires saving and restoring the thread's state, which consumes CPU resources. By setting the number of worker threads to the number of CPU cores, it ensures that each CPU core is fully utilized, preventing resource waste due to excessive threads.
[0117] During processing, if a request involves I / O operations (such as database queries), the thread will initiate an asynchronous I / O request and immediately return control, allowing other threads to continue processing the remaining requests without waiting for the I / O to complete.
[0118] In one optional implementation, the step of initiating an asynchronous remote procedure call request for the target request to the service provider through the service caller includes: sending the asynchronous remote procedure call request for the target request to the network end through the container instance corresponding to the target request; receiving the asynchronous remote procedure call request initiated by the service caller through the network end, and forwarding the asynchronous remote procedure call request to the service provider to request the service provider to process the target request.
[0119] The service caller sends the prepared asynchronous remote procedure call request to the network through its corresponding container instance.
[0120] The network layer, acting as an intermediary, receives asynchronous remote procedure call (RPC) requests from service callers and forwards them to service providers. The network layer has a listening mechanism capable of processing and receiving requests from different service callers. Upon receiving an RPC request, the network layer parses it to extract the request type, request parameters, target service address, and reference to the asynchronous handle. The network layer then forwards the parsed RPC request to the appropriate service provider.
[0121] After receiving the forwarded asynchronous remote procedure call request, the service provider begins processing the target request. The service provider executes corresponding business logic based on the request content, including accessing the database and calling other services.
[0122] Through the above steps, the service caller can effectively initiate asynchronous remote procedure call requests to the service provider. The entire process achieves non-blocking request behavior, allowing the service caller to continue executing other tasks after initiating the request without waiting for a response from the service provider. This improves concurrency handling capabilities and response speed, enabling more efficient processing of a large number of requests in high-concurrency scenarios.
[0123] Figure 6 is a schematic diagram of the overall flow of a request processing method according to an embodiment of this disclosure. As shown in Figure 6, in this process, firstly, the service caller initiates an asynchronous remote procedure call request for the target request and sends it to the network end. Next, the asynchronous I / O module returns an asynchronous handle to the service caller for subsequent request tracing. Then, the service caller caches the target request and its response context for later access. Next, the network end forwards the received asynchronous remote procedure call request to the service provider, requesting it to process the target request. Upon receiving the request, the service provider executes the corresponding business logic and asynchronously calls the middleware module to complete the processing. After processing, the service provider sends a response to the asynchronous I / O module. Finally, the asynchronous I / O module returns the processing result of the target request to the service caller, completing the entire asynchronous processing flow.
[0124] The verification embodiments of this disclosure use a Linux operating system and an 8-core CPU to start a single instance for continuous stress testing to evaluate the performance of the fully asynchronous high-concurrency service solution.
[0125] Figure 7 is a schematic diagram illustrating the verification results of a request processing system provided in one embodiment of this disclosure. Referring to Figure 7, the upper line represents the performance of the synchronous model, and the lower line represents the performance of the asynchronous model. In the traditional synchronous programming model before optimization, the system response time increases significantly with the increase of concurrent requests. This is because the traditional model generates thread blocking when processing requests, leading to wasted system resources and prolonged response time. However, after implementing the fully asynchronous scheme of this disclosure, the system can effectively avoid thread blocking, significantly reduce response time, and ensure that it maintains low latency even under high concurrency.
[0126] Figure 8 is a schematic diagram illustrating the single-machine QPS (Queries Per Second) verification in an embodiment of this disclosure. Referring to Figure 8, the left bar represents the image processing interface, and the right bar represents ID Mapping. In the two bar pairs, the left bar represents the performance of the synchronous model, and the right bar represents the performance of the asynchronous model. It can be seen that by implementing the solution of this disclosure, the single-machine performance is improved by nearly 6 times in IO-intensive capabilities, reaching 10,000 QPS. This result is far superior to the performance of the traditional synchronous programming model. The QPS calculation formula for the traditional model is:
[0127] In the implementation data, assuming an average interface latency of 50ms and 100 threads, the calculated QPS is 2000. This demonstrates that adopting the fully asynchronous solution disclosed herein significantly improves the system's throughput when handling I / O-intensive requests.
[0128] Figure 9 is a schematic diagram illustrating CPU performance verification in an embodiment of this disclosure. Referring to Figure 9, the lower line represents the asynchronous model, and the middle line represents the synchronous model. The fully asynchronous scheme of this disclosure significantly reduces CPU utilization by eliminating thread switching and waiting overhead. In the traditional synchronous model, CPU resources are heavily consumed due to frequent thread creation and destruction, as well as the time spent waiting for I / O operations to complete. However, by adopting the fully asynchronous scheme, the CPU load is effectively reduced, system resources are utilized more efficiently, and overall performance is improved.
[0129] Figure 10 is a schematic diagram of the coding complexity verification in an embodiment of this disclosure. Referring to Figure 10, although JDK 19 (Java Development Kit version 19) will soon support coroutines and may bring some performance improvements, the fully asynchronous solution of this disclosure still outperforms the coroutine implementation in terms of performance. This is because the implementation of coroutines requires recording IO interrupt points in the heap, increasing coding complexity and runtime overhead. In contrast, this disclosure, through a fully asynchronous link mode, can manage IO operations more efficiently, reducing unnecessary overhead and thus offering a performance advantage.
[0130] In summary, this disclosure significantly reduces request processing time, especially maintaining stable response times under high concurrency. It also increases the QPS of a single instance to 10,000 / s, far exceeding the performance of traditional synchronous models, while reducing CPU utilization and optimizing resource efficiency. Furthermore, it demonstrates superior performance compared to the upcoming JDK 19 coroutines.
[0131] This disclosure also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various processes described above in an embodiment of a request processing method and achieves the same technical effects. To avoid repetition, it will not be described again here.
[0132] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, electronic devices, and storage media. Therefore, embodiments of this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this disclosure can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0133] This disclosure describes embodiments of methods and apparatus according to embodiments of this disclosure with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams. These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowchart illustrations and / or one or more block diagrams. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.
[0134] While preferred embodiments of the present disclosure have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the present disclosure.
[0135] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the term "comprising" or any other variations thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element. The above provides a detailed description of a request processing system, method, and electronic device provided by this disclosure. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure; at the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.
[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0137] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
A request processing system, the system comprising a service caller, an asynchronous I / O module, and a service provider; The service invocation terminal is used to initiate an asynchronous remote procedure call request to the service provider for the target request to be processed. The asynchronous remote procedure call request is used to instruct the service provider to process the target request. The processing result of the target request is traced based on the asynchronous handle, and the chain of the next operation is determined; The asynchronous I / O module is used to generate the asynchronous handle according to the target request and return the asynchronous handle to the service caller. The service provider is used to respond to the asynchronous remote procedure call request, process the target request, and return the processing result of the target request to the service caller through the asynchronous I / O module. The system according to claim 1, wherein, The service call client is deployed with multiple container instances via the Kubernetes platform; The Kubernetes platform evenly distributes the multiple requests obtained by the service caller to the multiple container instances, and the multiple container instances are used to process the asynchronous remote procedure call requests for the multiple requests in parallel. The system according to claim 2, wherein, The number of worker threads for the multiple container instances is equal to the number of CPU cores in the container. The system according to claim 1, wherein, The system also includes a middleware module and a database; The service provider is used to asynchronously call the middleware module and distribute multiple requests from the service caller to multiple business processing instances for parallel processing through the middleware module. The business processing instance is used to asynchronously call the database to perform data access operations and return the processing results to the service caller through the asynchronous IO module. The system according to claim 1, wherein, The system also includes a network terminal; The network terminal is configured to receive the asynchronous remote procedure call request initiated by the service call terminal, and forward the asynchronous remote procedure call request to the service provider terminal to request the service provider terminal to process the target request. The system according to claim 4, wherein, The middleware module distributes the multiple requests to the multiple business processing instances using a load balancing strategy of round-robin, random, or least connections. A request processing method, applied to the system as described in any one of claims 1 to 6, the method comprising: Obtain the target request to be processed through the service caller; The asynchronous handle generated by the asynchronous I / O module based on the target request is obtained through the service call terminal; The service caller initiates an asynchronous remote procedure call request to the service provider for the target request. The asynchronous remote procedure call request is used to instruct the service provider to process the target request. The service caller tracks the processing result of the target request based on the asynchronous handle and determines the next step in the process. The method according to claim 7, wherein, The step of obtaining the asynchronous I / O module generated by the asynchronous I / O module based on the target request through the service caller includes: The asynchronous I / O module parses the content of the target request and extracts the request parameters and context information. The asynchronous I / O module creates a corresponding asynchronous handle based on the request parameters and the context information. The asynchronous I / O module caches the target request and the asynchronous handle corresponding to the target request. The method according to claim 7, wherein, The method further includes: Through the Kubernetes platform, multiple requests obtained by the service caller are evenly distributed to multiple container instances deployed on the Kubernetes platform, wherein the number of worker threads of the multiple container instances is equal to the number of container CPU cores; The asynchronous remote procedure call requests for the multiple requests are processed in parallel by the multiple container instances. The method according to claim 9, wherein, The service caller initiates an asynchronous remote procedure call request for the target request to the service provider, including: The asynchronous remote procedure call request for the target request is sent to the network through the container instance corresponding to the target request; The network terminal receives the asynchronous remote procedure call request initiated by the service caller and forwards the asynchronous remote procedure call request to the service provider to request the service provider to process the target request. The method according to claim 8, wherein, The step of creating a corresponding asynchronous handle through the asynchronous I / O module based on the request parameters and the context information includes: The asynchronous I / O module ensures the uniqueness of the asynchronous handle by selecting one of the following methods: generating a random ID, using a timestamp, and other unique identification methods. The method according to claim 8, wherein, After caching the target request and the corresponding asynchronous handle through the asynchronous I / O module, the method further includes: Once the target request has been processed, the service caller clears the target request and the corresponding asynchronous handle from the cache. An electronic device, comprising: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as described in any one of claims 7 to 12. A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the method as claimed in any one of claims 7 to 12. A computer program product includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 7 to 12.