Non-blocking API Thread Management for Downstream Service Calls
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer systems face inefficiencies and high resource usage due to the instantiation of thousands of threads waiting for responses from downstream services, leading to increased latency and memory footprint issues when processing high volumes of requests.
Innovation Solution
Implementing a non-blocking application programming interface (API) that allows threads to perform calls to downstream services without waiting for responses, enabling them to process other tasks concurrently and reducing the need for idle threads, with a system architecture that includes task queues and thread pools to manage and execute these non-blocking calls.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional blocking calls are used to wait for downstream service responses, then request processing can be completed accurately, but the number of instantiated threads increases and system resources are consumed
Solution Approach 1:
The patent segments the thread lifecycle into active processing phases and idle waiting phases. By using non-blocking calls, threads are separated from the waiting state, allowing the system to instantiate fewer threads while maintaining the ability to process multiple requests. The segmentation also applies to the call mechanism itself, separating the call initiation from the response reception.
Solution Approach 2:
The patent implements preliminary action by establishing a non-blocking call mechanism before the actual waiting period. Threads perform the call action and immediately transition to processing other tasks without entering a blocked state. This preliminary setup of non-blocking communication protocols allows the system to maintain thread availability while processing downstream service requests.
2Productivity
If more threads are instantiated to handle high request volumes, then request processing capacity increases, but latency increases due to threads waiting for responses
Solution Approach 1:
The patent ensures continuity of useful action by preventing threads from entering idle waiting states. Through non-blocking calls, threads continuously perform productive work - either processing requests or handling other tasks - rather than remaining blocked awaiting downstream responses. This eliminates wasted time in the thread lifecycle and maintains constant productive output.
Solution Approach 2:
The patent introduces dynamics into the thread management system by allowing threads to dynamically switch between processing different tasks based on availability. Instead of static thread assignment where threads are bound to specific waiting calls, threads can dynamically pick up new tasks while previous calls complete asynchronously, optimizing the utilization of processing capacity over time.
3Measurement precision
If blocking calls are used to ensure proper response handling, then task completion can be tracked accurately, but memory footprint increases due to idle threads
Solution Approach 1:
The patent introduces an intermediary mechanism - the non-blocking call interface - that decouples the thread from the response waiting process. This intermediary layer handles the communication with downstream services while threads remain free to perform other work. The intermediary maintains task completion tracking through alternative means such as callbacks or event-driven notifications, preserving measurement precision without requiring dedicated waiting threads.
Data Source
AI summary
Techniques are disclosed relating to task execution with non-blocking calls. A computer system may receive a request to perform an operation comprising a plurality of tasks, each of which corresponds to a node in a graph. A particular one of the plurality of tasks specifies a call to a downstream service. The computer system may maintain a plurality of task queues, each of which is associated with a thread pool. The computer system may enqueue, in an order specified by the graph, the plurality of tasks in one or more of the plurality of task queues. The computer system may process the plurality of tasks. Such processing may include a thread of a particular queue in which the particular task is enqueued performing a non-blocking call to the downstream service. After processing the plurality of tasks, the computer system may return a result of performing the operation.