Distributed machine learning inference system in private network environment and operation method thereof

Abstract

Disclosed is a distributed machine learning inference system in a private network environment. The system according to an embodiment may comprise: a user equipment; a serve queue configured to receive an inference request from the user equipment, store identification information of the inference request in a message queue, schedule the inference request received from the message queue by using autoscaling, on the basis of stack information of the message queue and activation information of a plurality of servers, and perform scale-to-zero on a first server from among the plurality of servers if a first server from among the plurality of servers satisfies a predetermined criterion; and the plurality of servers configured to generate an inference result in response to an inference request allocated from the serve queue.

Description

Distributed machine learning inference system in a private network environment and method of operation thereof

[0001] The following embodiments relate to a distributed machine learning inference system in a private network environment and a method of operation thereof. More specifically, the following embodiments relate to a distributed machine learning inference system and a method of operation thereof that enables efficient resource utilization and asynchronous request processing through a serverless architecture in a private network environment.

[0002] In the modern information technology environment, Artificial Intelligence (AI) and Machine Learning (ML) play important roles in various fields. In these technologies, models are trained to perform specific tasks based on large amounts of data, and these trained models produce results such as predictions or classifications of input data to provide users with customized services or advanced analytical results. Machine learning models are utilized in diverse application areas, including image recognition, natural language processing, and speech recognition. In particular, for applications requiring real-time inference (e.g., autonomous driving, security surveillance, industrial automation, etc.), fast and efficient machine learning inference systems are essential.

[0003] Currently, most machine learning inference is performed in cloud environments. While the cloud offers high processing power and scalability, it has limitations such as data transmission latency, security issues, privacy concerns, and dependency on internet connectivity. In particular, existing cloud-centric approaches may not be suitable for private network environments handling sensitive data or those with limited network connectivity.

[0004] To address this, distributed machine learning inference systems are gaining attention. By distributing tasks across multiple devices or servers, these systems can reduce latency, minimize network traffic, and enhance security by keeping data local.

[0005] The problem that the present disclosure aims to solve is to provide a distributed machine learning inference system in a private network environment.

[0006] To utilize pre-trained machine learning models in real-world business environments, a system capable of performing inference is required. This system receives input data, derives results through the model, and provides those results to the user. To efficiently handle a large volume of inference requests, such a system operates in a distributed environment and performs parallel processing by utilizing multiple server computers.

[0007] However, conventional distributed machine learning inference systems do not fully support 'scale-to-zero'. In particular, conventional distributed machine learning inference systems lack asynchronous request management capabilities that temporarily suspend inference requests and process them after the server starts.

[0008] Due to the aforementioned problems, it was difficult to fully implement a serverless architecture and efficiently manage resources.

[0009] Accordingly, the inventor(s) of the present disclosure have resolved the aforementioned limitations or difficulties and developed a new distributed machine learning inference system that operates in a private network environment and features autoscaling capabilities.

[0010] The problems that the present disclosure aims to solve are not limited to those described above, and problems not mentioned will be clearly understood by those skilled in the art from the present specification and the accompanying drawings.

[0011] A distributed machine learning inference system in a private network environment according to one embodiment may include: a user device; a sub-queue configured to receive an inference request from the user device, store identification information of the inference request in a message queue, schedule the inference request received from the message queue using auto-scaling based on stack information of the message queue and activation information of a plurality of servers, and perform scale-to-zero on the first server among the plurality of servers when the first server among the plurality of servers satisfies a predetermined criterion; and the plurality of servers configured to generate an inference result in response to an inference request allocated from the sub-queue.

[0012] According to a feature of the present disclosure, the predetermined criterion may be that at least one of the plurality of servers is in an inactive state during a predetermined first time and has not received an inference request from the user device during a predetermined second time.

[0013] According to another feature of the present disclosure, the sub-queue may be further configured to transmit a termination signal to the first server among the plurality of servers when the predetermined criteria are satisfied.

[0014] According to another feature of the present disclosure, the inference request includes a first inference request, and the sub-queue may be further configured to detect the first inference request that failed to generate an inference result, control the message queue to increase the priority of the first inference request, and resend the first inference request to a second server among the plurality of servers.

[0015] According to another feature of the present disclosure, the inference request includes a second inference request, and the sub-queue may be further configured to transmit a start signal to a third server among the plurality of servers when the second inference request is received and the plurality of servers are in a zero-to-one situation, store the second inference request in the message queue to hold, and transmit the second inference request to the third server among the plurality of servers after a predetermined third time.

[0016] According to another feature of the present disclosure, storage may further include an input storage and an output storage, wherein the input storage is configured to store data for the inference request and the output storage is configured to store data for the inference result.

[0017] According to another feature of the present disclosure, the autoscaling may dynamically control the activation status of each of the plurality of servers using the Horizontal Pod Autoscaler (HPA) function of Kubernetes.

[0018] A method of operation of a computing device for distributed machine learning inference in a private network environment according to one embodiment may include the step of receiving an inference request from a user device, wherein the inference request includes a first inference request and a second inference request; the step of storing identification information of the inference request in a message queue; the step of scheduling the inference request received from the message queue using autoscaling based on stack information of the message queue and activation information of a plurality of servers; and the step of transmitting a termination signal to the first server among the plurality of servers when the first server among the plurality of servers satisfies a predetermined criterion.

[0019] A computer program according to one embodiment may be a computer program stored on a computer-readable recording medium to perform a method of operation of a computing device for distributed machine learning inference in a private network environment described above, combined with a computer which is hardware.

[0020] The means for solving the problem of the present disclosure are not limited to the means for solving the problem described above, and means for solving the problem not mentioned will be clearly understood by those skilled in the art to which the present disclosure belongs from the present specification and the attached drawings.

[0021] According to the embodiments, by dynamically adjusting the number of server resources according to the load of the distributed machine learning inference system through auto-scaling and scale-to-zero functions, unnecessary resource waste can be prevented and costs can be reduced.

[0022] According to the embodiments, through an asynchronous request processing method utilizing a message queue, inference requests can be received even when server resources are unavailable and processed after the server starts, thereby enabling the provision of stable services to the user.

[0023] According to the embodiments, requests can be safely reprocessed even when errors occur in the processor or server through a monitoring and error handling mechanism, and the reliability and stability of the distributed machine learning inference system can be enhanced.

[0024] According to the embodiments, various inference services can be operated simultaneously through an integrated architecture linking Kubernetes and a message queue system, and the scalability and flexibility of the system can be improved.

[0025] According to the embodiments, it is designed to operate in a private network environment without relying on cloud services, making it usable even in environments where data security and regulatory compliance are critical.

[0026] This enables companies or institutions operating machine learning inference services to provide stable services while efficiently managing computing resources, and allows for cost reduction and improved service quality.

[0027] The effects of the present disclosure are not limited to the effects described above, and unmentioned effects will be clearly understood by those skilled in the art from the present specification and the accompanying drawings.

[0028] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present specification, provide embodiments of the present specification and explain the technical features of the present specification together with the detailed description.

[0029] FIG. 1 is a conceptual diagram illustrating a distributed machine learning inference system according to one comparative example of the present disclosure.

[0030] FIG. 2 is a block diagram showing a distributed machine learning inference system according to one comparative example of the present disclosure.

[0031] FIG. 3 is a block diagram showing the configuration of a distributed machine learning inference system according to one embodiment of the present disclosure.

[0032] FIG. 4 is a block diagram showing the configuration of a computing device for distributed machine learning inference according to one embodiment of the present disclosure.

[0033] FIG. 5 is a block diagram illustrating an example of the operation of a distributed machine learning inference system according to one embodiment of the present disclosure.

[0034] FIG. 6 is a flowchart illustrating a method of operation of a computing device for distributed machine learning inference according to one embodiment of the present disclosure.

[0035] Specific structural or functional descriptions of embodiments according to the concept of the present disclosure disclosed herein are provided merely for the purpose of explaining embodiments according to the concept of the present disclosure, and embodiments according to the concept of the present disclosure may be implemented in various forms and are not limited to the embodiments described herein.

[0036] Embodiments according to the concept of the present disclosure may be subject to various modifications and may take various forms; therefore, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present disclosure to specific disclosed forms, and includes modifications, equivalents, or substitutions that fall within the spirit and scope of the present disclosure.

[0037] Terms such as "first" or "second" may be used to describe various components, but said components shall not be limited by said terms. For the sole purpose of distinguishing one component from another, for example, without departing from the scope of rights according to the concept of the present disclosure, the first component may be named the second component, and similarly, the second component may be named the first component.

[0038] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. Conversely, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Expressions describing the relationships between components, such as "between," "exactly between," or "directly adjacent to," should be interpreted in the same way.

[0039] The terms used herein are used merely to describe specific embodiments and are not intended to limit the disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0040] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this disclosure pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0041] In this specification, the term "processor" may refer to hardware capable of performing functions and operations according to each name described in this specification, computer program code capable of performing specific functions and operations, or an electronic recording medium loaded with computer program code capable of performing specific functions and operations.

[0042] In other words, the term "processor" may mean a functional and / or structural combination of hardware for carrying out the technical concept of the present disclosure and / or software for driving said hardware.

[0043] Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or restricted by these embodiments. Identical reference numerals in each drawing indicate identical components.

[0044] FIG. 1 is a conceptual diagram illustrating a distributed machine learning inference system according to one comparative example of the present disclosure.

[0045] Referring to FIG. 1, a distributed machine learning inference system (1000) according to one comparative example of the present disclosure may include a user device (100) and / or a plurality of servers (200). In this specification, 'server' may also be referred to as a computing resource, a computational resource, a server resource, or a server computer.

[0046] In order for a user to utilize a pre-trained machine learning model in a work environment, a system that performs an inference process (e.g., a distributed machine learning inference system (1000)) is required. For example, a user can use a user device (100) to request multiple servers (200) to infer analysis results related to a photo using a photo as input.

[0047] In one comparative example of the present disclosure, a plurality of servers (200) may be configured in a cluster form. Referring to FIG. 1, the plurality of servers (200) may be composed of (N+1) server computers. The plurality of servers (200) may be components owned by a user and configured in a private network environment. The plurality of servers (200) may be implemented with computational resources including hardware resources such as CPU, GPU, memory, and storage required at each stage of the machine learning lifecycle. Each of the plurality of servers (200) may execute actual ML / AI workloads. The plurality of servers (200) may be configured based on containers and may be managed using Kubernetes.

[0048] A distributed machine learning inference system (1000) according to one comparative example of the present disclosure can implement dynamic allocation of server resources by utilizing a container orchestration platform such as Kubernetes to support autoscaling. For example, the distributed machine learning inference system (1000) can deploy inference services through the Deployment function of Kubernetes and perform autoscaling according to load using a Horizontal Pod Autoscaler (HPA). In addition, the distributed machine learning inference system (1000) can adjust the number of actual physical nodes by utilizing a Cluster Autoscaler.

[0049] Operating a system that performs large-scale inference processes, such as a distributed machine learning inference system (1000), requires massive computing resources (e.g., multiple servers (200)), which can lead to increased costs and wasted resources. When utilizing the infrastructure of a cloud service provider, costs increase depending on the computing resources used, so efficient resource management is emerging as an important issue.

[0050] To address this, a distributed machine learning inference system according to one embodiment of the present disclosure (see 1000 in FIG. 3) incorporates autoscaling technology that dynamically adjusts the number of server resources. "Autoscaling" refers to a technology that efficiently utilizes resources by automatically adjusting the number of servers according to the load of the system. In particular, "scale-to-zero" or "serverless" technology, which reduces the number of servers to zero when the system is idle, can maximize the efficiency of resource utilization.

[0051] FIG. 2 is a conceptual diagram showing a distributed machine learning inference system according to one comparative example of the present disclosure.

[0052] Referring to FIG. 2, a distributed machine learning inference system (2000) in a zero-to-one situation is shown. The distributed machine learning inference system (2000) in a zero-to-one situation includes a user device (100) and a server (202). In FIG. 2, the server (202) indicates a state in which none of the multiple servers are running, and one server is to be started.

[0053] A zero-to-one situation in serverless computing refers to a situation where a server computer (or, serverless application or service) for inference is reactivated from an inactive state (Scale-to-Zero) upon receiving an inference request from a user device (100).

[0054] While a zero-to-one scenario allows you to leverage the flexibility and cost savings of serverless, optimizing performance requires managing cold starts, which are initialization delay issues.

[0055] However, the distributed machine learning inference system (2000) does not fully support 'scale-to-zero'. Kubernetes is designed to maintain at least one Pod by default, and the ability to reduce the number of Pods to zero is limited. Additionally, when the number of running servers is zero and an inference request is received from a user device (100) to multiple servers, there is a lack of a mechanism to immediately allocate resources, load the execution environment, and process the request because the multiple servers are inactive, and additional time is required to process the inference request. In particular, there is a lack of an asynchronous request management function that temporarily suspends the inference request and processes it after the server starts. That is, in a zero-to-one situation, the distributed machine learning inference system (2000) lacks a configuration (300) that asynchronously processes and manages the zero-to-one situation in conjunction with auto-scaling between the user device (100) and the server (202).

[0056] In this situation, an efficient asynchronous processing method is required to hold requests from the user device (100) and process them after starting the server (202). To implement this, a request management and processing system utilizing a message queue is required, and an integrated architectural design of the entire system is necessary.

[0057] While patterns utilizing message queues to defer and process inference requests are conventionally known, there are few instances where this is integrated with a distributed machine learning inference system to form an overall architecture. Furthermore, linking conventional message queue systems with distributed machine learning inference systems requires additional development and complex configuration, and there is a lack of solutions available, particularly for private network environments.

[0058] Furthermore, detecting the state of a message queue and reflecting it in the operation of a distributed machine learning inference system is a technically complex task. Dynamically adjusting the number of server resources based on the length of the message queue, and ensuring the stability and efficiency of the system, requires comprehensive system configuration and management. Due to these challenges, existing methods face limitations in perfectly implementing a serverless architecture and efficiently managing resources.

[0059] FIG. 3 is a block diagram showing the configuration of a distributed machine learning inference system according to one embodiment of the present disclosure.

[0060] Referring to FIG. 3, a distributed machine learning inference system (3000) according to one embodiment of the present disclosure may include a user device (100), a plurality of servers (200), a sub-queue (Serve Queue) (300), a central system (302), and / or storage (304).

[0061] The distributed machine learning inference system (3000) operates in a private network environment and is implemented as an architecture of a distributed machine learning inference system that supports a serverless structure. The distributed machine learning inference system (3000) can be configured around a sub-queue (300) to receive and process inference requests from user devices (100) and to efficiently manage server resources.

[0062] According to one embodiment of the present disclosure, a sub-queue (300) can receive, hold, or process a user's inference request and respond to the user's inference request. Additionally, the sub-queue (300) can dynamically manage a plurality of servers (200) and control auto-scaling.

[0063] The sub-queue (300) may include a handler (30), a processor (32), a monitor (34), and / or an exporter (36). In this specification, the sub-queue (300) and each component included in the sub-queue (300) may mean hardware capable of performing functions and operations according to each name described in this specification, computer program code capable of performing specific functions and operations, or an electronic recording medium loaded with computer program code capable of performing specific functions and operations. The sub-queue (300) may be executed in a cloud managed by a user. The sub-queue (300) may mean the computing device itself or a part thereof for distributed machine learning inference in a private network environment, or software built on the computing device. The term sub-queue (300) may mean a functional and / or structural combination of hardware for carrying out the technical concept of the present disclosure and / or software for driving said hardware.

[0064] A handler (30) can receive an inference request from a user device (100). The handler (30) can forward or add the inference request received from the user device (100) to a message queue. The inference request received by the handler (30) can be stored in the message queue. In response to the inference request, the handler (30) can detect the inference results generated by multiple servers (200) and send them to the user device (100). The handler (30) can monitor the output storage included in the storage (304). The handler (30) can deliver new result data to the user device (100) whenever new result data is added. Through this asynchronous request processing and user response, the user can use the inference service in an environment similar to directly communicating with the actual server via the sub-queue (300), message queue, and output storage.

[0065] According to an embodiment of the present disclosure, the message queue utilizes Redis for holding and managing inference requests. There may be multiple message queues. Since the sub-queue (300) can process various types of inference requests simultaneously, an independent message queue is configured for each inference request. Additionally, an 'in-flight queue' may be maintained separately to manage requests currently being processed. In the following, 'message queue' may refer to one of the multiple message queues included in the distributed machine learning inference system (3000) according to an embodiment of the present disclosure.

[0066] When the processor (32) retrieves an inference request from the message queue and processes it, the information of the inference request may be recorded in the 'In-flight Queue'. The information recorded in the message queue for the inference request includes identification information and / or metadata of the inference request and the time when the processor (32) last sent a 'processing' signal. If a 'processing' signal is not received from the processor (32) for a certain period of time, it is determined that an error has occurred in the processor (32), and the monitor (34) moves the request back to the message queue for reprocessing.

[0067] The processor (32) can schedule inference requests received from the message queue using autoscaling based on the stack information of the message queue and the activation information of the plurality of servers (200). The stack information of the message queue may include information on how many inference requests are stored in the message queue, and / or identification information for each stored inference request. The activation information of the plurality of servers (200) may include information on how many servers (i.e., activated servers) have resources allocated to them to execute inference, and which of the servers included in the plurality of servers (200) are activated.

[0068] For example, if none of the multiple servers (200) are active, that is, in a zero-to-one situation, the processor (32) may transmit a signal to start at least one of the multiple servers (200) and may forward and store the inference request received from the user device (100) in a message queue to hold the inference request. The signal to start at least one of the multiple servers (200) may be referred to as a 'start signal'. At this time, the processor (32) may set the inference request to be processed again after a predetermined time (e.g., at least 60 seconds), taking into account the time when at least one of the multiple servers (200) is started.

[0069] As another example, if there is one or more active servers among the multiple servers (200), the processor (32) may select one of the multiple active servers (200) and submit an inference request to that server. The processor (32) may receive an inference result for the submitted inference request from that server, which is one of the multiple active servers (200). The inference result received by the processor (32) may be stored in an output storage in real time.

[0070] The processor (32) stores the inference results of the machine learning model in real time in an output storage included in the storage (304), and can divide the result data into fixed sizes and deliver them to the user.

[0071] The monitor (36) can continuously monitor the status of the distributed machine learning inference system (3000) to maintain the stability of the system. The monitor (36) can perform scale-to-zero based on predetermined criteria and system status information. In this specification, "predetermined criteria" refers to criteria for determining whether to perform scale-to-zero. "System status information" refers to real-time status information of the distributed machine learning inference system (3000) for processing inference requests received at a specific time, obtained as a result of monitoring by the monitor (36). "System status information" may include stack information of a message queue at a specific time, the length of a specific message queue, activation information of multiple servers (200) at a specific time, etc.

[0072] For example, if a certain amount of time (e.g., 10 minutes) has elapsed since the first server among the multiple servers (200) started and has not received an inference request for a certain amount of time (e.g., 10 minutes) from the present, the monitor (36) may send a signal to the first server to terminate the first server. In addition, the monitor (36) can detect an inference request that failed to process and increase the priority of the message queue corresponding to the inference request to retry it, thereby improving the stability of the system and ensuring the reliability of the inference request.

[0073] As described above, a distributed machine learning inference system (300) in a private network environment according to one embodiment of the present disclosure can schedule inference requests using autoscaling and dynamically adjust and manage the number of servers according to the load of the system (3000) by performing scale-to-zero.

[0074] The exporter (36) provides system status information, including the length of the message queue, to the outside. Here, the outside may be a central system (302). The reason for providing system status information to the outside is to expose metrics so that they can be utilized by Kubernetes Horizontal Pod Autoscaler (HPA) or the like that perform autoscaling.

[0075] Multiple servers (200) can generate inference results in response to inference requests allocated from a sub-queue (300). According to one embodiment, the multiple servers (200) can be managed through the deployment function of Kubernetes. The startup and shutdown of each of the multiple servers (200) are achieved by the central system (302) receiving signals generated from the processor (32) and monitor (34) of the sub-queue (300) and adjusting the scale of the Kubernetes deployment. Scaling refers to the process of adjusting the number of pods to handle the workload of the application. When adjusting one or more of the multiple servers (200), the central system (302) collects system status information including message queue length information provided by the exporter (36), converts the collected system status information into a metric usable by the HPA and provides it to the HPA, and performs auto-scaling according to the load of each server using the HPA. However, in the case where there are zero pods, a monitor (34) according to one embodiment of the present disclosure dynamically manages multiple servers (200) by performing scale-to-zero based on predetermined criteria and system status information.

[0076] Since the input and output data of a machine learning model may be large in volume, storage (304) may be used to manage them efficiently. According to one embodiment, storage (304) may be a storage or database connected to a sub-queue (300) via wired or wireless connection as illustrated. Storage (304) may be a simple data storage such as Key-Value storage or Blob storage. Storage (304) may include an input storage and / or an output storage. Storage (304) may store input data for inference requests and / or output data for inference results. On the other hand, the sub-queue (300) stores and manages only identification information or metadata of the input data and / or output data using a message queue.

[0077] A user device (100) is a device of a user who wishes to transmit an inference request for a specific task to a machine learning model. The user device (100) may include a smartphone, a tablet PC (Personal Computer), a laptop, and a PC, etc. The user device (100) may install and / or run a web, mobile application, or program provided by at least one server (200) for distributed machine learning inference in a private network environment. For example, the user device (100) may access a web page and may output a user interface screen that can create an inference request containing various components. In this specification, the inference request may be one of the business processes. For example, the inference request may be text such as, “Schedule a study plan by referring to my incorrect answer notes.”

[0078] FIG. 4 is a block diagram showing the configuration of a computing device for distributed machine learning inference according to one embodiment of the present disclosure.

[0079] Referring to FIG. 4, a computing device (400) for distributed machine learning inference according to one embodiment of the present disclosure may be a computing device that provides inference results of a machine learning model in response to a request from a user (or user device), or a user device on which a web page, application, and / or program capable of providing inference results of a machine learning model is installed and executed. The computing device (400) for distributed machine learning inference may correspond to the sub-queue (300) of FIG. 3 or hardware on which the sub-queue (300) is built.

[0080] A computing device (400) for distributed machine learning inference may include a communication interface (40), memory (420), an I / O interface (430) and / or a processor (440), and each component may communicate with one or more communication buses or signal lines.

[0081] The communication interface (410) can be connected to a user device (not shown) via a wired / wireless communication network to exchange data. For example, the communication interface (410) can receive inference requests from the user device or receive inference results from server resources. Additionally, the communication interface (410) can transmit monitoring data including logs or metrics to the user device. The monitoring data transmitted to the user device may be data in which logs or metrics have been compressed, batched, and indexed.

[0082] Meanwhile, a communication interface (410) that enables the transmission and reception of such data includes a wired communication port (411) and a wireless circuit (412), wherein the wired communication port (411) may include one or more wired interfaces, for example, Ethernet, Universal Serial Bus (USB), IEEE1394 (e.g., FireWire, Apple; i.Link, Sony; Lynx, Texas Instruments), etc. Additionally, the wireless circuit (412) may transmit and receive data with an external device via an RF signal or an optical signal. Furthermore, wireless communication may use at least one of a plurality of communication standards, protocols, and technologies, such as GSM (Global System for Mobile Communications), EDGE (Enhanced Data rates for GSM Evolution), CDMA (Code-Division Multiple Access), TDMA (Time Division Multiple Access), Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol.

[0083] The memory (420) can store various data used in the computing device (400) for distributed machine learning inference. For example, the memory (420) can store logs, metrics, system metrics, and / or metadata of files generated from a workload. As another example, the memory (120) can store length information of a message queue according to one embodiment of the present disclosure, metrics available in a Horizontal Pod Autoscaler (HPA), and system status information.

[0084] In various embodiments, the memory (420) may include a volatile or non-volatile recording medium capable of storing various data, commands and / or information. For example, the memory (420) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), Random Access Memory (RAM), Static RAM (SRAM), Read-Only Memory (ROM), Programmable ROM (PROM), Electronically Erasable ROM (EPROM), network storage, cloud, and blockchain database.

[0085] In various embodiments, the memory (420) may store at least one configuration of an operating system (421), a communication module (422), a user interface module (423), and one or more applications (424).

[0086] An operating system (421) (e.g., embedded operating systems such as LINUX, UNIX, MAC OS, WINDOWS, VxWorks, etc.) may include various software components and drivers for controlling and managing general system operations (e.g., memory management, storage device control, power management, etc.) and may support communication between various hardware, firmware, and software components.

[0087] The communication module (422) can support communication with another device through the communication interface (410). The communication module (422) may include various software components for processing data received by the wired communication port (411) or wireless circuit (412) of the communication interface (410).

[0088] The user interface module (423) can receive user requests or inputs from a keyboard, touch screen, keyboard, mouse, microphone, etc. through the I / O interface (430) and provide a user interface on the display.

[0089] The application (424) may include a program or module configured to be executed by one or more processors (440). Here, the application providing services for handling all processes for distributed machine learning inference, including machine learning management, message queue control for holding and managing inference requests, message queue monitoring, and inference request scheduling, may be implemented on a server farm.

[0090] The I / O interface (430) can connect at least one of an input / output device (not shown) of a computing device (400) for distributed machine learning inference, such as a display, keyboard, touch screen, and microphone, to a user interface module (423). The I / O interface (430) can receive user input (e.g., voice input, keyboard input, touch input, etc.) together with the user interface module (423) and process commands based on the received input.

[0091] The processor (440) is connected to a communication interface (410), memory (420), and I / O interface (430) to control the overall operation of a computing device (400) for distributed machine learning inference, and can execute various commands for distributed machine learning inference through an application or program stored in memory (420).

[0092] The processor (440) may correspond to a computing device such as a CPU (Central Processing Unit) or an AP (Application Processor). Additionally, the processor (440) may be implemented in the form of an Integrated Chip (IC), such as a System on Chip (SoC) that integrates various computing devices. Alternatively, the processor (440) may include a module for computing artificial neural network models, such as a Neural Processing Unit (NPU).

[0093] FIG. 5 is a block diagram illustrating an example of the operation of a distributed machine learning inference system according to one embodiment of the present disclosure.

[0094] Figure 5 illustrates the operation of a distributed machine learning inference system based on data, which outputs a learning schedule in response to a user's inference request, "Schedule a learning plan by referring to my incorrect answer notes."

[0095] The user device (100) can transmit an incorrect answer note corresponding to the input data to the handler (30). At this time, the handler (30) can transmit the incorrect answer note to the input storage (304_1) along with identification information (ID: a01983) for the corresponding incorrect answer note. The input storage (304_1) can store the incorrect answer note and the identification information for the corresponding incorrect answer note together. The handler (30) can transmit only the identification information (ID: a01983) of the data for the corresponding inference request to the message queue (31).

[0096] The message queue (31) can store identification information (ID: a01983) of the data for the corresponding inference request in the processor (32). The processor (32) checks the received identification information and requests data corresponding to the identification information from the input storage (304_1), and can receive an incorrect answer note from the input storage (304_1).

[0097] The processor (32) can schedule inference requests. Specifically, the processor (32) can deploy an inference service using the Deployment function of Kubernetes and perform autoscaling based on load using the Horizontal Pod Autoscaler (HPA). Accordingly, the processor (32) can send an incorrect answer note to the first server, which is one of the multiple servers (200).

[0098] For example, a first server, which is one of a plurality of servers (200), can output at least a portion of the learning schedule as an inference result using a machine learning model that takes an inference request as input. A second server, a third server and / or a fourth server, which is one of the plurality of servers (200), can output at least a portion of the learning schedule as an inference result. At least a portion of the learning schedule may be a learning schedule for one week of a user's four-week learning schedule and may be referred to as a 'chunk' or 'data chunk'.

[0099] At least a portion of the learning schedule output by the first server, which is one of the multiple servers (200) (e.g., the learning schedule for the first week of the user's 4-week learning schedule) can be transmitted in real time to the processor (32). The processor (32) can transmit at least a portion of the received learning schedule to the output storage (304_2) along with identification information (ID: a01983) for the learning schedule. The output storage (304_2) can store the learning schedule and the identification information for the learning schedule together.

[0100] After the handler (30) transmits the identification information (ID: a01983) of the data for the inference request to the message queue (31), it can continuously monitor the output storage (304_2). When at least a portion of the learning schedule corresponding to some of the inference results is stored in the output storage (304_2), the handler (30) can detect this and return at least a portion of the learning schedule and its identification information to the user device (100). At least a portion of the learning schedule output by the second server, which is one of the multiple servers (200) (e.g., the learning schedule for the second week of the user's 4-week learning schedule), is transmitted to the processor (32) and stored in the output storage (304_2), just like at least a portion of the learning schedule output by the first server, and the handler (30) can detect this and return it to the user device (100).

[0101] Through this, the user can receive at least a portion of the inference results output by one server included in the multiple servers (200) in real time via the sub-queue (300), message queue (31), and output storage (304_2), thereby enabling the user to use the inference service in an environment similar to directly communicating with an actual server computer.

[0102] FIG. 6 is a flowchart illustrating a method of operation of a computing device for distributed machine learning inference according to one embodiment of the present disclosure. In the following description, for convenience of explanation, the computing device for distributed machine learning inference is referred to as a "computing device."

[0103] The computing device (400) can receive an inference request from a user device (S600).

[0104] The computing device (400) can store identification information of the inference request in a message queue (S602).

[0105] The computing device (400) can schedule the inference request received from the message queue using autoscaling based on the stack information of the message queue and the activation information of a plurality of servers (S604).

[0106] The computing device (400) may transmit a termination signal to the first server among the plurality of servers when the first server among the plurality of servers satisfies a predetermined criterion (S606). Here, the predetermined criterion may mean that at least one of the plurality of servers is in an inactive state for a predetermined first time and has not received an inference request from a user device for a predetermined second time.

[0107] In one embodiment, when a first inference request is received and a plurality of servers (200) are in a zero-to-one situation, the computing device (400) may transmit a start signal to any server (e.g., a second server) assigned by scheduling among the plurality of servers (200). The computing device (400) may store the first inference request in a message queue to hold it. The computing device (400) may transmit the first inference request to the second server after a predetermined time.

[0108] In one embodiment, the computing device (400) may detect a second inference request that failed to generate an inference result. The computing device (400) may control a message queue to increase the priority of the second inference request. The computing device (400) may resend the second inference request to any server (e.g., a third server) assigned by scheduling among a plurality of servers (200).

[0109] A distributed machine learning inference system (3000) in a private network environment according to one embodiment may include a computer program stored on a computer-readable recording medium to perform S600 to S606 of FIG. 6 described above, combined with a computer or computing device which is hardware.

[0110] A distributed machine learning inference system (3000) may be implemented as a computing device comprising at least one processor that executes instructions of programs loaded into memory. A program comprising instructions described to execute S600 to S606 of FIG. 6 described above may be loaded into the memory of the computing device.

[0111] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0112] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0113] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware device may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0114] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0115] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

[0116] The distributed machine learning inference system in a private network environment and the method of operation thereof as described above can be applied to fields utilizing machine learning models and artificial intelligence models.

Claims

1. User device; A sub-queue (Serve queue) configured to receive an inference request from the above-mentioned user device, store identification information of the inference request in a message queue, schedule the inference request received from the message queue using auto-scaling based on stack information of the message queue and activation information of a plurality of servers, and, if a first server among the plurality of servers satisfies a predetermined criterion, perform scale-to-zero on the first server among the plurality of servers; and A plurality of servers configured to generate inference results in response to inference requests allocated from the above sub-queue; comprising Distributed machine learning inference system in a private network environment.

2. In Paragraph 1, The above predetermined criteria are, A case in which at least one of the plurality of servers is inactive during a predetermined first time and has not received an inference request from the user device during a predetermined second time, Distributed machine learning inference system in a private network environment.

3. In Paragraph 2, The above sub-queue is, If the above predetermined criteria are satisfied, the configuration is further configured to transmit a termination signal to the first server among the plurality of servers. Distributed machine learning inference system in a private network environment.

4. In Paragraph 3, The above inference request includes a first inference request, and The above sub-queue is, The system is further configured to detect the first inference request that failed to generate an inference result, control the message queue to increase the priority of the first inference request, and resend the first inference request to a second server among the plurality of servers. Distributed machine learning inference system in a private network environment.

5. In Paragraph 3, The above inference request includes a second inference request, and The above sub-queue is, When the second inference request is received and the plurality of servers are in a zero-to-one situation, the system is further configured to transmit a start signal to the third server among the plurality of servers, store the second inference request in the message queue to hold it, and transmit the second inference request to the third server among the plurality of servers after a predetermined third time. Distributed machine learning inference system in a private network environment.

6. In Paragraph 1, Storage including input storage and output storage; further including, The above input storage is configured to store data for the inference request, and The above output storage is configured to store data regarding the inference result, Distributed machine learning inference system in a private network environment.

7. In Paragraph 6, The above autoscaling is, Dynamically adjusting the activation status of each of the aforementioned multiple servers using the Kubernetes Horizontal Pod Autoscaler (HPA) feature, Distributed machine learning inference system in a private network environment.

8. A method of operation of a computing device for distributed machine learning inference in a private network environment, A step of receiving an inference request from a user device, wherein the inference request includes a first inference request and a second inference request; A step of storing identification information of the above inference request in a message queue; A step of scheduling the inference request received from the message queue using autoscaling based on the stack information of the message queue and the activation information of a plurality of servers; and A step of transmitting a termination signal to the first server among the plurality of servers when the first server among the plurality of servers satisfies a predetermined criterion; method.

9. In Paragraph 8, When the above first inference request is received and the above plurality of servers are in a zero-to-one situation, a step of transmitting a start signal to the second server among the above plurality of servers; The step of storing the above first inference request in the message queue to hold it; and The method further comprises the step of transmitting the first inference request to the second server among the plurality of servers after a predetermined time. method.

10. In Paragraph 9, A step of detecting the second inference request that failed to generate an inference result from a plurality of servers; A step of controlling the message queue to increase the priority of the second inference request; and The method further comprises the step of retransmitting the second inference request to the second server among the plurality of servers. method.

11. Instructions that cause a computer to execute the method according to claim 8, comprising Non-transient computer-readable recording medium.

Images