GPU-NPU-based ai system and method for dynamic update and distributed processing

Abstract

The present invention relates to an AI model dynamic update system including an edge device and a cloud server, wherein the edge device includes: a situation recognition unit for recognizing a situation by detecting a surrounding environment; a model management unit that selects a pre-trained AI model suitable for the current situation and loads the pre-trained AI model to an NPU; an inference unit for inferring data acquired by the situation recognition unit as the AI model; and a control unit for controlling, on the basis of a result of the inference, a platform on which the edge device is mounted, the cloud server includes a learning unit that trains the AI model with a GPU by using the received data and provides an updated model to the edge device, the edge device performs inference only with the NPU without including a GPU, and the GPU of the cloud server is dedicated to model training and updating.

Description

GPU-NPU-based dynamic update and distributed processing AI system and method

[0001] The present invention relates to a GPU-NPU-based dynamic update and distributed processing AI system and method.

[0002] Artificial intelligence (AI) technology in conventional edge computing environments has primarily evolved to perform inference functions within a single device; initially, the main challenge was to rapidly derive inference results even in environments with limited computational resources and battery life by pre-loading fixed AI models.

[0003] However, this approach has structural limitations, such as the difficulty of flexibly responding to new situations outside the given model and the inevitable gradual degradation of inference performance over time.

[0004] Recently, as the complexity of AI models and data diversity have increased rapidly, there is a growing demand to process training and inference in parallel by interconnecting high-performance GPUs on cloud servers with low-power NPUs on edge devices.

[0005] However, in conventional technology, GPUs and NPUs are separated so that each device operates independently or is simply placed in parallel within the same device, making active system operation difficult.

[0006] In particular, during the model transmission process, the entire model must be periodically replaced or manually updated, which leads to excessive network costs and energy consumption. Additionally, the potential for performance degradation or data loss in quantized models was not considered or could not be corrected in conventional structures.

[0007] The present invention aims to provide a dynamic update system capable of high-performance AI inference and adaptive learning even with limited computational resources through a distributed cooperation structure between an edge device and a cloud server.

[0008] In addition, the present invention aims to implement a distributed GPU-NPU structure in which the cloud performs AI model training and the edge device performs real-time inference through a lightweight model.

[0009] In addition, the present invention aims to provide a cloud-based model that is easy to apply while maintaining accuracy by quantizing and optimizing the learned model according to the hardware constraints and network conditions of the edge.

[0010] In addition, the present invention aims to enable the system to autonomously adapt to environmental changes by having the edge select an AI model based on the result of situational awareness and automatically collect data and reflect it in cloud learning when the inference result is ambiguous.

[0011] According to one embodiment of the present invention, an AI model dynamic update system comprising an edge device and a cloud server can be provided, wherein the edge device comprises a situation awareness unit that detects a surrounding environment and recognizes a situation, a model management unit that selects a pre-trained AI model suitable for the current situation and loads it onto an NPU, an inference unit that infers data acquired from the situation awareness unit using the AI ​​model, and a control unit that controls a platform equipped with the edge device based on the inference result; and the cloud server comprises a learning unit that trains the AI ​​model using a GPU with the transmitted data and provides the updated model to the edge device; wherein the edge device performs inference using only an NPU without including a GPU, and the GPU of the cloud server is responsible for model training and updating.

[0012] In addition, according to one embodiment of the present invention, if the AI ​​model required for the current situation is not present on the edge device, the edge device may request the AI ​​model required for the current situation from the cloud server to download it and load it onto the NPU.

[0013] In addition, according to one embodiment of the present invention, the model management unit can replace the model through a virtualized container-based execution environment so that the service of the edge device is not interrupted when switching to a new AI model.

[0014] In addition, according to one embodiment of the present invention, the edge device can apply feedback data received from the cloud server as new training data to further train the AI ​​model in real time.

[0015] In addition, according to one embodiment of the present invention, the updated AI model generated through the additional learning can be immediately deployed to the edge device and applied to inference by replacing the existing running model.

[0016] In addition, according to one embodiment of the present invention, the edge device may further include a feedback unit that temporarily stores a plurality of types of data including at least one of video and audio, determines whether there is ambiguity in the inference result, and transmits the collected data to the cloud server.

[0017] In addition, according to one embodiment of the present invention, if the feedback unit determines that the inference result is ambiguous, it can buffer multiple sensor data at that time for a certain period to collect it as feedback data and transmit the feedback data to a cloud server.

[0018] In addition, according to one embodiment of the present invention, when the edge device or the NPU of the edge device is provided in multiple numbers, the inference task can be distributed and processed across multiple NPUs.

[0019] In addition, according to one embodiment of the present invention, when an inference task is distributed and processed across a plurality of NPUs, each NPU can receive different AI models from a cloud server and perform inference.

[0020] In addition, according to one embodiment of the present invention, when the cloud server or the GPU of the cloud server is provided with a plurality of units, data uploaded from the edge device can be distributed in parallel for learning.

[0021] According to one embodiment of the present invention, a method for dynamically updating an AI model using an edge device and a cloud server comprises: a step in which the edge device detects a surrounding environment and recognizes a situation; a step in which a pre-trained AI model is provided from the cloud server according to the recognized situation and loaded into an NPU; a step in which input data is inferred by the NPU of the edge device and the platform equipped with the edge device is controlled; a step in which the edge device transmits data used for inference to the cloud server; a step in which the cloud server uses the transmitted data to train and update an AI model using a GPU; a step in which the updated AI model is provided to the edge device and the edge device applies the received updated AI model to the NPU; and a step in which the updated AI model is replaced with an existing model and applied to the platform control; wherein the edge device performs inference using only the NPU without including a GPU, and the GPU of the cloud server is responsible for model training and updating.

[0022] In addition, according to one embodiment of the present invention, the model management unit can maintain the context by saving and taking over the computational state information of the previously running AI model when switching to a new AI model.

[0023] In addition, according to one embodiment of the present invention, the model management unit can preload a plurality of AI models into a virtualized container-based execution environment and, when the situation changes, activate only the model most suitable for the situation and apply it immediately to inference.

[0024] In addition, according to one embodiment of the present invention, if there are two or more updated AI models, the cloud server can evaluate the accuracy, model size, and inference speed of each candidate model to select the optimal AI model to be deployed to the edge device.

[0025] According to one embodiment of the present invention, an AI system comprising an edge device and a cloud server can be provided, wherein the edge device includes a situation awareness unit that detects a surrounding environment and recognizes a situation, a model management unit that selects a pre-trained AI model suitable for the current situation and loads it onto an NPU, an inference unit that infers data acquired from the situation awareness unit using the AI ​​model, a control unit that controls a platform equipped with the edge device based on the inference result, and a feedback unit that determines the ambiguity of the inference result, collects ambiguous data and multiple sensor data at a corresponding time point, and transmits them to the cloud; and the cloud server includes a learning unit that uses data transmitted from the edge device to train an AI model on a GPU and provides an updated model to the edge device, and a model storage and distribution unit that manages the AI ​​model version of the entire system and distributes the updated model to the corresponding edge devices.

[0026] In addition, according to one embodiment of the present invention, if there is no AI model suitable for the current situation at the edge device, the model management unit may request and download the corresponding AI model from the cloud server and load it into the NPU.

[0027] In addition, according to one embodiment of the present invention, the model management unit compares the result of recognizing the current situation with the information on possible situations for each of the plurality of AI models, and if the information on possible situations matches the result of recognizing the current situation or if the match rate is greater than or equal to a preset standard, the AI ​​model can be determined to be a suitable model.

[0028] In addition, according to one embodiment of the present invention, the model management unit can replace the model through a virtualized container-based execution environment so that the service of the edge device is not interrupted when switching to a new AI model.

[0029] In addition, according to one embodiment of the present invention, the model management unit predicts a situation that is highly likely to occur in the future based on the current situation recognition result and past situation change pattern, and preloads an AI model corresponding to the predicted situation into the NPU in advance so as to respond immediately when the situation changes.

[0030] In addition, according to one embodiment of the present invention, the sensitivity to situational changes among a plurality of situational AI models is analyzed, and a high switching priority is set for models sensitive to situational changes, so that the corresponding models can be activated preferentially in the event of a sudden change in situation.

[0031] In addition, according to one embodiment of the present invention, the feedback unit determines the result as ambiguous when the reliability of the inference result falls below a predetermined threshold, buffers multiple types of sensor data at the time when the ambiguous result occurred for a certain period to collect them as feedback data, and then transmits the feedback data to the cloud server.

[0032] Additionally, according to one embodiment of the present invention, the cloud server may further include a recovery unit that generates a recovery AI model or provides a previous version model to the edge device when it receives a model failure notification from the edge device.

[0033] In addition, according to one embodiment of the present invention, if the edge device determines that a failure has occurred in the AI ​​model during the inference process, it can immediately switch to a pre-prepared backup AI model and notify the cloud server of the failure.

[0034] In addition, according to one embodiment of the present invention, the cloud server can automatically reflect feedback data transmitted from the edge device as new training data to further train the AI ​​model, and provide the further trained AI model to the edge device without user intervention, thereby enabling continuous improvement of model performance.

[0035] According to one embodiment of the present invention, a method for updating an AI model using an edge device and a cloud server comprises: a step in which the edge device detects a surrounding environment and recognizes a situation; a step in which a pre-trained AI model is provided from the cloud server according to the recognized situation and loaded into an NPU; a step in which input data is inferred by the NPU of the edge device and the platform on which the edge device is mounted is controlled; a step in which the edge device transmits the data used for inference to the cloud server; a step in which a feedback unit of the edge device determines the ambiguity of the inference result and, if ambiguous, collects multiple sensor data at that time and transmits it to the cloud server; a step in which the cloud server uses the transmitted data to train and update an AI model with a GPU; and a step in which a model storage and distribution unit of the cloud server provides the updated AI model to the edge device; wherein the edge device performs inference using only an NPU without including a GPU, and the GPU of the cloud server is responsible for model training and updating.

[0036] In addition, according to one embodiment of the present invention, an AI vision device utilizing dynamic updates of an AI model can be provided.

[0037] In addition, according to one embodiment of the present invention, the model management unit can dynamically adjust the model compression rate or the number of parameters by considering the available memory capacity of the edge device when loading an AI model selected based on the situation recognition result.

[0038] In addition, according to one embodiment of the present invention, multiple AI models having different reliability criteria can be simultaneously activated, and the optimal model can be selected and applied according to the situation.

[0039] According to one embodiment of the present invention, an AI model dynamic update system comprising an edge device and a cloud server can be provided, wherein the edge device comprises a situation awareness unit that senses a situation by sensing a surrounding environment, a model management unit that selects a pre-trained AI model suitable for the current situation and loads it onto an NPU, an inference unit that infers data acquired from the situation awareness unit using the AI ​​model, and a control unit that controls a platform equipped with the edge device based on the inference result; and the cloud server comprises a learning unit that trains the AI ​​model on a GPU using data transmitted from the edge device and provides the updated model to the edge device, and a quantization unit that quantizes the parameters of the updated AI model to match the precision supported by the NPU of the edge device and provides them to the edge device.

[0040] In addition, according to one embodiment of the present invention, the quantization unit verifies the expected accuracy of the model after quantization before quantizing the updated AI model into a format suitable for the NPU precision of the edge device, and can compensate for the degradation of model performance caused by quantization by transmitting correction information including the value of the upper bit precision or a scaling parameter for important weights.

[0041] In addition, according to one embodiment of the present invention, the quantization unit can dynamically set the number of quantization bits per layer depending on the situation by considering the NPU specifications and memory usage of the edge device.

[0042] In addition, according to one embodiment of the present invention, the quantization unit can adaptively adjust the overall quantization level of the AI ​​model according to the power state or network state of the edge device.

[0043] In addition, according to one embodiment of the present invention, the cloud server may further include an optimization unit that evaluates the accuracy, model size, and inference speed of each model when a plurality of updated AI model candidates exist, and selects the optimal AI model to be distributed to an edge device.

[0044] In addition, according to one embodiment of the present invention, the optimization unit can group devices based on at least one of memory capacity, network status, and power status, based on situation awareness information collected per edge device, and select and distribute an AI model optimized for each group.

[0045] In addition, according to one embodiment of the present invention, the cloud server may further include a data management unit that evaluates the importance of data received from the edge device based on learning value, selects only the data with high importance, performs compression or resolution reduction processing, and then provides it to the learning unit.

[0046] In addition, according to one embodiment of the present invention, the edge device may further include a feedback unit that temporarily stores a plurality of types of data including at least one of video and audio, determines whether there is ambiguity in the inference result, and transmits the collected data to the cloud server.

[0047] In addition, according to one embodiment of the present invention, it is possible.

[0048] In addition, according to one embodiment of the present invention, the cloud server can achieve continuous model performance improvement by automatically reflecting the feedback data as new training data to further train the AI ​​model and providing the further trained model to the edge device.

[0049] According to one embodiment of the present invention, a method for dynamically updating an AI model using an edge device and a cloud server comprises: a step in which the edge device detects a surrounding environment and recognizes a situation; a step in which a pre-trained AI model is provided from the cloud server according to the recognized situation and loaded into an NPU; a step in which input data is inferred by the NPU of the edge device and a platform equipped with the edge device is controlled; a step in which the edge device transmits data used for inference to the cloud server; a step in which the cloud server trains and updates an AI model using a GPU with the transmitted data; a step in which the updated AI model is converted into a quantized model suitable for the NPU of the edge device; a step in which the edge device provides the quantized model to the edge device and applies the quantized AI model received to the NPU; and a step in which input data is inferred by applying the quantized AI model and a platform is controlled based on the inference result; wherein the edge device performs inference using only the NPU without including a GPU, and the GPU of the cloud server is responsible for model training and updating.

[0050] In addition, according to one embodiment of the present invention, the quantization unit can dynamically set the number of quantization bits for each layer of the AI ​​model depending on the situation, taking into account the NPU hardware specifications and available memory capacity of the edge device.

[0051] In addition, according to one embodiment of the present invention, the quantization unit can adaptively adjust the overall quantization level of the AI ​​model according to the power state or network state of the edge device.

[0052] In addition, according to one embodiment of the present invention, the quantization unit verifies the expected accuracy of the model after quantization before quantizing the updated AI model to the NPU precision of the edge device, and for weights of high importance, transmits correction information consisting of values ​​of higher bit precision or scaling parameters together to compensate for the degradation of model performance caused by quantization.

[0053] In addition, according to one embodiment of the present invention, the updated AI model can be provided to the edge device in a compressed transmission method that reduces network bandwidth consumption and energy consumption by transmitting only the parameters changed compared to the previous version or by converting and transmitting the model in a compressed format.

[0054] In addition, according to one embodiment of the present invention, an AI vision device utilizing dynamic updates of an AI model can be provided.

[0055] According to the present invention, through a cooperative structure between an edge device and a cloud server, high-performance AI inference and real-time updates are possible even in resource-constrained environments, thereby effectively resolving the limitations of performance degradation and manual updates associated with the operation of existing static AI models.

[0056] Furthermore, according to the present invention, by distributing the roles between the high-performance GPU in the cloud and the low-power NPU at the edge, the efficiency and scalability of the entire system can be simultaneously secured by separating the processing of learning and inference. In particular, through quantization and model optimization techniques, it is possible to apply a model optimized for edge devices while reducing the communication load.

[0057] In addition, according to the present invention, by selecting an AI model based on situational awareness at the edge, and independently determining the ambiguity of inference results to collect and transmit feedback data, an active structure can be provided that enables the system to autonomously adapt to and learn from environmental changes.

[0058] Therefore, the present invention has the effect of simultaneously realizing real-time capabilities, adaptability, continuous performance improvement, and resource optimization of edge AI systems, and can provide high potential for application in various fields such as autonomous driving, smart manufacturing, and security surveillance.

[0059] FIG. 1 is a block diagram illustrating the overall configuration of an edge-cloud cooperation-based AI model update system according to one embodiment of the present invention.

[0060] FIG. 2 is a flowchart illustrating the flow of AI model selection, inference, and feedback data collection operations within an edge device according to an embodiment of the present invention.

[0061] FIG. 3 is a flowchart illustrating the flow of receiving feedback data, training an AI model, and updating operations within a cloud server according to an embodiment of the present invention.

[0062] FIG. 4 is a block diagram illustrating the functional configuration of a model management unit according to one embodiment of the present invention.

[0063] FIG. 5 is a block diagram illustrating the functional configuration of a quantization unit according to one embodiment of the present invention.

[0064] FIG. 6 is a block diagram illustrating the functional configuration of an optimization unit according to one embodiment of the present invention.

[0065] FIG. 7 is a flowchart illustrating the process of predicting situation changes and performing pre-model loading in an edge device according to an embodiment of the present invention.

[0066] FIG. 8 is a conceptual diagram illustrating a distributed processing configuration utilizing multiple NPUs and multiple GPUs according to an embodiment of the present invention.

[0067] The terms used in the embodiments are for illustrative purposes only and should not be interpreted restrictively. The terms "include" or "have" mean that the mentioned features or elements are present and do not exclude the possibility of other elements being present. Terms not specifically defined are to be interpreted according to their meanings in the relevant technical field and not in an overly formal sense.

[0068] Embodiments of the present invention can be applied to various types of systems composed of edge devices, cloud servers, etc., and can also be implemented on various platforms such as autonomous vehicles, factory robots, drones, and smart city surveillance cameras.

[0069] Preferred embodiments of the present invention will be described in detail below with reference to the attached drawings. However, the embodiments described below are merely exemplary configurations for effectively understanding the technical features of the present invention, and the scope of the invention is not limited to specific drawings or embodiments. For example, even if a component is explicitly numbered first, second, etc. in the drawings, such numbers are merely identifiers to distinguish the components from one another and do not indicate order or importance.

[0070] Furthermore, singular expressions used throughout the specification include a plural meaning unless specifically indicated in the context, and terms such as "includes" and "have" are used in a sense that does not exclude the addition of elements other than those enumerated. Unless a specific term is otherwise defined, such terms should be interpreted within the scope of their ordinary meaning to those skilled in the art and should not be interpreted in an overly restrictive sense.

[0071] FIG. 1 is a block diagram showing the overall configuration of an edge-cloud cooperative AI model update system according to an embodiment of the present invention. Referring to FIG. 1, the system may be configured to include an edge device (10) and a cloud server (20).

[0072] The edge device (10) is a terminal device that performs sensing and inference in the field, and may include, for example, an AI computing device mounted on an autonomous vehicle, a control device of an industrial robot, or a CCTV camera.

[0073] The edge device (10) may include a situation awareness unit (110) that detects the surrounding environment in real time and recognizes the current situation, a model management unit (120) that selects a pre-trained AI model best suited to the current situation and loads the model into the NPU of the edge device, an inference unit (130) that includes an NPU and infers input data through the loaded AI model, a control unit (140) that controls a platform equipped with the edge device (10) based on the inference results, and a feedback unit (150) that evaluates the reliability of the inference results, collects additional data if ambiguous, and transmits it to the cloud.

[0074] The edge device (10) may be configured to perform inference using only a low-power NPU without including a GPU for lightweighting, and may have the function of collecting data from multiple sensors and storing multiple AI models in local memory.

[0075] Meanwhile, the platform (10) equipped with the edge device (10) can be a machine or system that is controlled by the edge device, such as a driving system of an autonomous vehicle, a driving unit of a robot, or a flight control module of a drone.

[0076] The control unit (130) can actively move or operate the actual physical device by outputting a control signal to the platform (10) according to the inference result.

[0077] The cloud server (20) can receive data transmitted from the edge device (10), perform high-performance computations, create a new or updated AI model, and provide it back to the edge device.

[0078] To this end, the cloud server (20) may include a training unit (210) that trains an AI model based on a GPU using collected feedback data, a model storage and distribution unit (220) that stores the trained model by version and distributes it to multiple edge devices, a quantization unit (230) that converts the parameters of the trained AI model into a format suitable for the NPU specifications of the edge devices, an optimization unit (240) that selects and distributes an optimal model considering the state information of the edge devices, a data management unit (250) that evaluates the usefulness of the transmitted feedback data and selects / compresses only important data for use in training, and a recovery unit (260) that creates a recovery model or rolls back to a previous version of the model and provides it when a failure occurs in the edge device or the model.

[0079] These cloud servers (20) are connected to edge devices via the internet or a dedicated network, and may be a single physical server as well as a GPU cluster composed of multiple servers.

[0080] In addition, through the resource management unit (270), the NPU / GPU utilization of the entire edge-cloud system can be monitored in real time, and the function of distributing inference operations to available NPUs or distributing training operations to available GPUs according to the workload can be implemented. The detailed operation and interaction of each component will be described in detail later with reference to the drawings.

[0081] FIG. 2 is a flowchart illustrating the flow of an edge-cloud cooperative AI model update method according to the present invention. Referring to FIG. 2, a series of processes is shown in which an edge device (10) and a cloud server (20) cooperate to apply an optimal AI model in real time and continuously update it. In step S11, the edge device (10) can detect the surrounding environment and recognize the situation.

[0082] Specifically, the situation awareness unit (110) collects input data from one or more sensors mounted on the edge device. For example, it simultaneously acquires various information such as images, distance, speed, sound, location, illuminance, and weather from sensors such as a camera, LiDAR, radar, microphone, GPS, and IMU (inertial measurement unit) to determine the current surrounding environment conditions.

[0083] Through this multimodal sensing, the edge device (10) can recognize the situation or scenario in which the platform (10) is situated.

[0084] The situation awareness unit (110) can perform preprocessing and simple analysis on the collected sensor data to derive a situation awareness result that represents the current environment. For example, in the case of an autonomous vehicle, it can analyze camera images and various sensor data to determine situations such as "nighttime urban environment," "driving on a highway in rainy weather," or "occurrence of an obstacle ahead."

[0085] The results of these situation awareness can be used to determine which AI model to use in a subsequent step. The situation awareness unit (110) predefines the types of possible situations and the tasks required in those situations, and can classify which of the cases a real-time input belongs to or determine whether it is a scenario in which multiple situations are combined.

[0086] Through the situation awareness step (S11), the edge device can identify the surrounding environment and work context it is currently in, and based on this, establish a basis for selecting the optimal model.

[0087] Next, in step S12, you can select an AI model suitable for the current situation or request a model from a cloud server if necessary.

[0088] The model management unit (120) receives the previously derived situation recognition result (S11) as input and can identify the model most suitable for the situation among multiple AI models stored in advance. The edge device (10) may have multiple AI models (e.g., day driving model, night driving model, bad weather specialized model, urban / highway model, etc.) that were previously provided from the cloud and stored, and the model management unit (120) can manage information on the corresponding situations for each of these as metadata.

[0089] Continuous response information refers to data indicating the environment or conditions under which each AI model was trained to be used; for example, it may have tags or feature information such as Model A being optimized for "daytime + clear weather" situations, and Model B being optimized for "nighttime + rainy days" situations.

[0090] The model management unit (120) can evaluate the degree of agreement by comparing the current situation recognition result with the situation information that each candidate model can respond to. The model with the highest degree of agreement is selected as the model suitable for the current situation, and it can be decided to use the model only if the degree of agreement is greater than or equal to a preset standard.

[0091] For example, if the current situation is "night, rain," the agreement level of the model corresponding to the "night" condition and the model corresponding to the "rain" condition can be calculated. If there is a single model that satisfies both conditions, that model is selected; however, if there is no model that is a perfect match, the appropriate model can be determined after establishing priorities by considering factors such as which is more important—the "night" or "rain" model—or which model performed better in similar past situations.

[0092] As another embodiment, model selection can be implemented to be performed through a small meta-AI model rather than through simple rule or tag comparison. For instance, a lightweight context classification neural network can be run on an edge device to receive current environment feature vectors as input and recommend the most suitable AI model. This approach is a modified embodiment that enables flexible adaptation to complex mappings between contexts and models through learning.

[0093] Meanwhile, if the model management unit (120) determines that a model suitable for the current situation does not exist locally (at the edge device), it may request a new AI model from the cloud server (20).

[0094] At this time, when the edge device (10) transmits current situation information together with the cloud (200), the model storage and distribution unit (220) of the cloud server can search for a model suitable for the situation and send it down to the edge.

[0095] The downloaded model is stored in the memory of the edge device, and the model selected through step S12 can be immediately loaded into the NPU. Through this model selection / request step (S12), the edge device is able to secure and use the AI ​​model with the best performance for situations that change at every moment.

[0096] As a result, it is possible to respond flexibly to various situations compared to using a single fixed model, and adaptability to the environment can be significantly improved by receiving and utilizing new models in real time when necessary.

[0097] When a model is determined, inference is performed using the corresponding AI model in step S13. Specifically, the selected AI model is loaded onto the inference unit (130) of the edge device (10) under the control of the model management unit (120), and the NPU performs real-time inference using the latest sensor data transmitted from the situation awareness unit (110) as input data. At this time, the inference unit (130) utilizes NPU hardware optimized to smoothly perform deep learning operations even with the limited resources of the edge device, and fast and efficient inference is possible even without a GPU.

[0098] For example, when a vision camera video frame is input, object detection or context-aware AI models can be used to infer risk factors (vehicles, pedestrians, etc.) or semantic information in the current frame, and in the case of audio input, speech recognition or specific event detection models can be executed.

[0099] According to another embodiment of the present invention, the edge device (10) may infer multiple models in a time-sharing manner using a single NPU, or may infer different models in parallel by incorporating multiple NPUs.

[0100] For example, if equipped with two NPUs, one NPU can execute an image recognition model and the other a speech recognition model simultaneously to perform multimodal inference in parallel. Alternatively, it is possible to increase inference speed through parallel computation by distributing a single very complex model across the two NPUs.

[0101] As a result of the operation of this inference unit (130), the edge device (10) obtains a judgment or recognition result regarding the current situation, and the result can be used for platform control or feedback judgment in the next step.

[0102] By executing the inference step (S13), the edge terminal can immediately obtain AI inference results on-site without communication delays with the cloud, thereby enabling rapid response in applications requiring real-time performance (e.g., autonomous driving). Additionally, since this process is performed at the edge, basic inference services can continue even if the network is unstable or delays occur.

[0103] In step S14, platform control operations can be performed based on the inference results. The control unit (130) accepts the AI ​​model inference results obtained in S13 and immediately generates control commands corresponding to the platform (10) equipped with the edge device (10), thereby enabling rapid response to dangerous situations or real-time execution of automated operations.

[0104] For example, in the case of an edge device of an autonomous vehicle, if the inference result includes a judgment such as "need to stop abruptly due to an obstacle ahead," the control unit (130) can immediately issue a command to operate the vehicle's braking device to reduce speed, in the case of a robot platform, control to stop the movement of the robot arm or change the path according to the inference result, and in the case of a drone, can instruct an avoidance maneuver according to the obstacle recognition result.

[0105] In step S15, the feedback unit (150) can automatically determine whether there is ambiguity by comparing the reliability of the current inference result with a predetermined reliability threshold. If the reliability of the inference result falls below the threshold, the result is determined to be ambiguous, additional data is collected for a certain period from the time the ambiguous situation occurred, and the risk situation where feedback data was collected can be addressed quickly.

[0106] Generally, deep learning-based models output a probability value or score indicating the confidence of the prediction along with a predicted value for the input, and the feedback unit (150) can calculate the degree of confidence of the result using these values ​​or through a separate ambiguity determination algorithm.

[0107] For example, in the case of a classification model, a threshold criterion can be applied such that if the highest prediction probability is 90% or higher, it is considered a relatively confident result, and if it is less than 60%, it is considered an ambiguous result. Alternatively, in the case of an object detection model, a situation where the score for the bounding box of a detected object is low or multiple candidates are detected with similar scores can be considered ambiguous.

[0108] When collecting additional data, the edge device (10) can buffer and store multiple types of sensor data at that moment for a short period of time. This is because, in cases where an ambiguous judgment is made based on only one sensor's information, images from different angles or additional sensor information are obtained together to perform a more accurate analysis later.

[0109] For example, if the identity of an object in the camera image is unclear, distance data from the LiDAR sensor mounted on the vehicle, rear camera image, and vehicle speed and acceleration information are collected for several milliseconds to several seconds at the same time and organized into a single feedback data set. Additionally, sound data can be captured and included using an audio sensor. The multimodal data collected in this way is temporarily stored in the feedback unit (150) and can then be prepared for upload to a cloud server.

[0110] On the other hand, if the inference result is clear and sufficiently reliable, the feedback unit (150) may not perform separate data collection.

[0111] Through this ambiguity judgment and data collection step (S15), the system can accurately identify in what situations the currently used model lacks confidence, and obtain the effect of securing training data to resolve that type of situation in the future.

[0112] In step S16, the feedback unit (150) determines that the collected multimodal data is ambiguous and transmits it to the cloud server (20) via the network. Basically, only the feedback data buffered in S15 is selectively transmitted, but depending on the implementation, general inference result data performed by the edge device may also be periodically uploaded to the cloud. For example, the system may be designed so that the edge device periodically reports detection results or statistical information regarding an event to the server.

[0113] However, in a preferred embodiment, detailed sensor source data that is not strictly necessary is not sent to the cloud for bandwidth saving and security purposes. The feedback unit (150) may filter out duplicate or low-importance parts from the collected data or compress them before transmission.

[0114] When the edge device (10) executes step S16, the cloud server (20) can recognize that new data has arrived and initiate a processing procedure. This prepares ambiguous cases detected by the edge device to be transmitted to the cloud and utilized for central learning, while the edge device can continue the real-time inference of step S13 without interruption.

[0115] In other words, feedback data uplinks are performed in the background, and edge devices can continue inference and platform control with the current model.

[0116] Next, the process on the cloud server (20) side proceeds (steps S17 and S18). In step S17, the training unit (210) performs additional training and updates to improve the performance of the existing AI model by applying the new data (feedback data, etc.) uploaded in S16 as new training data.

[0117] Here, learning may refer to additional fine-tuning or continual learning techniques in the case of machine learning algorithms, especially deep learning models. In this process, the cloud server (20) can utilize GPU resources with high computational power to quickly incorporate this new data into the training.

[0118] For example, when the system of the present invention is applied to an autonomous vehicle, video data of "pedestrians that were difficult to detect at night crosswalks" uploaded from the vehicle is immediately trained on a central server GPU, allowing the model to be actively updated to better recognize such cases.

[0119] The learning unit (210) can operate in two ways depending on the situation. First, it can enhance the performance of the same AI model currently used by edge devices by performing additional continual learning. In this case, the weights of the model are fine-tuned with new data, so that misclassification or uncertainty in specific cases can be improved.

[0120] Second, there is a method of generating an entirely new AI model by training it separately (training from scratch). For example, if the uploaded data represents a completely different situation from before (such as a new road environment or a new type of object), the cloud can either develop a completely new model to process that data or additionally train a model with a structure different from the existing one.

[0121] These new models are subsequently provided to the edge as needed, and edge devices may manage multiple models in a model pool and switch between them as needed. The training unit (210) selects an optimal training strategy based on the characteristics and amount of input data, and may also reduce training time through parallel processing using a GPU cluster.

[0122] In addition, only data with high learning value among the raw data uploaded from the data management unit (250) can be selected and preprocessed, and then provided to the learning unit (210).

[0123] For example, if the video data consists of repeated very similar scenes, only a portion can be used and the rest discarded; or, if the resolution is excessively high, the data can be cropped to focus on important objects or the resolution lowered for use in training.

[0124] Through this automatic learning step (S17), the cloud server continuously improves the model to the latest state. By reflecting data from various situations aggregated from edge devices, the overall performance (accuracy, reliability, etc.) of the AI ​​model improves over time. In particular, an active learning loop is implemented in that the model continuously evolves (self-improves) by utilizing data collected from the field without human intervention.

[0125] When the model update with new data is completed by the learning unit (210), in step S18, the cloud server can provide the updated AI model to the edge device.

[0126] Specifically, in the process of distributing a new model that has been created or improved through learning (S18), the model storage and distribution unit (220) assigns version information to the updated AI model, stores it in a database, and transmits it in the form of a download to an edge device (10) that requires the model.

[0127] The edge device (10) stores the received model in memory and then loads a new model in a virtualized execution environment under the control of the model management unit (120). At this time, virtual container-based switching can be performed to smoothly replace the old model that was previously running with the new model.

[0128] That is, the edge device (10) utilizes an isolated execution environment, such as a container or a sandbox, to preload a new model instance into an area that has the same resource allocation as the currently running AI model instance.

[0129] Then, following the schedule synchronization procedure, the existing model is unloaded and the pointer is switched to the new model, thereby replacing the model used for inference. Since all of these processes are carried out in the background within a short period of time, services on edge devices can be maintained without interruption.

[0130] As a result, edge devices can perform more accurate inference and respond quickly to environmental changes. Since models are managed and deployed centrally and in batches, consistent AI model version management can be maintained across multiple devices, and the burden of individual edge devices having to train models on their own can be eliminated.

[0131] In other words, by remotely leveraging the powerful GPU computations of the cloud, high-performance AI services can be enjoyed using only a lightweight NPU. Due to this distributed cooperative structure, edge devices gain the benefits of lightweight design and low power consumption, while cloud servers benefit from continuously improving AI model performance by collecting rich data from multiple devices for training.

[0132] For example, there are cases where, during autonomous driving, a sudden shower causes visibility to deteriorate to a state similar to "night + rain," making the model unsure whether an object captured by the camera is a person or a sign, and the model wavers with a 50%:50% probability.

[0133] At this time, the feedback unit (150) detecting this immediately stores the front camera image, LiDAR distance data, and rain condition sensor of that moment as feedback data for several seconds and transmits it to the cloud to retrain the vehicle's existing model by reflecting the latest data, or if necessary, to train a new integrated model for bad weather day and night.

[0134] Referring to Fig. 3, detailed procedures performed on the cloud side from steps S21 to S25 are illustrated.

[0135] In step S21, the cloud server can receive feedback data from the edge device. This is a step corresponding to S16 of Fig. 2 described earlier, and is a process of receiving one or more data sets transmitted from the edge through the network in the cloud.

[0136] The cloud server (20) manages the received data by assigning metadata (timestamp, source ID, etc.) to it, and the data management unit (250) analyzes the data first. The data management unit (250) plays the role of evaluating the importance of the uploaded data.

[0137] For example, it is possible to classify whether the data pertains to commonly occurring situations (such as normal driving data) or rare and novel events (such as new types of obstacles).

[0138] In addition, each data can be scored by estimating its potential to contribute to the improvement of the existing model's performance. To do this, methods such as receiving and analyzing the model's inference logs or reliability information together, or measuring the error by feeding the input into the current model, can be used. Data deemed to have low importance is not used for training immediately but is merely stored or discarded, and data with high importance is selected and delivered to the training unit (210), thereby allowing for efficient utilization of limited GPU resources.

[0139] In step S22, training is performed using the GPU of the cloud server. This is the same as S17 in FIG. 2, and is a step in which the training unit (210) additionally trains the AI ​​model with new data or creates a new model.

[0140] As previously explained in the description of FIG. 2, the specific operation involves executing a GPU-based deep learning algorithm to adjust the model parameters. The cloud server (20) may process multiple training tasks in parallel depending on the situation. For example, if different types of feedback data arrive simultaneously from multiple edge devices, if there are multiple GPU resources, separate training instances can be launched in parallel to respond quickly. (Details regarding the parallel operation of multiple GPUs will be further explained in FIG. 8, which will be described later.)

[0141] The learning unit (210) can apply learning techniques for unreliable data (such as semi-supervised learning) by semi-automatically generating labels when new input data requires labeling, or by using it for learning by attaching provisional labels based on the prediction values ​​of existing models in cases where the data is judged to be ambiguous. As a result of this step S22, an AI model (or models) that reflects the latest data is prepared, and in the next process, it is optimized and deployed to the edge.

[0142] In step S23, a quantization optimization process is performed. This is a step to lighten the trained AI model so that it can operate smoothly on edge devices. The specific operation will be explained in detail later through Fig. 5.

[0143] Next, in step S24, the updated AI model is transmitted to the edge device. This is the same process as S18 in FIG. 2 and is handled by the model storage and distribution unit (220).

[0144] The final model, after undergoing processes such as quantization, is downloaded to the target edge device(s) via the network.

[0145] If the model is required by multiple edge devices, it can be deployed simultaneously to multiple edge nodes. Conversely, if the model is updated due to a special situation on a specific device, it can be initially deployed to that device and then expanded to other devices later if necessary.

[0146] During deployment, the cloud server monitors the success of the transmission and can guarantee reliable transmission by retransmitting or performing a backup transmission through an adjacent server node if there is no response within a certain period.

[0147] The edge device that receives the model immediately loads the new model into the NPU and incorporates it into the inference (process S18), and the cloud server verifies this, thereby completing one update cycle. Finally, step S25 is labeled as platform control, which indicates the completion of the entire process and a return to normal operating mode rather than a special operation.

[0148] FIG. 4 is a conceptual diagram showing the functional configuration of a model management unit (120) of an edge device according to the present invention. More specifically, FIG. 4 illustrates the flow of logic for managing and switching models within an edge terminal from step S31 to S35.

[0149] Referring to FIG. 4, in step S31, the model management unit (120) receives the situation recognition result and starts the operation.

[0150] S31 is a part linked to S11 of FIG. 2, which provides current situation information detected by the situation awareness unit (110) to the model management unit. In the next step S32, the current situation can be compared with locally stored model candidates.

[0151] That is, a list of multiple AI models and their metadata (such as information on possible situations) that are stored in advance on the edge device (10) can be retrieved to search for a model suitable for use in the current situation.

[0152] If there are multiple model candidates responsible for matching or similar situations, all of them are selected as the primary candidate group. In step S33, a model is requested from the cloud server if necessary. If it is determined that there is no suitable model among the local candidates, or if a better model is expected, the model management unit (120) may request verification from the cloud model storage and distribution unit (220) via a communication module.

[0153] For example, if the currently perceived situation is a "very foggy dawn," there may not be a local model specialized for foggy environments. In this case, a model can be found by sending keywords describing the situation (dense fog, dawn) to the cloud.

[0154] The cloud server (20) searches the database and, if a model matching the condition exists, immediately sends it to the edge, and if not, it may proceed with new training.

[0155] As an example, an edge device can request and redownload models that were previously received but are not currently resident in memory. For instance, if some models have been unloaded from storage due to insufficient memory, the device can request the cloud to load those models back into memory.

[0156] In step S34, a suitability judgment can be performed. For local candidates (S32) and / or models provided from the cloud (S33), the most suitable AI model can be finally determined by calculating the degree of agreement or suitability with the current situation.

[0157] Specifically, a similarity metric is calculated between the situation range covered by each candidate model and the current situation perception result. At this stage, fuzzy matching or scoring techniques can be used beyond simple matching. For example, if the current situation includes two or more features (night + rain), Model A fits "night" with a score of 0.8 and "rain" with a score of 0.4, while Model B fits "night" with a score of 0.5 and "rain" with a score of 0.9. After calculating the scores in this manner, the higher one is selected by comparing the weighted sum scores.

[0158] The model management unit (120) determines and selects a model that meets the situation above a preset standard as a suitable model, and if the suitability of all candidates falls short of the standard, it may select the one with the highest suitability as a second option or use the basic model.

[0159] Through step S34, the final selected model is determined, and then in step S35, the model transition procedure can be executed. The model management unit (120) can utilize virtualization container-based model transition technology to smoothly transition from the model currently running on the edge device to the newly selected model.

[0160] To prevent potential errors during the model switching process, the model management unit (120) performs the switching after verifying the integrity and loading status of the new model. If the new model fails to load, a safety mechanism such as fail-safe to continue using the existing model may be provided.

[0161] In accordance with steps S31 to S35 of Fig. 4, management logic for applying the optimal model for each situation is implemented within the edge device, thereby enabling efficient operation of multiple models while extracting the best performance for each situation. Additionally, through the model selection criteria of S32 to S34, the edge device can automatically evaluate the suitability for the situation and perform intelligent operations by automatically selecting and using the optimal model without human intervention.

[0162] FIG. 5 is a flowchart showing the functional configuration of a quantization unit (230) according to the present invention. Referring to FIG. 5, the detailed operation of a quantization unit that converts an updated AI model within a cloud server (20) into an edge device-friendly model is illustrated from steps S41 to S46.

[0163] First, in step S41, an updated AI model is received as input. This means that the latest model, which has just completed additional training from the training unit (210), is passed on. Since this model typically includes floating-point precision neural network parameters trained on a GPU, the computational load and memory requirements are too high to use directly on an edge NPU.

[0164] Therefore, analysis to lighten the original model input in S41 can be started. In step S42, the NPU specifications, available memory, power status, etc. of the edge device are collected. This is a step in which the quantization unit (230) identifies the constraints that must be reflected in the model optimization.

[0165] The cloud server (20) may have hardware profile information that the edge device (10) has previously transmitted or registered. For example, the information includes the arithmetic operation bit width (e.g., 8-bit, 16-bit) and operation type (INT8, FP16, etc.) supported by the edge device's NPU, the total memory capacity and current free memory of the device, the remaining battery level and power supply characteristics of the device, and the network connection status.

[0166] The quantization unit (230) receives this information and determines how much to compress the model and what format to convert it into during the subsequent quantization and model conversion process. In step S43, the optimal number of quantization bits per layer can be set.

[0167] Generally, it is convenient to apply quantization to all layers of a neural network model uniformly at the same bit level (e.g., 8 bits for all layers), but there may be a significant loss of accuracy in some important layers. Therefore, the quantization unit (230) of the present invention analyzes the sensitivity and distribution of weights for each layer and can set various precision levels, such as 8 bits for some layers and 6 bits for others.

[0168] For example, in a Convolutional Neural Network (CNN), layers close to the final output are maintained at 8 bits because feature maps are differentiated and even small changes can significantly affect prediction, while large convolutional layers in the early part are boldly given lower bits if lowering them to 4 or 6 bits has little impact on overall accuracy.

[0169] In step S44, the quantization level can be adjusted according to the power / network conditions of the edge device. This is a step for fine-tuning the number of bits per layer, which was initially determined, by increasing or decreasing it, taking into account dynamic elements (power status, communication status, etc.) among the constraint information collected in S42.

[0170] For example, if the current communication speed of the edge device is very low, the bits of some less important layers may be lowered by one step to reduce the file size, and conversely, if the battery of the edge device is sufficient and communication is good, the bits may be raised to improve accuracy. In this way, the quantization unit (230) can perform flexible optimization by reflecting the real-time state information of the edge device.

[0171] In step S45, correction parameters for important weights can be generated. This is a proprietary process for correcting errors caused by quantization, specifically providing additional information regarding parameters of high importance within the model.

[0172] For example, if some weights are large or there is a risk of loss during quantization, the major bit values ​​or scale information of those weights can be stored separately to compensate.

[0173] Then, during inference on edge devices, this correction information is utilized to approximate the output of the originally intended model by adding a small correction operation (e.g., multiplying by a pre-stored scale) to the result of the simple quantization operation.

[0174] This method can significantly compensate for the degradation in inference accuracy caused by quantization. Correction parameters are selectively used only for values ​​that produce an effect while increasing the amount of transmitted data by only a very small amount. Finally, in step S46, a final model with completed quantization can be generated.

[0175] The quantized model is received by the edge device (10) and becomes a form that can be executed immediately on the NPU. If necessary, the quantization unit (230) can verify the accuracy of the model once more after quantization.

[0176] For example, test data is input into a cloud GPU using a quantized model to perform inference, and it is verified whether the accuracy is maintained above a threshold. If the drop in accuracy is greater than expected, the process of adjusting quantization bit settings or adding correction parameters and re-verifying is repeated to ultimately complete a quantized model that satisfies both performance and efficiency.

[0177] The model created in this way is transferred to the model storage / distribution unit (220) and then passed to the transmission step to the edge (S24 / S18).

[0178] Through steps S41 to S46 of Fig. 5, the cloud server can provide a large neural network model optimized enough to run on edge terminals, thereby enabling the use of advanced AI functions even with limited edge resources.

[0179] In particular, the quantization unit (230) of the present invention applies an adaptive quantization technique rather than fixed quantization, so that different optimizations can be performed for each edge device or situation, thereby always realizing the best point in terms of performance-resource efficiency. In addition, since the degradation of accuracy due to quantization is minimized by correcting critical factors, reliability can be ensured while increasing efficiency during network transmission or real-time inference.

[0180] FIG. 6 is a flowchart illustrating the functional configuration of an optimization unit (240) according to the present invention. Referring to FIG. 6, steps S51 through S55 illustrate a procedure in which a cloud server (20) selects an optimal model among a plurality of models and distributes it to edge devices.

[0181] First, in step S51, update candidate models are received. This is based on the premise that there is one or more models completed by the learning unit (210).

[0182] According to the embodiment, the cloud server may have multiple model candidates. For example, two or more model versions may have been trained with different algorithms or parameters during the training phase, or multiple versions of the model may have been created with different bit width combinations during the quantization phase.

[0183] The optimization unit (240) quantitatively evaluates each of the model candidates given as input and selects the best one among them, or, in some cases, selects several by group (high-performance model, lightweight model, low-power model). In step S52, the accuracy, model size, and inference speed indicators of each model can be calculated.

[0184] This is the process of calculating performance metrics that serve as criteria for evaluating model candidates; regarding accuracy metrics, various indicators suitable for the model's application, such as the F1-score, can be utilized in addition to general-purpose classification accuracy (%). Next, model size refers to the capacity of the model file or memory usage, and can vary significantly depending on whether quantization is performed. Additionally, inference speed can be expressed as the expected number of inferences per second or the time required for a single inference when the model is executed on the hardware specifications of an edge device.

[0185] This inference speed can be estimated through simulations on the cloud or benchmark information measured on similar devices in the past can be used. The optimization unit (240) can quantify the three representative indicators (accuracy, magnitude, and speed) to construct a model-specific performance profile.

[0186] In step S53, the status of the target edge devices can be collected. That is, the memory capacity, network status, power status, etc., of each edge device being considered for model deployment can be identified.

[0187] Some edge devices feature the latest hardware with ample memory and fast NPUs, while others are older and may be resource-constrained; some are connected to a power source with always sufficient power available, whereas others are battery-powered, making energy efficiency a priority.

[0188] The cloud can either register and know this information in advance or query the edges in real-time to receive responses. In step S54, group classification can be performed based on the collected device conditions. That is, it is a process of determining the group classification of a single edge device or grouping multiple edge devices into clusters with similar conditions.

[0189] For example, classification can be based on memory capacity or network status. Additionally, classification criteria can be defined in various ways, such as grouping by power consumption or usage (e.g., autonomous vehicles, CCTV). Multidimensional clustering can also be performed by considering multiple criteria together.

[0190] Through this classification process, the optimization unit (240) can identify the priority characteristics of the model required by each group. For example, the low-spec group prioritizes "model size" optimization, the high-spec group prioritizes "accuracy" maximization, and the mid-spec group may want a compromise that ensures "speed (real-time)" above a certain standard.

[0191] These requirements can be expressed as weight values ​​for each group and reflected in the model evaluation indicators, and the optimal model for each group can be selected and distributed through step S55.

[0192] The optimization unit (240) manages even these multi-model scenarios and may decide to deploy two or more models in parallel if necessary.

[0193] Through the processes S51 to S55 of Fig. 6, the cloud server can achieve balanced optimization of the entire system by automatically selecting and providing the optimal model in terms of performance and efficiency among several candidate models.

[0194] In other words, rather than uniformly deploying a single model to all edges, it supplies different models tailored to the situation, helping each device achieve optimal performance in its own environment.

[0195] FIG. 7 is a flowchart illustrating the process of predicting a situation change and model pre-loading according to the present invention. Referring to FIG. 7, the edge device (10) predicts a situation that is likely to occur in the future based on the current situation and past patterns and responds in advance, and the function is illustrated in steps S61 through S64.

[0196] First, in step S61, the current situation is recognized. This is basically the same operation of the situation recognition unit (110) as in S11 of FIG. 2, in which the edge device identifies the current environmental state in real time.

[0197] In the next step S62, past patterns can be analyzed. The edge device (10) may accumulate and possess a record of past situational changes, or a cloud server may provide the edge with the analysis of past data.

[0198] In this stage, it is possible to statistically analyze what situations occurred in what order during a recent period, how long specific situations lasted, and what tendencies there were in transitioning to subsequent situations.

[0199] In step S63, future situations can be predicted. By combining the previously analyzed past patterns (S62) and the currently detected situation (S61), situations that are highly likely to occur in the near future are probabilistically estimated. This can be understood as using a type of situation prediction model.

[0200] For a simple implementation, one can identify which situations frequently occurred in the past under conditions similar to the current situation and select a few with the highest probability as prediction results.

[0201] In particular, more sophisticated predictions can be performed in conjunction with a cloud server (20), in which case the edge device can operate by receiving future situation information from the outside and preparing in advance. The predicted future situation may not be just one, and multiple possibilities may be considered together.

[0202] In step S64, a model required for the predicted situation can be preloaded. That is, in preparation for the future situation judged to be most likely in S63, an AI model to be used in that situation is prepared in advance on the edge device (10).

[0203] Specifically, a model that may differ from the currently used model is loaded into memory or even loaded into the NPU in advance and put on standby just before switching. Then, when the predicted situation actually occurs and the situation recognition unit (110) detects it, it can immediately switch to the standby model without delay to perform inference.

[0204] In this case, it is also possible to load multiple models into separate containers and selectively activate only the appropriate model when a situation arises.

[0205] Through this model preloading technique, the time required for model switching is drastically reduced, resulting in ultra-low latency during switching and maximizing responsiveness to rapid changes in the situation.

[0206] FIG. 8 is a conceptual diagram illustrating a multi-NPU and multi-GPU parallel processing configuration according to the present invention.

[0207] The system of the present invention is not limited to operation between a single edge device and a single cloud server, but can be extended to multiple edge devices and multiple server nodes.

[0208] Referring to FIG. 8, a plurality of inference units (131, 132, 133, edge devices or multiple NPUs within the edge devices) are shown on the left, and a plurality of learning units (211, 212, 213, multiple GPU nodes in the cloud) are shown on the right.

[0209] This represents a distributed system consisting of multiple edge-to-multiple servers. First, considering the case where there are multiple edge devices (10), each edge device can operate individually with the same internal configuration (context awareness, model management, inference, feedback, etc.) as described above.

[0210] These various edge devices are connected to a common cloud server (or server pool) to upload data and receive models. The cloud server (20) can collect data streams from numerous edges simultaneously, and a multi-GPU cluster can be operated to process them efficiently.

[0211] If the cloud learning unit (210) is implemented in a distributed manner using multiple GPU instances rather than a single one, step S22 (learning) can be parallelized and used for distributed deep learning, etc.

[0212] In such a multi-GPU distributed learning environment, the model storage / distribution unit (220) can combine partial models completed on each GPU or collect the final output and send it to the edge.

[0213] Meanwhile, there may be cases where multiple NPUs are provided within a single edge device. For example, if two or more NPU chips or modules are installed in an edge device, the system of the present invention can operate to distribute inference tasks to multiple NPUs.

[0214] The model management unit (120) monitors the availability and load status of multiple NPUs within the terminal and can divide and assign tasks to be inferred in parallel. In one embodiment, if an object detection model is run on NPU 1 and a speech recognition model is run on NPU 2 to process multimodal data simultaneously, both results can be obtained within one cycle and complementary judgments can be made.

[0215] The model management unit (120) oversees the utilization of multiple NPUs and can dynamically allocate inference instances based on the available resources of each NPU. In this multiple NPU structure, there is also the advantage that even if one NPU fails, another NPU can serve as a backup.

[0216] Likewise, on the cloud server side, the resource management unit (270) can efficiently distribute multiple training tasks to each GPU node.

[0217] In summary, the multi-NPU / GPU parallel processing configuration shown in FIG. 8 is an example demonstrating that the system of the present invention can be widely applied from small-scale environments to large-scale distributed environments.

[0218] According to another embodiment of the present invention, the system may additionally be provided with a recovery mechanism to ensure service continuity in the event of a model failure or error.

[0219] The edge device (10) and the cloud server (20) cooperate to recover model-level failures in real time, thereby ensuring that platform control functions are not interrupted even in unexpected situations.

[0220] Specifically, the edge device (10) includes a module that monitors the operation of the AI ​​model, and if it is determined that an error has occurred in the currently running AI model, it detects it immediately.

[0221] Examples of model errors include cases where no inference result output occurs for a certain period of time or where the output value is repeated identically, exhibiting abnormal behavior, or cases where the model process is halted due to software exceptions or memory access errors.

[0222] At this point, the edge device immediately switches to a pre-prepared backup AI model. This backup model (backup model) may be a simplified version capable of performing functions similar to the currently failed model, or it may be an older version of the model that has been previously used and whose stability has been verified.

[0223] The model management unit (120) can utilize the virtualization container technology described above to unload the failed model while simultaneously loading a backup model directly into the NPU and resuming inference. This ensures that platform control functions are maintained without interruption, even temporarily.

[0224] Next, the edge device (10) can transmit a signal to the cloud server (20) to notify it of a failure. This signal includes information such as a device identifier, the version or type of the failed model, and the nature of the error, enabling the cloud to diagnose the problem.

[0225] When the recovery unit (260) of the cloud server receives a notification, it can prepare the latest normal version or patched version of the corresponding model in response. If a previous stable version of the same model exists, it can be selected and prepared for distribution, and if it is confirmed that the cause of the failure is model file corruption or a bug, a new version modified by the developer can be obtained and replaced.

[0226] This series of recovery model preparation tasks can be designed to proceed quickly according to predetermined automation rules, so as to be performed without human intervention as much as possible. The prepared recovery model can be urgently transmitted to the corresponding edge device via the model storage / distribution unit (220).

[0227] The edge device (10) receives this, loads it into the background, replaces it with the backup model using a container switching method, and the edge device can resume operation with the normal new model.

[0228] Subsequently, the cloud server can improve the model by extracting the data or situations that caused the failure and reflecting them in the learning process.

[0229] This can further enhance the reliability of on-site edge-cloud AI systems and ensure service continuity even in exceptional situations that may occur during actual operation, thereby minimizing damage caused by failures.

[0230] Embodiments of the present invention have been specifically described above with reference to the drawings. The system and method of the present invention can be utilized in various application fields based on the edge-cloud collaboration structure as described above.

[0231] For example, the present invention can be applied to a control system for a factory automation robot to replace an AI model that controls robot movements according to changes in the production line situation (product type, work mode, etc.) and to continuously improve the robot's work accuracy by collecting error case data occurring during the operation and learning from it in the cloud.

[0232] In another embodiment, by applying the present invention to an autonomous flight system of a drone, obstacle avoidance performance can be improved by switching and applying a computer vision model or a control policy model according to indoor / outdoor conditions, weather conditions, and flight mission, and by sending images obtained during drone flight to the cloud to learn anomaly signs.

[0233] The present invention can also be applied to CCTV surveillance systems in smart factories or smart cities. By using different object recognition AI models for day and night, and during normal times and when events occur, and by automatically selecting rare accident videos at the edge and sending them to the cloud for intensive training, an intelligent security system can be constructed that detects accidents more accurately.

[0234] As such, the system of the present invention can be implemented in various modified and applied forms, and those skilled in the art may make various changes within the scope of the spirit of the invention in addition to the embodiments mentioned herein. Accordingly, the scope of the rights of the present invention should be interpreted within the scope of the contents described in the appended claims and their legal equivalents.

Claims

1. In an AI model dynamic update system including edge devices and cloud servers, The edge device comprises: a situation awareness unit that detects the surrounding environment and recognizes the situation; a model management unit that selects and loads a pre-trained AI model suitable for the current situation; an inference unit that includes an NPU and infers data acquired from the situation awareness unit using the AI ​​model; and a control unit that controls a platform equipped with the edge device based on the inference results. The cloud server includes a learning unit that uses the transmitted data to train an AI model on a GPU and provides the updated model to the edge device; An AI model dynamic update system characterized in that the edge device above performs inference using only an NPU without including a GPU, and the GPU of the cloud server is dedicated to model training and updating.

2. In Paragraph 1, The above edge device is An AI model dynamic update system characterized by requesting and downloading the AI ​​model required for the current situation from the cloud server and loading it into the NPU when the AI ​​model required for the current situation is not present on the edge device.

3. In Paragraph 2, The above model management unit is An AI model dynamic update system characterized by replacing the model through a virtualized container-based execution environment so that edge device services are not interrupted when switching to a new AI model.

4. In Paragraph 1, The above edge device is The AI ​​model is further trained in real time by applying feedback data received from the aforementioned cloud server as new training data, and An AI model dynamic update system characterized by immediately deploying an updated AI model generated through the above additional learning to the above edge device and applying it to inference by replacing the existing running model.

5. In Paragraph 1, The above edge device is Further including a feedback unit that temporarily stores multiple types of data including at least one of video and audio, determines whether there is ambiguity in the inference result, and transmits the collected data to the cloud server. An AI model dynamic update system characterized by buffering multiple sensor data at a given time for a certain period to collect feedback data and transmitting the feedback data to a cloud server when the inference result is determined to be ambiguous.

6. In Paragraph 1, When multiple edge devices or NPUs of the edge devices are provided, inference tasks are distributed and processed across multiple NPUs, and An AI model dynamic update system characterized by distributing inference tasks across multiple NPUs, wherein each NPU receives a different AI model from a cloud server to perform inference.

7. In Paragraph 1, The above cloud server is An AI model dynamic update system characterized by further including a quantization unit that quantizes the parameters of the updated AI model to match the precision of the NPU supported by the edge device and provides them to the edge device.

8. In Paragraph 7, The above quantization unit is Before quantizing the updated AI model into a format suitable for the NPU precision of the edge device, verify the expected accuracy of the model after quantization, and An AI model dynamic update system characterized by compensating for model performance degradation caused by quantization by transmitting correction information, including values ​​of higher bit precision or scaling parameters, together with important weights.

9. In Paragraph 8, The above quantization unit is An AI model dynamic update system characterized by dynamically setting the number of quantization bits per layer according to the situation, taking into account the NPU specifications and memory usage of the edge device.

10. In Paragraph 8, The above quantization unit is AI model dynamic update system characterized by adaptively adjusting the overall quantization level of the AI ​​model according to the power state or network state of the edge device.

11. In Paragraph 1, The above cloud server is An AI model dynamic update system characterized by further including an optimization unit that evaluates the accuracy, model size, and inference speed of each model and selects the optimal AI model to be deployed to edge devices when multiple updated AI model candidates exist.

12. In Paragraph 11, The above optimization unit is An AI model dynamic update system characterized by grouping devices based on at least one of memory capacity, network status, and power status, based on situational awareness information collected per edge device, and selecting and distributing an AI model optimized for each group.

13. In Paragraph 1, The above edge device is It further includes a feedback unit that determines the ambiguity of the inference result, collects ambiguous data and multiple sensor data at that time, and transmits them to the cloud; The above cloud server is An AI model dynamic update system characterized by further including a model storage and distribution unit that manages AI model versions of the entire system and distributes updated models to corresponding edge devices.

14. In Paragraph 13, The above model management unit is An AI model dynamic update system characterized by the fact that, when an AI model suitable for the current situation does not exist, the model management unit requests and downloads the corresponding AI model from the cloud server and loads it into the NPU.

15. In Paragraph 14, The above model management unit is An AI model dynamic update system characterized by comparing the current situation recognition result with the response capability situation information of each of a plurality of AI models, and determining the AI ​​model as a suitable model if the response capability situation information matches the current situation recognition result or if the match rate is greater than or equal to a preset standard.

16. In Paragraph 13, The above model management unit is An AI model dynamic update system characterized by predicting situations likely to occur in the future based on current situation recognition results and past situation change patterns, and preloading an AI model corresponding to the predicted situation into the NPU in advance to respond immediately when the situation changes.

17. In Paragraph 13, The above model management unit is An AI model dynamic update system characterized by analyzing the sensitivity to situational changes among multiple situational AI models, setting a high switching priority for models sensitive to situational changes, and prioritizing the activation of such models in the event of a sudden change in situation.

18. In Paragraph 1, The above cloud server is A recovery unit that generates a recovery AI model or provides a previous version model to the edge device upon receiving a model failure notification from the edge device; further comprising The above edge device is An AI model dynamic update system characterized by immediately switching to a pre-prepared backup AI model and notifying the cloud server of the failure when it is determined that a failure has occurred in the AI ​​model during the inference process.

Images