Framework device for automating virtual sensor operation and retraining

Abstract

The present invention relates to a framework device for automating virtual sensor operation and retraining, the device comprising: a virtual sensor generation unit for packaging a model and an operation module of a virtual sensor to build a Docker image; a Docker hub unit for storing the Docker image through version management and managing and distributing the Docker image; a Docker image management unit for generating and managing a container for automated operation processing of the Docker image and dynamically controlling resource allocation of the container; a Docker operation service unit for executing the Docker image as an independent process in the container through a Docker daemon (104) and monitoring whether the container operates normally; a virtual sensor execution environment unit for setting an execution environment required for the virtual sensor through the container; and a data repository for storing the Docker image and data required for training, retraining, and operation of the virtual sensor.

Description

Framework device for virtual sensor operation and retraining automation

[0001] The present invention relates to a method for implementing a framework for automating model retraining in the virtual sensor operation stage, and to a framework device for automating virtual sensor operation and retraining that can provide more convenient services to creation groups and operation groups by providing an automation process to users in a service platform utilizing virtual sensors.

[0002]

[0003] Due to recent advancements in IoT technology, interest in autonomous vehicles and digital twin-related businesses is growing, and the reality is that this interest is also expanding to the utilization of virtual sensors in the same field.

[0004] While virtual sensors are operated to replace existing physical sensors, it is essential to undergo a model retraining process due to factors such as discrepancies with existing data.

[0005] In the case of retraining, it is conducted as a manual procedural process in which users directly handle many areas, ranging from data management to the retraining process and model derivation, by operating a separate retraining-related system. The reality is that this process is complex in terms of procedures and structure, and is subject to significant time constraints.

[0006] Korean Published Patent No. 10-2024-0082184 relates to a continuous integration and distribution system for a deep learning framework application service model, comprising: a plurality of edge servers providing deep learning inference services; a plurality of distributed servers each equipped with a deep learning framework application query-based deep learning database server; and a main server managing the plurality of distributed servers to perform distributed training on a learning model; a Software Configuration Management (SCM) repository that automatically processes revision, version control, backup, and rollback processes of a service model table, which is the result of the distributed training learning model service model; and a controller that distributes the service model table to the edge servers for execution according to a preset distribution policy when there is a change in the service model table in the SCM repository.

[0007]

[0008] [Prior Art Literature]

[0009] [Patent Literature]

[0010] Korean Published Patent No. 10-2024-0082184 (June 10, 2024)

[0011]

[0012] One embodiment of the present invention aims to provide a framework device for automating virtual sensor operation and retraining, capable of building a Docker image by packaging a virtual sensor model and an operation module.

[0013] One embodiment of the present invention aims to provide a framework device for automating virtual sensor operation and retraining, which can set security accessibility to private or public images by analyzing the sensitivity of sensing data output from a virtual sensor model.

[0014] One embodiment of the present invention aims to provide a framework device for virtual sensor operation and retraining automation that manages the lifecycle of containers through creation, start, stop, and deletion via a Docker daemon and monitors the operation of containers through log records.

[0015]

[0016] Among the embodiments, the framework device for automating virtual sensor operation and retraining includes: a virtual sensor creation unit that packages a virtual sensor model and operation module to build a Docker image; a Docker hub unit that stores the Docker image through version control and performs management and distribution of the Docker image; a Docker image management unit that creates and manages a container for automated operation processing of the Docker image and dynamically controls resource allocation of the container; a Docker operation service unit that executes the Docker image as an independent process in the container through a Docker daemon (104) and monitors whether the container is operating normally; a virtual sensor execution environment unit that sets up an execution environment required for the virtual sensor through the container; and a data storage unit that stores data required for learning, retraining, and operation of the virtual sensor and the Docker image.

[0017] The above Docker Hub unit can analyze the sensitivity of the sensing data output from the virtual sensor model and set the security accessibility of the Docker image as a private image or a public image.

[0018] The above Docker operation service unit manages the lifecycle of the container through the creation, start, stop, and deletion of the container via the above Docker daemon, and monitors the operation of the container through log records so that if the container terminates abnormally, the container can be automatically restarted through the log records.

[0019] The virtual sensor execution environment unit described above provides an execution environment including necessary libraries, dependencies, and configuration files to enable the virtual sensor to operate smoothly, and supports the virtual sensor in accessing necessary data to analyze and learn in real time.

[0020]

[0021] The disclosed technology may have the following effects. However, this does not mean that a specific embodiment must include all of the following effects or only the following effects; therefore, the scope of the rights of the disclosed technology should not be understood as being limited by this.

[0022] A framework device for automating virtual sensor operation and retraining according to one embodiment of the present invention can build a Docker image by packaging a virtual sensor model and an operation module.

[0023] A framework device for automating virtual sensor operation and retraining according to one embodiment of the present invention can analyze the sensitivity of sensing data output from a virtual sensor model and set security accessibility to a private image or a public image.

[0024] A framework device for virtual sensor operation and retraining automation according to one embodiment of the present invention can manage the lifecycle of containers through creation, start, stop, and deletion via a Docker daemon and monitor the operation of containers through log records.

[0025]

[0026] FIG. 1 is a diagram illustrating the functional configuration of a framework device according to the present invention.

[0027] Figure 2 is a diagram illustrating the system configuration of the framework device of Figure 1.

[0028] FIG. 3 is a flowchart illustrating an embodiment of a framework method for automating virtual sensor operation and relearning according to the present invention.

[0029] FIG. 4 is a diagram illustrating a relearning process according to an embodiment of the present invention.

[0030] FIG. 5 is a diagram illustrating a virtual sensor framework operation business process according to an embodiment of the present invention.

[0031]

[0032] The description of the present invention is merely an example for structural or functional explanation, and therefore the scope of the present invention should not be interpreted as being limited by the examples described in the text. That is, since the examples are subject to various modifications and may take various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical concept. Furthermore, the objectives or effects presented in the present invention do not imply that a specific example must include all of them or only such effects; therefore, the scope of the present invention should not be understood as being limited by them.

[0033] Meanwhile, the meaning of the terms described in this application should be understood as follows.

[0034] Terms such as "first," "second," etc., are intended to distinguish one component from another, and the scope of rights shall not be limited by these terms. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0035] When it is stated that one component is "connected" to another component, it should be understood that it may be directly connected to that other component, or that there may be other components in between. Conversely, when it is stated that one component is "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationships between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0036] A singular expression should be understood to include a plural expression unless the context clearly indicates otherwise, and terms such as "include" or "have" are intended to specify the existence of the implemented features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood not to preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0037] In each step, identifiers (e.g., a, b, c, etc.) are used for convenience of explanation and do not describe the order of the steps; the steps may occur differently from the specified order unless a specific order is clearly indicated in the context. That is, the steps may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

[0038] The present invention may be implemented as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Additionally, the computer-readable recording medium may be distributed across networked computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0039] Unless otherwise defined, all terms used herein have the same meaning as generally understood by those skilled in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with the context of the relevant technology and should not be interpreted as having an ideal or overly formal meaning unless explicitly defined in this application.

[0040]

[0041] FIG. 1 is a diagram illustrating the functional configuration of a framework device according to the present invention.

[0042] Referring to FIG. 1, the framework device (10) may include a virtual sensor creation unit (100), a Docker hub unit (101), a Docker image management unit (103), a Docker operation service unit (105), a virtual sensor execution environment unit (107), and a data storage unit (110).

[0043] At this time, embodiments of the present invention are not required to include all of the above components simultaneously; depending on each embodiment, some of the components may be omitted, or some or all of the components may be selectively included. The operation of each component will be described in detail below.

[0044]

[0045] The virtual sensor generation unit (100) can build a Docker image by packaging the virtual sensor's model and operation module. Here, the virtual sensor may correspond to a device that software-implements the functions performed by a real sensor, for example, a program that provides data output from a real sensor through a data analysis model and algorithm, such as a vibration sensor and a monitoring sensor. Additionally, the model module may correspond to a software algorithm or mathematical model developed to suit a specific task or purpose to be performed by the virtual sensor, and the operation module may correspond to support code that performs tasks such as resource allocation, data flow control, error detection, and recovery to ensure the stable operation of the model. The virtual sensor generation unit (100) can provide a consistent environment by generating a Docker image that includes an operating system, libraries, code, and configuration files necessary for the operation of a specific virtual sensor. Here, the virtual sensor generation unit (100) can perform consistent operation in the same environment when executed as a container by converting a software stack including the virtual sensor, model module, and operation module into a Docker image.

[0046] The Docker Hub (101) can store Docker images through version control and perform management and distribution of Docker images. Here, the Docker Hub (101) can create each version by distinguishing changes in the Docker image with tags and store each version in the data store (110) so that a specific version of the Docker image can be loaded when needed. For example, the Docker Hub (101) can store versions in the data store (110) by assigning tags such as 1.0, 1.1, and 1.2 to each version during the process of building a Docker image that reflects new features or changes, and can roll back to a previous version in case of an error or perform recovery operations if the distribution of a new version fails.

[0047] Additionally, the Docker hub (101) can build and store new versions of Docker images by reflecting changes in the model or environment used by the virtual sensor through version management of Docker images, and, not necessarily limited to this, can secure storage space of the data storage (110) by deleting old versions of Docker images that have not been used for a certain period of time.

[0048] In one embodiment, the Docker Hub (101) can analyze the sensitivity of the sensing data output from the virtual sensor model and set the security accessibility of the Docker image as a private image or a public image. Here, the Docker Hub (101) can perform a sensitivity analysis of the sensing data and analyze whether the sensing data contains sensitive data such as personal information, corporate confidential information, or sensitive industrial data, and is not necessarily limited thereto, but can set specific criteria for sensitivity based on the content, importance, and confidentiality of the sensing data. The Docker Hub (101) can determine whether to set the Docker image containing the sensing data to private or public based on the results of the sensitivity analysis of the sensing data collected from each virtual sensor.

[0049] The Docker image management unit (103) can create and manage containers for the automated operation processing of Docker images and dynamically control the resource allocation of containers. Here, the containers may correspond to providing an independent environment in which a virtual sensor is to run. The Docker image management unit (103) can create containers suitable for the operating environment of the virtual sensor based on images stored in the Docker hub unit (101). For example, the Docker image management unit (103) can create instances to provide the necessary execution environment for the images, check the execution status of the instances through container management, and perform container lifecycle management tasks such as starting, stopping, and deleting if necessary to maintain an optimal state. Through this, the Docker image management unit (103) can dynamically control the resource allocation of containers by distributing the load of containers when specific events (e.g., increased load of virtual sensor, abnormal termination, etc.) occur.

[0050] The Docker operation service unit (105) can run Docker images as independent processes within containers through the Docker daemon (104) and monitor whether the containers are operating normally. Here, the Docker daemon (104) may correspond to a background process that performs overall management tasks such as creating, running, stopping, and deleting containers. The Docker operation service unit (105) can interact with the Docker daemon (104) to run Docker images as independent processes. For example, during the process of converting Docker images into containers and running them, the Docker operation service unit (105) can operate them individually so as to reduce interference between containers and not affect other processes by ensuring that each container operates as an independent process with its own memory space and CPU resources.

[0051] Additionally, the Docker operation service unit (105) can monitor whether the container is operating normally through the Docker daemon (104). For example, the Docker operation service unit (105) can monitor whether the container is operating normally, detect an event when a specific virtual sensor terminates abnormally due to insufficient memory, and automatically restart the container to minimize system downtime. In one embodiment, the Docker operation service unit (105) can analyze container logs to identify containers that frequently terminate abnormally and report the container logs to prevent additional event situations from occurring.

[0052] In one embodiment, the Docker operation service unit (105) manages the lifecycle of containers through the creation, start, stop, and deletion of containers via the Docker daemon (104), and monitors the operation of containers through log records so that if a container terminates abnormally, the container can be automatically restarted via the log records. Here, the Docker operation service unit (105) can optimize system resources by performing container lifecycle management through the Docker daemon (104). For example, if the execution of a specific virtual sensor is required during a specific time period, the Docker operation service unit (105) can optimize system resources by creating and operating a container only during that time period, and stopping and deleting the container after the analysis is finished.

[0053] In one embodiment, the Docker operation service unit (105) stores all operation and error records that occur while the container is running as logs in the data store (110) and can track the container status through monitoring the logs. For example, if an abnormal termination situation occurs due to insufficient memory, the Docker operation service unit (105) can identify the reason for the termination through the logs and immediately restart the container to prevent system interruption. Additionally, if the container terminates abnormally due to a network connection problem, the Docker operation service unit (105) can recover the problem through the restart function and take necessary measures by checking the network status through the log records.

[0054] The virtual sensor execution environment unit (107) can set up the execution environment required for the virtual sensor through a container. Here, the virtual sensor execution environment unit (107) can prepare software configurations necessary for the smooth operation of the virtual sensor inside the container through the execution environment settings. For example, the virtual sensor execution environment unit (107) can set up an execution environment through a container that includes Python libraries including Pandas and Numpy, libraries such as Scipy required for signal processing, dependencies for specific libraries, and configuration files that store configuration values ​​necessary for virtual sensor operation, such as the sensor data collection cycle or network settings.

[0055] In one embodiment, the virtual sensor execution environment unit (107) provides an execution environment including necessary libraries, dependencies, and configuration files to enable the virtual sensor to operate smoothly, and supports the virtual sensor in accessing necessary data to analyze and learn in real time. Here, the virtual sensor execution environment unit (107) may provide an execution environment including libraries required during the process of the virtual sensor processing data or performing specific analysis tasks. For example, the virtual sensor execution environment unit (107) may manage the virtual sensor's dependencies by installing the libraries required by the virtual sensor through the Docker daemon (104).

[0056] In one embodiment, the virtual sensor execution environment unit (107) can immediately process data collected by the virtual sensor in conjunction with an external data source and generate analysis results. Here, the virtual sensor execution environment unit (107) can set up an interface with the connected data source so that sensor data is continuously fed in through a data collection engine. Additionally, if the sensor data needs to be periodically updated according to environmental changes, the virtual sensor execution environment unit (107) can receive and analyze real-time data from the data storage (110) and perform an immediate response according to the results. The virtual sensor execution environment unit (107) is not necessarily limited to this and can support the virtual sensor to smoothly perform relearning by providing authentication information and connection information necessary to access data stored in the data storage (110).

[0057] The data storage (110) may correspond to a storage device that stores various information required during the operation process of the framework device (10). For example, the data storage (110) may store data and Docker images required for the learning, retraining, and operation of the virtual sensor, but is not necessarily limited thereto, and may store information collected or processed in various forms during the process in which the framework device (10) performs the framework method for automating the operation and retraining of the virtual sensor according to the present invention.

[0058] In addition, in FIG. 1, the data storage (110) is included in the framework device (10) as a logical storage device, but is not necessarily limited thereto and can be shown as an independent device.

[0059]

[0060] Figure 2 is a diagram illustrating the system configuration of the framework device of Figure 1.

[0061] Referring to FIG. 2, the framework device (10) may include a processor (210), memory (230), user input / output unit (250), network input / output unit (270), and communication port unit (290).

[0062] The processor (210) can execute a virtual sensor operation and relearning automation procedure according to an embodiment of the present invention, manage memory (230) that is read or written during this process, and schedule the synchronization time between volatile memory and non-volatile memory in memory (230). The processor (210) can control the overall operation of the framework device (10) and is electrically connected to the memory (230), user input / output unit (250), and network input / output unit (270) to control the data flow between them. The processor (210) can be implemented as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) of the framework device (10).

[0063] The memory (230) may include an auxiliary storage device implemented as non-volatile memory such as an SSD (Solid State Disk) or HDD (Hard Disk Drive) and used to store all data required by the framework device (10), and may include a main memory implemented as volatile memory such as RAM (Random Access Memory). Additionally, the memory (230) may store a set of instructions that execute the framework method for virtual sensor operation and relearning automation according to the present invention by being executed by an electrically connected processor (210).

[0064] The user input / output unit (250) includes an environment for receiving user input and an environment for outputting specific information to the user, and may include an input device including an adapter such as a touch pad, touch screen, virtual keyboard, or pointing device, and an output device including an adapter such as a monitor or touch screen. In one embodiment, the user input / output unit (250) may correspond to a computing device connected via remote access, and in such case, the framework device (10) may be performed as an independent server.

[0065] The network input / output unit (270) provides a communication environment for connecting to a virtual sensor through a network and may include an adapter for communication such as a LAN (Local Area Network), MAN (Metropolitan Area Network), WAN (Wide Area Network), and VAN (Value Added Network). Additionally, the network input / output unit (270) may be implemented to provide short-range communication functions such as WiFi and Bluetooth, or wireless communication functions of 4G or higher, for wireless transmission of data.

[0066] The communication port section (290) can be implemented as a port mapping table that performs data routing during the process of transmitting and receiving data through a network. Here, the communication port section (290) can distinguish communication sessions between the Docker hub section (101) and the server and prevent data collisions during the process of transmitting and receiving data by assigning a unique source port to the Docker hub section (101).

[0067]

[0068] FIG. 3 is a flowchart illustrating an embodiment of a framework method for automating virtual sensor operation and relearning according to the present invention.

[0069] Referring to FIG. 3, the framework device (10) can build a Docker image by packaging the model and operation module of the virtual sensor through the virtual sensor generation unit (100) (step S310). The framework device (10) can store the Docker image through version control via the Docker hub unit (101) and perform management and distribution of the Docker image (step S420).

[0070] Additionally, the framework device (10) can create and manage containers for automated operational processing of Docker images based on the Docker image management unit (103) and dynamically control resource allocation of the containers (step S430). The framework device (10) can run the Docker image as a process independent of the container through the Docker daemon (104) based on the Docker operation service unit (105) and monitor whether the container is operating normally (step S470).

[0071] The framework device (10) can set up the execution environment required for the virtual sensor through a container based on the virtual sensor execution environment unit (107). The framework device (10) can store data and Docker images required for the learning, retraining, and operation of the virtual sensor through the data storage (110).

[0072]

[0073] FIG. 4 is a diagram illustrating a relearning process according to an embodiment of the present invention.

[0074] Referring to FIG. 4, the framework device (10) can generate learning information within the framework device (10) and proceed with learning according to the relearning process when a relearning event is generated. Here, the framework device (10) can act as a trigger to automatically start relearning when it detects a relearning event, such as when the error rate of the virtual sensor increases or when receiving a new type of data. For example, when the framework device (10) performs anomaly detection from environmental data in the virtual sensor, it can generate a learning dataset reflecting the latest data pattern and update the model to the latest state.

[0075] In one embodiment, the framework device (10) may undergo a model error correction step according to a model correction process after training is completed, and the model after the model correction process is completed may derive an updated model after deployment. Here, the model correction process may correspond to a step of fine-tuning the performance of the model after training is completed and correcting errors not found during the retraining process. The framework device (10) may manage the model to maintain an optimal state by performing a model error correction step according to a model correction process, such as correcting data bias, correcting errors, and improving generalization performance.

[0076]

[0077] FIG. 5 is a diagram illustrating a vehicle reverse driving scenario according to an embodiment of the present invention.

[0078] Referring to FIG. 5, the framework device (10) can perform an entire process consisting of eight steps: a model creation step, a virtual sensor registration step, a preparation step at the edge gateway, an actual framework operation step, an anomaly detection step for an error, a model retraining step, a new version update step, and a framework re-operation step.

[0079] The framework device (10) can generate a virtual sensor model, such as data collection, preprocessing, model design, and initial training, during the model generation stage, and can set various algorithms and parameters according to the tasks to be performed by the virtual sensor model. For example, the framework device (10) can generate an anomaly detection virtual sensor model to detect anomalies by analyzing data patterns in a specific state.

[0080] Additionally, the framework device (10) can perform a virtual sensor registration step to register and operate the model created in the model creation step to the virtual sensor system. For example, the framework device (10) can package the model into a container to build a Docker image and register it to the framework device (10), thereby providing an independent environment containing the necessary settings and dependency files for the virtual sensor.

[0081] Next, the framework device (10) can deploy the virtual sensor to the edge gateway during the preparation phase at the edge gateway and perform preparations for operation in the actual environment. For example, the framework device (10) can interact with real-time data in the field through the edge gateway and collect and analyze data through the virtual sensor by distributing configuration files and network settings so that the virtual sensor can be connected to the edge gateway.

[0082] The framework device (10) can perform an immediate response when a preset threshold is exceeded or an exception occurs by actually collecting, analyzing, and monitoring data through a virtual sensor during the actual framework operation phase. For example, the framework device (10) can perform an anomaly detection phase for errors, which determines that a specific data pattern or prediction result deviates from the expected range as an anomaly and provides a warning signal. Next, the framework device (10) can perform a model retraining phase to retrain and update the model when the performance of the existing model deteriorates or a new data pattern occurs, and can update a new version. The framework device (10) can perform a framework re-operation phase to operate the updated model again within the framework so that the entire system operates in an up-to-date state.

[0083]

[0084] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims.

[0085]

[0086] [National R&D projects that supported this invention]

[0087] [Project ID] 1711160509

[0088] [Project No.] 2022-0-00591-001

[0089] [Ministry Name] Ministry of Science and ICT

[0090] [Specialized Research Management Agency] Korea Institute of Information & Communication Technology Planning & Evaluation, affiliated with the National Research Foundation of Korea

[0091] [Research Project Name] ICT Convergence Industry Innovation Technology Development (R&D)

[0092] [Research Project Title] Development of Virtual Sensor Framework Technology to Resolve Sensor Blind Spots in Digital Twin Environments

[0093] [Contribution Rate] 1 / 1

[0094] [Organizing Agency] Electronics and Telecommunications Research Institute

[0095] [Research Period] 2022-04.01~2025.12.31

[0096]

[0097] [Explanation of the symbol]

[0098] 10: Framework Device

[0099] 100: Virtual Sensor Creation Section 101: Docker Hub Section

[0100] 103: Docker Image Management Department 104: Docker Daemon

[0101] 105: Docker Operations Service Department 107: Virtual Sensor Execution Environment Department

[0102] 109: MQTT communication module 110: Data storage

[0103] 210: Processor 230: Memory

[0104] 250: User I / O Section 270: Network I / O Section

[0105] 290: Communication port section

Claims

1. A virtual sensor creation unit that packages the virtual sensor model and operation module to build a Docker image; A Docker Hub unit that stores the above Docker image through version control and performs management and distribution of the above Docker image; A Docker image management unit that creates and manages containers for automated operational processing of the above Docker image and dynamically controls resource allocation of the said containers; A Docker operation service unit that runs the above Docker image as an independent process in the above container through a Docker daemon and monitors whether the above container is operating normally; A virtual sensor execution environment unit that sets up the execution environment required for the virtual sensor through the above container; and A framework device for automating virtual sensor operation and retraining, comprising a data repository that stores data required for the learning, retraining, and operation of the virtual sensor and the Docker image.

2. In paragraph 1, the Docker hub portion A framework device for automating virtual sensor operation and retraining, characterized by analyzing the sensitivity of sensing data output from the model of the virtual sensor and setting security accessibility to the Docker image as a private image or a public image.

3. In paragraph 1, the above Docker operation service unit A framework device for virtual sensor operation and relearning automation, characterized by managing the lifecycle of the container through creation, start, stop, and deletion via the Docker daemon, monitoring the operation of the container through log records, and automatically restarting the container through the log records if the container terminates abnormally.

4. In paragraph 1, the virtual sensor execution environment A framework device for automating virtual sensor operation and retraining, characterized by providing an execution environment including necessary libraries, dependencies, and configuration files to enable the virtual sensor to operate smoothly, and supporting the virtual sensor to access necessary data to analyze and learn in real time.

Images